Skip to content

Static, self-contained render with mapnik built in (linux/musl + macOS) - #2

Merged
boydjohnson merged 8 commits into
mainfrom
feature/static-render
Sep 17, 2026
Merged

boydjohnson merged 8 commits into
mainfrom
feature/static-render

Conversation

@boydjohnson

@boydjohnson boydjohnson commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

Builds mapnik from source with the osmflat datasource compiled in as a built-in plugin, and links render against it. The result loads nothing at runtime — no libmapnik, no osmflat.input, no system fonts.

cmake -S . -B build-static -DCMAKE_BUILD_TYPE=Release \
    -DOSMFLAT_STATIC_RENDER=ON -DOSMFLAT_FULLY_STATIC=ON   # -FULLY_ is musl-only
cmake --build build-static --target render

The dynamic build is untouched and remains the default.

platform runtime deps archive
x86_64-linux-musl none (fully static) 10.9 MB
aarch64-linux-musl none (fully static) 10.2 MB
aarch64-apple-darwin /usr/lib + /System only 22 MB

All three render byte-identical output to the dynamic build (md5 d48d0556925e17e872d0ac25229319d2 on the style-relations fixture).

How osmflat gets into a static libmapnik

Mapnik's static-plugin table is compile-time (src/datasource_cache_static.cpp + src/CMakeLists.txt), so cmake/static-mapnik/patch-mapnik.cmake edits the fetched tree: it adds plugins/input/osmflat (sources referenced from this repo, no copies), links it into libmapnik, and registers it in the static datasource map. Each edit anchors on an exact upstream string, fails loudly if mapnik moved it, and is skipped when already applied.

The scale_denominator trap

Dropping PROJ keeps a 9 MB proj.db out of the binary, but a PROJ-less mapnik knows only epsg:4326 and classifies it as geographic — making scale_denominator ~111319x larger than what PROJ reports for the equivalent +proj=longlat +datum=WGS84 +no_defs. Every style here is tuned against the degree-based numbers (0.1 ≈ neighborhood), so that would have silently disabled every MaxScaleDenominator gate while still rendering something plausible. (First attempt did exactly this: 214688 where the dynamic build says 1.92857.)

The patch instead makes a PROJ-less projection classify +proj=longlat exactly as PROJ does. Other proj4 strings still throw.

Platform wrinkles

musl: harfbuzz's CMake config names the shared library (attempted static link of dynamic object), so the pkg-config fallback is forced; cargo asks for -lgcc_s, which musl ships only shared, so Corrosion's link interface gets gcc_eh.

macOS: Homebrew's libharfbuzz.a is built against graphite2, which has no static archive — so harfbuzz is built from source with just freetype and CoreText (dropping glib/pcre2/intl too). icu4c/zlib/bzip2 are keg-only, so their prefixes are passed explicitly. Homebrew's Boost::regex names icu libs bare, which wouldn't resolve and would prefer dylibs; mapnik's own workaround drops them. The build script fails if otool -L shows anything outside /usr/lib or /System.

CI

scripts/build-static-render.sh (Alpine, via docker run — identical to a local podman run) and scripts/build-static-render-macos.sh build, test and package; scripts/package-render.sh is shared. static-render.yml runs all three platforms on every PR and caches the build tree — mapnik is ~16 min cold on x86_64, ~1 min warm.

release.yml builds the same three archives from a clean tree (no cache) on a v* tag, requires the tag to match project(... VERSION), smoke-tests each unpacked linux archive on a bare alpine image, and publishes with a SHA256SUMS. workflow_dispatch builds without publishing.

Licensing

The repo had no license file; it's now MIT. Apache-2.0 was considered and dropped because it's incompatible with LGPL-2.1 for a combined work, and the static binary links LGPL mapnik. Each archive ships LICENSE-MIT, mapnik's LICENSE.mapnik, and a NOTICE naming every statically linked library plus how to rebuild and relink.

Not addressed: x86_64 macOS (the matrix could gain it; only arm64 is built), and a warm-cache CI measurement — the key naming changed in the last commit, so the first restore happens on the next push.

🤖 Generated with Claude Code

boydjohnson and others added 8 commits September 17, 2026 12:52
OSMFLAT_STATIC_RENDER=ON fetches mapnik v4.2.2, patches osmflat into its
compile-time static-plugin table, and links render against the static
libmapnik. OSMFLAT_FULLY_STATIC=ON adds -static, so on musl the binary
needs nothing at runtime -- no libmapnik, no osmflat.input, no fonts.

Dropping PROJ avoids shipping proj.db, but a PROJ-less mapnik knows only
epsg:4326 and classifies it as geographic, which scales scale_denominator
by ~111319x against what PROJ reports for the equivalent
'+proj=longlat +datum=WGS84 +no_defs' -- silently breaking every style's
MaxScaleDenominator gating. The patch teaches projection to classify
+proj=longlat the way PROJ does, so both builds agree (verified: 1.92857
for the same style/bbox/size on both).

scripts/build-static-render.sh builds, tests, and packages it in Alpine;
static-render.yml runs that script for x86_64 and aarch64.

Verified on aarch64: 16.5M stripped binary, ctest passes, the unpacked
tarball renders on a bare alpine image, and fonts/ beside the binary is
picked up (8173-byte labelled PNG vs 2785 without).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Mapnik is ~175 TUs and dominates the build (~16 min cold on x86_64).
Put everything reusable under $BUILD_DIR -- mapnik's fetched source and
objects, the cargo registry (CARGO_HOME), and downloaded apk packages --
so one actions/cache path covers it all.

Cache keys are unique per run (GitHub caches are immutable) with
restore-keys falling back to the newest cache for that arch; changed
build plumbing or a new mapnik tag starts a fresh lineage, and ninja
decides what to actually redo.

Locally: 468M tree, and a re-run with it in place takes 1m11s (one relink)
against ~5 min cold.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A v* tag builds the static render for x86_64 and aarch64, smoke-tests
each unpacked archive on a bare alpine image, and publishes both with a
SHA256SUMS. workflow_dispatch builds the same matrix without publishing.

Unlike CI, release builds skip the cache: a published binary should come
from a clean build of the tagged tree. The tag must match the project
VERSION, so archive names can't drift from the release.

Archives now carry mapnik's LGPL-2.1 text plus a NOTICE naming every
statically linked library and pointing at the sources and build script,
which is what the relinking requirement asks for.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The repo had no license file at all. Adds both texts, a README License
section, and ships them in the release archive.

The static archives are the one wrinkle: Apache-2.0 is considered
incompatible with LGPL-2.1, so a redistributed binary with mapnik linked
in is the MIT arm of the dual license combined with LGPL mapnik. The
NOTICE says so. Building the ordinary osmflat.input module against a
system mapnik is dynamic linking and raises none of this.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Apache-2.0 is considered incompatible with LGPL-2.1 for a combined work,
and the static render links LGPL mapnik. Dual licensing worked (a
redistributor would take the MIT arm), but MIT alone leaves nothing to
explain.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four pushes in quick succession each started a full static-render build
(~16 min cold on x86_64), all racing to build commits already replaced.
A concurrency group per workflow+ref keeps one run per PR branch. main
and tags share the group name but never cancel, so a release build can't
be killed by a later push.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The build runs as root in the container (apk needs it), and cargo leaves
files the runner user can't read, so the post-job cache save died on
'tar: ... Permission denied' -- the first cached run saved nothing.
chown the tree back before the cache step.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
macOS can't link libSystem statically, so this bundles mapnik and every
third-party library and leaves only macOS' own dylibs/frameworks dynamic;
the build script fails if otool -L shows anything outside /usr/lib or
/System.

Three macOS-specific problems:
- Homebrew's libharfbuzz.a is built against graphite2, which ships no
  static archive, so harfbuzz is built from source with just freetype and
  CoreText (which also drops glib/pcre2/intl).
- icu4c, zlib and bzip2 are keg-only, so their prefixes are passed
  explicitly or nothing finds the archives.
- Homebrew's Boost::regex names icu libs bare, which wouldn't resolve and
  would prefer dylibs; mapnik's own workaround drops them.

Packaging moves to scripts/package-render.sh, shared by both platforms.

Verified locally on arm64: links only /usr/lib + /System, ctest passes,
the unpacked archive renders with its bundled fonts, and output is
byte-identical to both the dynamic macOS build and the linux/musl static
build (md5 d48d0556925e17e872d0ac25229319d2).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@boydjohnson
boydjohnson merged commit 6981fb7 into main Sep 17, 2026
4 checks passed
@boydjohnson
boydjohnson deleted the feature/static-render branch September 17, 2026 19:18
@boydjohnson boydjohnson changed the title Static, self-contained render with mapnik built in Static, self-contained render with mapnik built in (linux/musl + macOS) Sep 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant