Skip to content

Embed ICU data in the linux static render (fixes regex filters and label wrapping) - #4

Merged
boydjohnson merged 1 commit into
mainfrom
fix/embed-icu-data
Sep 18, 2026
Merged

boydjohnson merged 1 commit into
mainfrom
fix/embed-icu-data

Conversation

@boydjohnson

Copy link
Copy Markdown
Owner

Regex filters and label wrapping have been broken in every linux static release (v0.1.0, v0.300.0). macOS is unaffected.

The bug

Alpine builds ICU with archive packaging: its libicudata.a is a 1,300-byte stub, and the real data is a separate icudt76l.dat that a relocated binary can't find. ICU's code was linked; its data wasn't. Two symptoms with that one cause:

symptom from effect
Could not initialize ICU resources for expression: ... Boost.Regex's ICU traits, rethrown by mapnik src/expression.cpp:53 Any .match()/.replace() filter makes the style fail to load
could not create BreakIterator: U_MISSING_RESOURCE_ERROR mapnik src/text/text_layout.cpp:251 wrap-width labels are logged about, not wrapped

ICU_DATA=/usr/share/icu/74.2 can't help: the binary links ICU 76, so it needs icudt76l, not a host's 74.

macOS was fine because Homebrew's libicudata.a carries the full 31.85 MB of data. (That's also why my earlier "regex works" claim was wrong — I had tested only on macOS.)

The fix

Embed icu-data-full's .dat into render with .incbin (cmake/icu-data.S.in, new OSMFLAT_ICU_DATA_FILE option) and register it with udata_setCommonData() at the top of main, before anything touches ICU. That's ICU's supported mechanism for stub-data builds. The embedded copy wins over ICU_DATA and any host ICU, so the host's ICU version doesn't matter — the same approach as the embedded proj.db.

Full data vs. English-only: icu-data-en is ~3 MB but drops non-English locales and the dictionary-based line breakers for Thai, Lao, Khmer, Burmese and CJK. Full data matches what macOS gets and costs ~30 MB of binary, ~12 MB of tarball:

linux-musl before after
binary 31.5 MB 61.9 MB
tarball 14.6 MB 26.4 MB

Why tests missed it, and the new ones

Every test ran inside the build container, where Alpine's ICU data dir was sitting on disk and quietly supplied the data.

  • render_icu_data (new ctest): a style with a case-insensitive .match() filter, a .replace(), and a wrap-width label (test/style-icu.xml). It runs with ICU_DATA pointed at nothing, so a data dir in the build environment can't mask a binary that lacks its own. It also fails on output mapnik only logs. Fails without the fix, passes with it. Passes on macOS too.
  • Release smoke test now runs that style on a bare Alpine image with no ICU data at all. Checked against the published v0.300.0 archive: it fails, printing the ICU error. (Written as an explicit if — ! grep ... can never fail a set -e script, because set -e ignores negated commands.)

Verified (aarch64)

  • Bare alpine:3.22, and ubuntu:24.04 with ICU 74 and ICU_DATA=/usr/share/icu/74.2 — regex filters, case-insensitive regex, .replace(), and wrapped labels all render.
  • Output is byte-identical to macOS (regex 10e35b70…, wrap 14e747b4…).
  • Visually: the regex selects the Baarle areas, .replace('-', ' - ') rewrites the names, and wrap-width breaks "Baarle - Nassau" across two lines.

Release

The v0.300.0 linux archives are affected, so this probably wants a 0.300.1 patch release. This PR doesn't bump the version — say if you'd like it included.

🤖 Generated with Claude Code

Regex filters and label wrapping were broken in every linux static
release (v0.1.0, v0.300.0): Alpine builds ICU with archive packaging, so
its libicudata.a is a 1300-byte stub and the data lives in a separate
icudt76l.dat a relocated binary can't find.

- Any .match()/.replace() filter failed to load: Boost.Regex's ICU
  traits throw 'Could not initialize ICU resources', rethrown by mapnik's
  expression.cpp.
- wrap-width labels logged 'could not create BreakIterator:
  U_MISSING_RESOURCE_ERROR' (mapnik text_layout.cpp) and didn't wrap.
- Pointing ICU_DATA at a host's data can't help: the binary links ICU 76
  and needs icudt76l.

macOS was unaffected: Homebrew's libicudata.a carries the full data.

Fix: embed icu-data-full's .dat into render via .incbin
(cmake/icu-data.S.in, OSMFLAT_ICU_DATA_FILE) and register it with
udata_setCommonData() before anything touches ICU. That copy wins over
ICU_DATA and any host ICU. Full data rather than icu-data-en (~3 MB)
keeps all locales and the dictionary line breakers (Thai, Lao, Khmer,
Burmese, CJK), matching macOS. Cost: ~30 MB binary, ~12 MB tarball.

It slipped through because every test ran inside the build container,
where Alpine's ICU data dir was on hand. The new render_icu_data test
(regex filter + .replace() + wrapped label) runs with ICU_DATA pointed
at nothing and fails on output that only gets logged. Confirmed it fails
without the fix and passes with it.

The release smoke test now runs it on a bare alpine image with no ICU
data. Checked against the published v0.300.0 archive: it fails, printing
the ICU error. (Written with an explicit if, not '! grep': set -e ignores
negated commands, so that form could never fail.)

Verified on aarch64: bare alpine, and ubuntu 24.04 with ICU 74 and
ICU_DATA=/usr/share/icu/74.2 -- all regex/wrap styles render, and output
is byte-identical to macOS.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@boydjohnson
boydjohnson merged commit 530dcf7 into main Sep 18, 2026
4 checks passed
@boydjohnson
boydjohnson deleted the fix/embed-icu-data branch September 18, 2026 22:35
@boydjohnson boydjohnson mentioned this pull request Sep 18, 2026
boydjohnson added a commit that referenced this pull request Sep 18, 2026
Patch release for the linux static render: v0.300.0's linux archives
lack ICU data, so regex filters fail to load and wrap-width labels
don't wrap. Fixed in #4 by embedding the data.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant