Extract the calibration tree builder into a self-contained workspace package

[anth-volk]

Fixes #218

Summary

This moves the calibration tree builder (about 15.5k lines) into a self-contained Bun workspace package, @policyengine/calibration-tree in frontend/packages/calibration-tree/. Its contents:

• the schema 6 types, parsers and serializers;
• the release and staging readers;
• Blob I/O;
• the publisher.

The dashboard and the publisher both import it. It is a refactor only: a golden record taken from the current code, before anything moved, shows the package reproduces every byte the builder writes.

The package has no imports from the rest of frontend/ (no @/ aliases, Next.js or server-only), declares its dependencies, and carries its own tests and typecheck. bun run publish:calibration-tree and .github/workflows/publish-calibration-tree.yml are unchanged. Moving the package into the collector repository and publishing it to npm is out of scope.

Commits

1. Record golden calibration tree outputs from the current builder (0d5ce7b)
   • Red: 2 failed. Both golden suites could not load because no golden record existed yet. A characterization test cannot fail against the code it pins; commit 2 carries the meaningful red-green.
   • Green: 16 passed. These are 12 builder, reader, comparison and manifest cases, plus 4 publisher repair tests.
   • Sensitivity check (not committed): with gzip level 9 changed to 6 in the bundle module, all 8 UK and Belgium golden tests failed. That includes the publisher refusing each repair with "Rebuilt calibration index differs from immutable manifest entry".
   • Gates: bun test 619 passed, 0 failed; tsc --noEmit clean; next build passed. The first build attempt failed to fetch Google Fonts, a network error; the rerun passed.
2. Move the calibration tree builder into a workspace package (e337d3d)
   • Red: 31 failed across 29 files, all for missing modules.
     • 28 suites could not load their module, for example Cannot find module './countries'. The golden suite failed on Cannot find module '@policyengine/calibration-tree/calibration-comparison-blob'.
     • The 3 boundary tests failed because the package manifest did not exist yet.
   • Green: 297 passed, 0 failed (package bun test, run from the package directory).
   • Gates:
     • frontend bun test 622 passed, 0 failed;
     • bun run lint clean (frontend tsc plus the package typecheck);
     • next build passed;
     • bun install --frozen-lockfile clean;
     • bun run publish:calibration-tree without credentials parses its arguments and stops at "BLOB_READ_WRITE_TOKEN is required.";
     • boundary-test sensitivity: adding next/server and @/lib/base-path imports to a package module produced 2 violations (reverted).
3. Run the calibration tree publisher from the package (e78b672)
   • Red: 3 failed. The publisher's unit suite and golden repair suite could not load ./publish-calibration-tree and @policyengine/calibration-tree/publish-calibration-tree, and the boundary test flagged the missing module and export.
   • Green: 15 passed (8 publisher unit tests, 4 golden repairs, 3 boundary tests).
   • Gates:
     • package bun test 309 passed;
     • package typecheck clean;
     • frontend bun test 622 passed, 0 failed;
     • bun run lint clean;
     • next build passed;
     • the workflow's exact invocation, bun run publish:calibration-tree -- --country uk --reconcile-staging --sha <40 hex> --dry-run, parses and stops at the credential check, as does the package's own script;
     • pytest frontend/tests/test_deploy_workflow.py: 12 passed.

Package boundary

The package is the transitive import closure of the publisher: 30 modules and 15,542 lines, plus their tests and fixtures. That closure already imported nothing outside itself except Node built-ins and @vercel/blob. It contains:

Area	Modules
Schema 6 and builder	calibration-tree-artifact, calibration-tree-bundle, calibration-tree-manifest, calibration-tree, calibration-explorer
Comparisons	calibration-comparison-bundle, calibration-comparison-blob, calibration-build-comparison, target-change
Blob I/O	calibration-tree-blob, calibration-release-locator
Inputs	latest-artifact, staging-artifact, staging-contract, countries, reforms, the target readers, loss attribution, label helpers
Publisher	publish-calibration-tree, keeping its command-line entry point

Blob I/O moved into the package. It already sits behind the injectable CalibrationTreeBlobClient, which defaults to @vercel/blob. The plan keeps both Vercel Blob stores at first, and the collector becomes their writer. That writer needs the same upload verification and manifest compare-and-swap, and those are part of the byte-exact contract. Removing the default client would have changed dashboard code paths, so it stays.

What stayed in the dashboard:

• the Next.js routes;
• calibration-tree-http (imports next/server);
• calibration-tree-request, target-change-tree and calibration-prefetch;
• the use-microcosm hook;
• the route tests;
• third-country-conformance.test.ts and its zz-release fixture;
• the reviewed-repository-import test, which spawns app routes.

They import the package through explicit subpath exports, one per module, such as @policyengine/calibration-tree/calibration-tree-bundle. There is no root barrel, so client bundles pull in only what they import.

Location: inside frontend/.

• Vercel builds with frontend as its root directory. Whether files outside it are available is a project setting this repository cannot verify.
• CI and the publisher workflow both run bun install --frozen-lockfile in frontend/.
• Turbopack's root is frontend.

The boundary is enforced by a test and by the package's own tsconfig, not by its directory.

Wiring:

• frontend/package.json gains workspaces: ["packages/*"] and the workspace:* dependency;
• lint also runs the package typecheck, so CI covers it without a workflow change;
• frontend/bunfig.toml sets linker = "hoisted", because Bun otherwise installs a project with workspaces using its isolated linker, which would change the node_modules layout that Next.js and Vercel build against;
• frontend/tsconfig.json excludes packages;
• bun.lock gains the workspace entry.

Enforcement: package-boundary.test.ts requires that:

• package modules import only each other, node: built-ins and declared dependencies (@vercel/blob);
• only test code may use bun:test, package self-references or development dependencies;
• every export resolves to a package module;
• every module is exported.

Identity proof

The golden record was produced from c630c91's code in commit 1, before any move. It is frontend/packages/calibration-tree/src/fixtures/calibration-tree-golden.json; commit 2 moved it at 100% similarity, with no content change.

The inputs reuse the module tests' fixtures:

• US release: the pinned 5,659-target legacy identities, given values and reported loss attribution. Its bundle spans 2 summary shards, 5 detail shards and 4 tiers.
• UK releases: a schema 8 hierarchy release built from the reader tests' targets and GSS geographies, and a release with loss attribution.
• Belgium release: the existing fixture.
• Staging: a US version 1 run, a UK version 2 run, and a UK run read from its staged dataset bundle.
• Comparisons: release to staging for US and UK, built through loadCalibrationComparisonSource over the stored bytes, in both modes.
• Manifest: holding every recorded build.

For every bundle file the record holds the path, the raw JSON SHA-256 and size, the gzip size, and the SHA-256 of the deflate stream. It also holds the build IDs, the index digests (the index file entries), the explorer root responses read back from the stored bytes, one comparison target detail, and the manifest serialization digest. The gzip header is asserted separately, because its operating-system byte is 3 on Linux and 19 on macOS.

publish-calibration-tree-golden.test.ts runs the publisher's own dry-run repair of the recorded US, UK and Belgium releases against a stubbed Hugging Face repository. The manifest entry carries the recorded build ID and index digest, and the publisher refuses any rebuilt index that differs. A further test confirms that a wrong digest is refused.

All 29 modules and their moved tests are byte-identical to the originals except 10 import lines that used the @/ alias, in staging-artifact.ts, latest-artifact.ts and source-label.ts. The publisher differs only in its 8 import lines. countries.ts is a 100% copy (git diff -M -C).

Risks and follow-ups

• countries compatibility re-export. frontend/lib/microcosm/countries.ts re-exports the package's registry. Its only remaining user is a type-only import in lib/server/microcosm/telemetry-collector-client.ts, which this PR deliberately leaves untouched because another change to that file is in flight. Switch that import and delete the re-export afterwards.
• Next.js fetch cache hint. The readers pass next: { revalidate }, which the dashboard's Next.js fetch honors and Bun ignores. A declaration seen only by the package typecheck (types/next-fetch-cache.d.ts) mirrors Next's type. The collector move should replace this with an injected fetch or caching policy.
• Direct environment reads. The package still reads process.env directly (Hugging Face tokens, repository overrides, size limits, the UK local-run directory), exactly as before.
• Country registry location. The registry now lives in the package, and AGENTS.md, CLAUDE.md, README.md and the docs point there. Once the package is published, adding a country becomes a package release plus a dashboard bump. Splitting dashboard-only fields (capabilities, labels) from the builder's registry may be worth doing then.
• Bun version and deflate output. The deflate digests pin the runtime's zlib: Bun bundles its own, while CI and the publisher use unpinned latest Bun. The record was made with Bun 1.4.2 on macOS. A Bun release that changes deflate output would fail these tests. Such a change also matters in production, because the index records gzip sizes and repairs refuse index changes. That risk already existed; these tests now surface it.
• Test time. The golden suites cost about 30 s of CPU, because the US bundle is built by the builder test and again by the publisher repair. They run in the frontend tests job.
• Traced fixtures. next build traces the package's JSON fixtures into the standalone output (52 KB), as it already does for lib/ fixtures. Server code is bundled.
• Unpublished package. The package is private: true until it is published.

Deploy implications

deploy.yml runs after CI succeeds on a push to main, so merging this deploys:

• a commit-named Modal backend (backend/ is unchanged);
• a Vercel production candidate that is verified and then promoted.

The frontend change is import paths plus workspace wiring, with the same runtime behavior. Vercel installs from frontend/, where the package lives, and reads the hoisted-linker setting there; an older Bun ignores that key and is hoisted by default. The publisher workflow is unchanged, and its next run executes the package module through the same npm script.

Environment variables

No variable is added, removed or renamed. The publisher workflow and deploy.yml are unchanged.

Variable	Read by	Provider and wiring	Verified
BLOB_READ_WRITE_TOKEN	publisher	Repository secret, mapped in the publish job's env: and inherited by bun run	Secret exists (name listed)
HF_TOKEN	publisher	Repository secret, publish job env:	Secret exists
POPULACE_UK_STAGING_HF_TOKEN	publisher (UK staging token_env)	Repository secret, publish job env:	Secret exists
CALIBRATION_TREE_BUILD_ENABLED	publish job if:	Repository variable; none is set, so the job runs	No repository variables exist (unset means enabled)
HUGGINGFACE_TOKEN, CALIBRATION_TREE_MAX_RAW_BYTES, CALIBRATION_TREE_MAX_GZIP_BYTES, POPULACE_*_HF_REPO and _REVISION overrides, MICROCOSM_UK_LOCAL_CALIBRATION_DIR	publisher and readers	Optional; not set in the workflow, so defaults apply	Not applicable
HF_TOKEN (dashboard)	readers on Vercel	deploy.yml passes --env HF_TOKEN=$HF_RELEASE_READ_TOKEN from the repository secret	Secret exists
BLOB_READ_WRITE_TOKEN, POPULACE_UK_STAGING_HF_TOKEN (dashboard)	tree routes and staging readers on Vercel	Vercel project environment, managed in Vercel	Unverified: not visible from the repository, and unchanged by this PR

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 15, 2026 8:38pm UTC