[anth-volk]

Fixes #267

What changes

1. Uploads come from the Hugging Face copy. When a collector record replaces a run's Hugging Face copy in the build monitor's list (mergeHostedDocuments), it now keeps the copy's upload record (withHuggingFaceUploads in build-monitor.ts): the version 2 and later delivery block, or the version 1 failed write check. Refreshing the selected run from the collector (refreshHostedRun) keeps the uploads the list read (keepListedUploads). Uploads are unknown only when neither source records them. Example: uk-frs-calibration-attempt-20261021T195620Z-a858c100 now reports uploaded, 39 of 39, to policyengine/populace-uk-staging.

2. Dry runs form a pipeline of their own. pipelineIdentity now gives run kind dry_run the key <country>:<pipeline>:dry_run and the label " (dry run)", as it already did for smoke. Dry runs get their own group in the run selector and no longer enter full runs' history, forecasts or stage statistics. The rule reads the run kind each run registers, so it applies to every country (the US release registers dry_run too).

• Spine builds needed no change: they register pipeline id uk-frs-spine and run kind spine. Production lists them under GB:uk-frs-spine, apart from calibration runs; a test records this.
• Not done (optional part): linking a spine build to the attempt that read its output. The match would be the spine's spine_h5_creation details.sha256 against the attempt's input_pinning details.dataset_sha256, but no spine run's events could be read to check that the values match (spine runs are not on Hugging Face, and the collector's events need the read credential).

3. A cached collector run list. Build progress and both Candidate routes now read the run list through collectorRunLists in build-runs.ts, one per country on each server instance:

• A list that answered is reused for 15 seconds.
• A request in flight is shared for 5 seconds after it was made. A later call makes its own request: once the Candidate tab has stopped waiting, nothing keeps the request running after the response, and Vercel can suspend the instance with it in flight, so sharing it could hand the later call its overdue timeout error.
• A failed request is dropped, so the next call asks again.
• Every collector request needs the collector's read credential from Secret Manager, and that read (createSecretReader in secret-manager.ts) follows the same rule: a read still in flight is shared for 5 seconds after it began, and a read that answered is reused for 60 seconds after it began. Before, a read in flight was shared for 60 seconds, so a new list request could still share the credential read an old one had left running, and receive its overdue timeout as "Unable to load the telemetry credential from Secret Manager."
• The Candidate tab (loadCollectorTimelines) waits at most 5 seconds for the list. After that it uses the list's latest earlier answer if that answered less than 60 seconds ago (two of the tab's 30-second polls), and otherwise shows the Hugging Face states with the note "The telemetry collector did not answer within 5 seconds, …". The late answer serves the next poll: without the earlier answer, a list that took 5 to 15 seconds was already past its 15 seconds of reuse at the next poll, so a collector that stayed that slow never set the tab's states. The late answer arrives only while the server instance keeps running after the response. Vercel documents that only for work passed to after() (its waitUntil), which this PR does not use, so on an otherwise idle instance the next poll can still show the note.
• Each run's own record is read as before: a running run's documents are fetched on every request.
• Build progress still waits for the list, because without it its run list would drop every collector-only run.
• The cache is per server instance, so on Vercel the Staging routes do not all share one list. Vercel groups Next.js routes into one function only when their settings, including maxDuration, match. staging/run sets 300 and the other three routes 120. On the preview deployment of e103f4f, builds, builds/run and staging/runs are one function and staging/run is another, which keeps its own cache. A Candidate poll therefore makes one list request per country in each of the two functions' instances, and only the Build progress list and the Candidate run list share one.

docs/build-monitor.md documents all three.

Tests (red, then green)

Commit	Before the fix	After
30d451a / 97687fd uploads	2 of 4 new tests fail (build-runs.test.ts: list and refreshed run report unknown)	pass
3221e9d / c7c0bbb dry runs	2 of 3 new tests fail (build-monitor.test.ts: key GB:uk-frs-calibration; the dry run is in the history)	pass
fe76db6 / bf8fded run list	3 of 4 new tests fail (build-runs.test.ts: 3 list requests instead of 1; a new request after 14 s; the Candidate tab waits for the slow list)	pass
ab9b668 / e103f4f earlier list (review)	1 of 1 new test fails (build-runs.test.ts: the poll 30 seconds after a 6-second list shows the "did not answer" note)	pass, with a second test that an answer a minute old no longer stands in
7a98b27 / 47ad374 request left running (second review)	1 of 2 new tests fails (build-runs.test.ts: the next poll shares the first poll's request and shows "could not be read (The operation was aborted due to timeout)")	pass; the other new test checks that a call within the 5 seconds still shares the request
7efea28 / a61441d credential read left running (third review)	1 of 1 new test fails (secret-manager.test.ts: a read 5 seconds after an unanswered one shares it and fails with "Unable to load the telemetry credential from Secret Manager.")	pass; the same test checks that a read within the 5 seconds is shared and an answered read is reused for 60 seconds

Local: bun test for the changed and related files (123 tests in 11 files) passes, and bun run lint (both tsc projects) passes. bun run build was left to CI. After the review commits, 128 tests in the 10 related files pass, and bun run lint passes. After the second review's commits, 130 tests in those files pass, and bun run lint passes. After the third review's commits, 131 tests in those files pass, and bun run lint passes.

Settings

Not changed. The four routes still read the collector through telemetry-collector-client.ts and secret-manager.ts. MICROCOSM_TELEMETRY_COLLECTOR_URL (the GitHub Production variable, passed by deploy.yml to vercel deploy --env, and set in the Vercel project), MICROCOSM_TELEMETRY_READ_SECRET_VERSION and MICROCOSM_TELEMETRY_GOOGLE_IDENTITY_PROVIDER are set in the Vercel project's Production and Preview settings. The Hugging Face reads these routes make (HF_TOKEN, from the GitHub repository secret HF_RELEASE_READ_TOKEN through vercel deploy --env; POPULACE_UK_STAGING_HF_TOKEN, set in the Vercel project) are unchanged too.

Overlaps

• Remove telemetry-service/ now that the collector deploys from its own repo #234 also edits docs/build-monitor.md, in other sections; git merge-tree merges it with this branch without conflicts.
• Read Microcosm UK releases from releases.json and build folders only #241 edits staging-artifact.ts, which this branch does not touch. Read Microcosm UK releases from releases.json and build folders only #241 already conflicts with main in publish-calibration-tree.ts and its golden test, independently of this branch.

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
calibration-diagnostics	Ready	Preview	Oct 27, 2026 11:15am UTC