Run a US build stage on Modal

[MaxGhenis]

Epic #956, acceleration item E. Main had no Modal code: git grep -i modal origin/main finds only prose, and the one earlier app, tools/modal_rebuild_us_sparse.py on modal-sparse-rebuild (48277ea, populace era), was never merged. This PR adds the smallest working path for running a heavy US stage on Modal instead of queueing it on the one 128 GiB machine.

What it adds

• tools/modal_us_stage.py: a Modal app that runs one stage of tools/build_us_acs_local_release.py (materialize, calibrate, qa, finalize, package or all). It also runs an inline runner-smoke tool.

  • The image is a shallow clone of the plan's pushed commit, synced with uv sync --all-packages --extra us --frozen from that tree's own uv.lock, on Python 3.14. The build asserts HEAD equals the commit and that the tree is clean. Each container also proves that the plan's branch contains the commit, because checkout -B would put any name on any commit. It does a commits-only fetch of the branch into a scratch repository, then merge-base --is-ancestor. The check and the run refuse an unverified branch, so the sha and branch the tool records are both real.
  • Inputs come from a content-addressed Modal volume (cas/sha256/<digest>/<name>) or from the Hugging Face Hub at an explicit revision. Each is verified against the plan's sha256 before the tool starts.
  • The tool runs with HF_HUB_OFFLINE=1 and without any credential-looking environment variable. HF_TOKEN from an attached Hub secret is used only by the runner, to download inputs. The receipt lists the names removed, never the values.
  • The tool's state is mirrored to a runs volume one file at a time (temporary name, then rename), so no file there is ever half-written. A receipt lists every file with its sha256, alongside the plan digest, commit, verified branch tip, runner and uv.lock digests, argv, wall time, peak RSS and list-price cost estimates. Before a stage uses a run's state, it verifies the state against the run's latest receipt with strict verify-receipt and refuses any difference.
  • Attempt ledger and lock. Modal restarts a preempted function on the same input, from zero, whatever retries says. Each attempt writes runs/<run_id>/attempts/<stage>-<id>.json when it starts and rewrites it every 120 s until it ends, including hashing and mirroring. It ends with an outcome: receipt, refused (by the lock or budget) or error. A restart charges every unfinished attempt of the same plan to max_wall_seconds. A budget stop writes a FAILED receipt, so relaunching the same plan gets the whole budget again. The ledger is also the run's lock. An attempt refuses to start while an earlier attempt of the run is still writing its record. A recent record gets one 270 s wait to tell a running attempt from one preemption just killed.
  • Opt-in "nonpreemptible": true: runs the heavy or light class on Modal's non-preemptible placement at 3x list price (modal.com/docs/guide/preemption). validate and the receipts price it that way; the check class refuses it.
  • The default mode is a cheap check. It covers the image, the clone and branch, the environment, the pinned tool's own _parse_args on the built argv, and the input digests. It also covers the run's state against its latest receipt, the attempts charged to the budget, and any attempt that may still be running. --run executes the stage and fails unless the receipt says COMPLETED.

• tools/modal_us_stage_plan.py: the pure, stdlib-only half. It validates plans and refuses:

  • loose pins: short shas, missing digests, and values that match only with a trailing newline (every pattern uses fullmatch);
  • flags the runner owns;
  • non-allowlisted options or environment, including credential-looking names under an allowed prefix;
  • non-string tool or stage;
  • path tricks.

  It also builds the argv, sets resource classes from the measured Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 peaks, handles hashing, mirroring, receipts and the ledger's charging and lock logic, and enforces a max_wall_seconds budget. Its CLI has validate, digest (sha256 plus the exact modal volume put line) and verify-receipt.

• packages/microcosm-build/tests/test_us_modal_stage_plan_tool.py: 129 tests. They cover argv, receipts, refusals, budget, ledger charging and lock, atomic mirroring, pulled-state verification, env stripping and non-preemptible pricing. The app's helpers are imported against a stub modal module, so CI runs them without the Modal client. The branch check runs against a local git remote. CI runs the file in the rest and us-am lanes (tools/ci_test_groups.py --verify ok).

• docs/us-modal-stage-runbook.md, plus three plans:

  • docs/us-modal-stage-example-plan.json, a replay of Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 materialize at cadaf418;
  • docs/us-modal-stage-smoke-plan.json;
  • docs/us-modal-stage-acceptance-20260923-plan.json, the 23 September state-surface materialize at 767312d6.

  The acceptance plan launched with four peak-limit env variables that do not reach materialize at 767312d6. They were removed afterwards, and the runbook records that the attempt ran the earlier version (plan sha256 5ec595c7…, now d1341ec0…).

Nothing here uploads to the Hub, touches latest.json or notifies anyone. Publication stays tools/publish_release.sh.

History note: three commits were made in a worktree shared with a second session. e70b231 also carries that session's nonpreemptible code and d0bab32 its first test, though neither message says so; 9fa2b15 and 5d5c6e9 describe them. They were pushed before this was caught, so the history was not rewritten. e70b231 alone fails one test that d0bab32 updates.

Running a stage

python3 tools/modal_us_stage_plan.py digest <files…>        # sha256 + the modal volume put lines
python3 tools/modal_us_stage_plan.py validate plan.json      # argv, class, estimate
MICROCOSM_MODAL_PLAN=plan.json modal run tools/modal_us_stage.py                  # check
MICROCOSM_MODAL_PLAN=plan.json modal run --detach tools/modal_us_stage.py --run   # run
modal volume get microcosm-us-stage-runs runs/<run_id> ./modal-runs/
python3 tools/modal_us_stage_plan.py verify-receipt ./modal-runs/<run_id>/receipts/<stage>-<utc>.json --state-root ./modal-runs/<run_id>/state

Sizing and cost

The classes are sized from the #974 measured peaks (totals SOI surface):

Stage	Measured locally	Class
materialize	74.8 GB, 5,067 s wall, single-threaded	heavy: 4 cores, 128 GiB, 8 h
calibrate	67.6 GB, 415 s	heavy
qa, finalize, package	22–23 GB	light: 2 cores, 48 GiB, 4 h

On the state SOI surface, the 23 September build measured materialize locally at a 77.9 GB peak and 5,604.5 s of wall. On Modal the same stage held 15 to 24 GB more RSS at the same chunk. It ran 3.3 to 7.6 times slower per chunk, and 3.93 times slower over chunks 1 to 29 (see below). Modal wall time and cost are therefore well above these local figures.

Modal list prices, read from modal.com/pricing on 22 September: $0.0000131 per core-second and $0.00000222 per GiB-second, billed on the higher of request and use. At those prices the heavy class costs about $1.21 an hour, or $3.63 non-preemptible. Materialize comes to about $1.71 at the local wall time and at most about $9.70 at the 8-hour timeout; max_wall_seconds can cap it lower.

The acceptance run was preempted twice in about 2.97 hours of running, about 0.67 preemptions an hour. At that rate a preemptible attempt longer than about an hour is more likely than not to be cut short, and each cut restarts it from zero. Non-preemptible placement pays for itself in expected cost from about 2.8 hours. The runbook therefore recommends "nonpreemptible": true for heavy stages expected to run longer than about an hour, with the arithmetic. Two events is a small sample, so the runbook presents the numbers as an order of magnitude.

What ran on Modal (policyengine workspace, 22 September)

• Volumes. Created microcosm-us-stage-inputs and microcosm-us-stage-runs, and uploaded three small Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 inputs: the ladder (447 KB), the staging summary (498 KB) and the chronicle_us_b571381 feed (165 MB).
• Check of the Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 replay plan. The image built from cadaf418. The synced environment has policyengine-us 2.2.1 and policyengine-core 3.32.5, the same versions Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 records, and the tree was clean. The pinned tool parsed the argv, and the three inputs were verified on the volume. The only problem reported was the refusal of staging_h5, which was not uploaded (10.7 GB).
• Smoke run. runner-smoke completed in 7 s. It verified two volume inputs and one public Hub file (policyengine/populace-us@85a1ccb0…/latest.json), wrote its state file and receipt, and verify-receipt --strict passed on the fetched state.

Total compute was one image build and a few minutes on 2-core, 8 GiB containers. The changes since 9fa2b15 (branch verification, the lock, atomic mirroring, pulled-state verification, env stripping) have been tested locally but have not yet run on Modal.

Acceptance attempt on Modal (23 September)

docs/us-modal-stage-acceptance-20260923-plan.json ran the overnight state-surface materialize (commit 767312d6) on Modal next to the local run, from the same inputs. It ran as app ap-mzp14wEyVLIFlYX5qQQxFZ on runner commit 1d80287af, before the attempt ledger existed. The 10.7 GB staging H5 went up by digest in 255 s. The check passed: clean clone, policyengine-us 2.2.1 and core 3.32.5 as local, argv parsed, and all four inputs verified on the volume.

• Loading the staging data took 280.6 s and 254.6 s on Modal against 79.3 s locally.
• Per chunk of 20,000 households, Modal took 144.0 to 354.5 s against 36.8 to 80.1 s locally (mean 66.3 s): 3.3 to 7.6 times slower. Over chunks 1 to 29 the totals were 6,832 s against 1,737 s, 3.93 times.
• Modal ran gVisor (kernel 4.19.0, x86_64, 4 cores visible) with Python 3.14.2, against 3.14.4 locally. The tool used about 97% of one core, so it was bound to a single core, not throttled. RSS after chunk 1 was 66.3 GB on Modal and 51.0 GB locally.
• Modal preempted the run twice, each time restarting it from zero:
  • attempt 1 after 54 minutes (at chunk 12 of 80);
  • attempt 2 after 2 hours 4 minutes (at chunk 29, peak RSS 74.48 GB).
• Attempt 3 was stopped 34 s in by modal app stop at 05:02:22 EDT, because a third attempt would have taken the total past $10. The run left no receipt and no state. It cost about $3.62 at list price, per Modal's workspace billing report.
• The local run completed: 4,459 targets, 1,588,854 households, targets_sha256 d843209746bdcf96a831fa90d6fa2fe7aa2969fa32290db35afa9889f7c5d377, 5,604.5 s wall, 77.94 GB peak. Its checkpoint digests are in the runbook for the next comparison.

So the heavy acceptance replay is still open. At the observed pace, one uninterrupted Modal materialize on this surface takes about 5.3 to 6.9 hours, against 1.6 hours locally. That is $6.40 to $8.40 preemptible if nothing preempts it, or $19 to $25 non-preemptible. The path works end to end up to the engine pass, but it does not yet make this stage faster or cheaper. Fanning the 80 chunks out across containers might, if the chunks are independent, which has not been checked.

Not yet covered

• The heavy acceptance replay. Neither the Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 materialize replay nor the 23 September state-surface run has finished on Modal, so no Modal run_identity, targets.json or checkpoint digests have been compared with a local run. The Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 replay should reproduce targets_sha256 0f447ce9… and 1,247 targets. At the Modal pace above it needs about 5.5 hours: about $6.70 preemptible, or about $20 non-preemptible, which the runbook recommends for a run that long.
• The graph-native executor (Consolidate the US survey graph build and enrichment pipeline #893). It is not registered. It lives on Consolidate the US survey graph build and enrichment pipeline #893's branch and needs a ToolSpec and a class sized from a full-scale measured peak.
• The base stage (tools/build_us_puf_support_base.py). It is not registered, because it takes the processed and restricted PUF files, and whether those may go on Modal needs a decision first.
• The staging stage. It takes a directory input, which the plan format does not support yet.
• Work lost to preemption. A preempted attempt's tool work is lost. State is mirrored only when the tool exits, and materialize runs as one long single-core loop with no resume point. The ledger bounds the cost across restarts, and non-preemptible placement avoids them. What Modal does with a container killed for memory has not been observed.
• The lock is best effort. Modal volumes have no atomic lock, so two attempts started within one volume commit of each other can both pass. Run one stage of a run at a time.

Test plan

• pytest packages/microcosm-build/tests/test_us_modal_stage_plan_tool.py: 129 passed at f8e8538 (run on an isolated git archive extract of that commit)
• uv run ruff check . and ruff format --check on the touched files
• tools/ci_test_groups.py --verify
• Modal check (image build and input verification) and the Modal smoke run, with the receipt verified locally (22 September, before the changes after 9fa2b15)
• Modal check and smoke run on the current head (branch verification, lock, pulled-state verification on Modal)
• Heavy Record the full-scale local ACS hours rebuild of 2026-09-22 (#765 build evidence) #974 materialize replay on Modal, matching targets_sha256
• 23 September state-surface materialize on Modal: the check passed, but the run was preempted twice and stopped at about $3.62 without a receipt, so there are no Modal outputs to compare yet

🤖 Generated with Claude Code