Split the CI Test job into shards and cut its runtime

[MaxGhenis]

Fixes #2099

What the hour was spent on

Measured from the log of run 38075384588 (10 October, Test passed in 59.6 minutes):

• YAML tests: 2.6 minutes (2,946 tests, serial).
• pytest: 55.8 minutes on four pytest-xdist workers, each busy for about 55 minutes: 220 worker-minutes for 3,532 tests.
• 51 *_properties.py files take 74% of the worker time. No test dominates: the slowest takes 3.5%.

Change

• Split the Test job into a YAML job, three shards of the Python tests (--shard K/N, defined in policyengine_uk/tests/conftest.py), and a final Test job. The final job fails unless both test jobs passed and .github/check_test_shards.py finds that every collected test ran in exactly one shard and none failed.
• Run the final Test job on ubuntu-24.04-arm, the shards' runners. On 10 October it waited 32 minutes for an ubuntu-latest runner after the shards had finished. On this PR's second run it started 4 seconds after the last shard.
• Measure coverage with sys.monitoring (COVERAGE_CORE=sysmon). On one property file this reported the same 12,262 of 18,401 lines, at 6% CPU overhead instead of 33%.

Measured result

Run	YAML	Python shards	Final Test	Wall time
Before: main, run 38075384588 (one job)	2.6 min	55.8 min (one job)	—	59.6 min
This PR, first run 38085920460 (ubuntu-latest final job)	4.2 min	14.8 / 19.9 / 15.2 min	5 s after a 31.8 min queue	56.3 min
This PR, rebased run 38149483980 (ARM final job)	4.3 min	18.8 / 23.8 / 15.5 min	5 s, started 4 s after the shards	24.0 min

The rebased run's shard 2 is red because of a defect already on main, not this change. test_dataframe_dataset.py::test_an_exported_frame_loads_back_and_exports_itself fails on main too (run 38143054188 at 7343f94), and Hypothesis spent over 5 minutes shrinking it, which is why that shard took 23.8 minutes. Its fix is being handled separately. Once it lands I'll rebase this PR and re-run.

Invariants (tested in policyengine_uk/tests/code_health/test_ci_test_shards.py)

1. shard_of(nodeid, N) is in 1..N and depends only on its arguments (SHA-256, not Python's per-process hash), so every runner and xdist worker agrees.
2. For any collection and any N, the N shards are disjoint and their union is the whole collection, so a test added later lands in exactly one shard.
3. parse_shard accepts exactly K/N with 1 <= K <= N.
4. junit_key gives a node id the name pytest gives it in JUnit XML (differential test against pytest's own mangle_test_address).
5. The check passes when every collected test ran in exactly one shard and none failed, and names any test that is missing, repeated, not collected or failed (property).
6. With real pytest runs, with and without xdist, the listing and the shards' JUnit files pass the check, and fail it when a shard's file is left out or repeated.

Review

GPT-6.1 Sol reviewed head 841d2a8 and approved it with no blocking findings (~/reviews/budget-ci-w2/review-2269-841d2a889.md). Its two minor findings are handled in the follow-up from workstream W2 of the Autumn Budget CI plan, which moves this check off JUnit:

• There is no direct test that a <failure> or <error> element counts as failed.
• JUnit naming can map two distinct node ids to one name, for example a/b/test_c.py::t and a/b.test_c.py::t.

That follow-up also stops the check from accepting skips and expected failures that no reviewer has accepted. This PR is unchanged in what it accepts.

Left for later

• A gc.freeze() plus GC-threshold change cut CPU on one property file from 77 to 61 seconds. It isn't in this PR, so the shard split can land first.
• Codecov uploads fail on PR and main runs ("Token required because branch is protected"), so the coverage data is computed and discarded.

axiom: n/a: CI infrastructure

🤖 Generated with Claude Code

[MaxGhenis]

Hand-off to the UK hub from W2 of the Autumn Budget CI plan (session local_6c1ab389). I finished this PR for its original session, which the usage limit cut off.

• Head: 841d2a8: the original commit rebased on main 6d89338, plus one commit that moves the final Test job onto the shards' ARM runners.
• Review: GPT-6.1 Sol, VERDICT: APPROVE at this head, no blocking findings. Its two minor findings are handled in Publish only the policyengine-uk release files whose checks passed #2306. Review: ~/reviews/budget-ci-w2/review-2269-841d2a889.md.
• CI: run 38149483980 passes everything except Test (Python 2/3), which fails on test_dataframe_dataset.py::test_an_exported_frame_loads_back_and_exports_itself. That failure is already on main (run 38143054188 at 7343f94), from the DataFrame round-trip work, not this PR. Once main's fix lands, update this branch and re-run; nothing else is owed here.
• Depends on: main's dataframe round-trip fix. Publish only the policyengine-uk release files whose checks passed #2306 (W2 release integrity) is stacked on this PR, so please land this first.
• Follow-ups owed: W2 (session local_6c1ab389) will comment on CI Test job takes 4-6 hours and has hit the 6-hour limit #2099 and PR Test job times out at 60 minutes: the suite now takes about an hour #2257 with this PR and the measured result once it merges. The gc.freeze() measurement is a separate chip; Codecov's missing token is in W2's admin request to the coordinating session.