Skip to content

Commit 9dd2359

Browse files
claudeclaude
committed
prereg(perf): step 6 — the training collection preregistration
Fixes the protocol of a training collection that has NOT happened. No clock, no observation, no fitted A_abs or R_rel, no selected N, no width, and no change to the frozen policy or the measurement harness. The owner's platform-scope ruling, applied: platform is NOT a fifth cell axis; the frozen four-part identity (rung, engine, workload, regime) is unchanged platform is an OUTER STRATUM: U_linux and U_windows within a stratum cold and warm are distinct cells and pool per rung across strata observations, medians, envelopes and widths are never pooled, averaged or summed selection the smallest ladder rung in Q_linux intersect Q_windows, where Q_p is the rungs within (1 + G) of that stratum's own best width empty NO_COMMON_N, an orchestration stop and not a fifth reproducibility verdict; no fallback rung, no stricter- platform tie-break, no second G, no second fit Rejected by name, each for a reason the document records: max of the per-stratum picks can select a rung admissible under neither margin; averaged widths let an unstable platform buy a stable one a wider tolerance; one platform as reference puts the other into D7 with no measurement model of its own. scripts/training/scope.py is the combiner, permitted by the ruling and placed OUTSIDE scripts/calibration/ so the step 4 freeze control keeps proving the frozen root grew no new file. It is pure: its only imports are __future__, collections.abc, fractions and the frozen policy, and it holds no constant value. It does NOT re-implement the (1 + G) test. The frozen select_n already reports widths and limit beside its own pick, so Q_p is read off that result and the margin is applied exactly once, inside frozen code. The control proves the reading is real rather than incidental by perturbing the limit the frozen function reported and requiring the answer to move; a combiner that recomputed the margin would ignore that and pass. On NO_COMMON_N the result carries no selected_n key at all, so a caller that reads it without checking the outcome raises KeyError instead of proceeding on a number the rule did not select. A third binding axis joins the two accepted ones: policy_implementation_digest c3068ed7fa88... measurement_harness_digest 562a7f7232da... training_scope_implementation_digest 614bf9efe6ba... computed over every committed *.py under scripts/training/ with the same framing step 4 froze, imported rather than restated. Preregistered: one environment identity per stratum covering the whole ladder; rungs 5, 15, 45 with five full runs each over the complete universe, fifteen runs per platform; four observations per cell from the consecutive pairs, never five convenient ones chosen after the outcome is visible; execution order seeded from c3068ed7fa880a70, the first sixteen hex digits of the step 4 policy digest, which existed and was accepted before this document; exactly one collection with a durable identity and no automatic retry; identity drift invalidates a stratum rather than gluing two epochs into one dataset; and if either stratum invalidates the collection yields no admissible step 8 input, with the survivor barred from becoming a single-stratum fallback model. Seven controls, both platforms. training-prereg-universe computes the 40 cells per stratum from the manifest and the instrument rather than trusting the number written in the artifact. Step 7 remains CLOSED. The holdout remains a separate owner decision and is deliberately not described here. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01CYLNQy6tLXqsV1CbuNqsSb
1 parent eedf6d3 commit 9dd2359

7 files changed

Lines changed: 839 additions & 2 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2959,6 +2959,11 @@ jobs:
29592959
# these controls hand it the committed pairs rather than re-stating them.
29602960
- name: Calibration design constants controls (step 5 ratified values)
29612961
run: python tests/test_calibration_constants.py
2962+
# Step 6 preregisters a training collection that has not happened. These
2963+
# controls prove the protocol is complete and bound, and that the scope
2964+
# combiner reads the frozen selector's own margin instead of copying it.
2965+
- name: Training preregistration controls (step 6 protocol + scope binding)
2966+
run: python tests/test_training_prereg.py
29622967
# Plan mode builds the arms, preflights THOSE bytes, freezes their
29632968
# identities and fixes the execution order — then returns before the
29642969
# timing function is reachable. Linux only: the arms are ELF binaries.

‎docs/evidence/calibration/p022-263a-training-preregistration.json‎

Lines changed: 152 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,152 @@
1+
{
2+
"abort_semantics": {
3+
"durable_record": "every abort writes a structured record naming stratum, rung, run index, cell where applicable, and the condition",
4+
"failed_invocation_or_exit_contract": "invalidates the run",
5+
"identity_drift": "invalidates the stratum",
6+
"missing_cell_or_incomplete_rung": "invalidates the run; a partial rung is not a smaller rung",
7+
"no_automatic_retry": "a refusal is recorded and the collection stops; there is no retry because CI blinked",
8+
"noise_probe_refusal": "invalidates the run that carried it",
9+
"repeat_after_abort": "permitted only by a rule already written here, or by a new owner ruling",
10+
"rss_mechanism_unavailable": "recorded and the affected field is null with a reason, never zero",
11+
"timeout_or_runner_loss": "invalidates the run"
12+
},
13+
"anchor_commit": "eedf6d3ed44ecf7bda69fa509dcd23b45960cf76",
14+
"artifact": "p022-263a-calibration-training-preregistration",
15+
"bindings": {
16+
"design_constants_blob_sha1": "d03c8455cbcaf71446f88aeea0957b8e0c56c880",
17+
"measurement_harness_digest": "562a7f7232dad2f4c79c6adfe0e1e7e25680b4b6f3444d54824bf0405e3c14b3",
18+
"policy_implementation_digest": "c3068ed7fa880a7083866ead25fe8bf65c87889d242d8af1f7eee01582cd5cbf",
19+
"training_scope_implementation_digest": "614bf9efe6ba2bb10e26a251c10d6c4a8fdaa99985d43ea03d76c6774447d25b",
20+
"training_scope_root": "scripts/training/"
21+
},
22+
"collection_protocol": {
23+
"environment_rule": "the whole ladder and all runs of one stratum belong to ONE preregistered environment identity for that stratum; measuring one rung on one runner and another rung on a different runner makes N not the only variable that changed",
24+
"observation_rule": "consecutive run pairs (r1,r2), (r2,r3), (r3,r4), (r4,r5), never a chosen five pairs after the outcome is visible",
25+
"observations_per_cell_per_rung": 4,
26+
"runs_per_rung_per_stratum": 5,
27+
"runs_per_stratum": 15,
28+
"runs_total": 30,
29+
"warmup_discards": 2
30+
},
31+
"design_constants": {
32+
"G": [
33+
1,
34+
10
35+
],
36+
"M": [
37+
2,
38+
1
39+
],
40+
"N_ladder": [
41+
5,
42+
15,
43+
45
44+
],
45+
"R_runs": 5,
46+
"q": [
47+
19,
48+
20
49+
]
50+
},
51+
"exactly_one_collection": {
52+
"identity": "the collection carries a durable identity from its first run",
53+
"outcomes": [
54+
"ADMISSIBLE",
55+
"INVALID"
56+
],
57+
"rule": "exactly one training collection is run; it is not the first satisfactory one of several",
58+
"stratum_failure": "if either stratum invalidates, the cross-platform collection yields no admissible step 8 input; the surviving stratum's numbers remain durable evidence of what happened and must never become a single-stratum fallback model"
59+
},
60+
"forbidden_in_step_6": [
61+
"any clock",
62+
"any new timing or resource observation",
63+
"any fitted A_abs or R_rel",
64+
"any selected N",
65+
"any fitted width",
66+
"any change to the frozen policy or the measurement harness"
67+
],
68+
"future_procedure": {
69+
"no_common_n": "an orchestration stop, not a fifth reproducibility verdict; the frozen four describe cells and run pairs and are untouched. No fallback rung, no stricter-stratum tie-break, no second margin, no second fit",
70+
"note": "described here so it is fixed before data; step 6 produces no numeric result of it",
71+
"rejected_alternatives": {
72+
"averaged_widths": "lets an unstable stratum buy a stable one a wider bound",
73+
"max_of_per_stratum_picks": "can name a rung admissible on neither stratum",
74+
"one_stratum_as_reference": "puts the other into D7 with no measurement model of its own"
75+
},
76+
"steps": [
77+
"per stratum, pool its cells within a rung and fit the envelope by the frozen fitter",
78+
"per stratum, compute each rung's width and the admissible rung set by the frozen selector",
79+
"intersect the strata's admissible sets and take the smallest ladder rung in the intersection",
80+
"if the intersection is empty, stop with NO_COMMON_N"
81+
]
82+
},
83+
"holdout": "not authorised by this preregistration and not described by it; the validation pair is a separate owner decision",
84+
"identity_drift_rule": "any change to a required identity field after collection starts invalidates that stratum's training collection; two epochs are never glued into one dataset",
85+
"order": {
86+
"seed_derivation": "the first 16 hex digits of policy_implementation_digest, frozen at step 4 and owner-accepted before this preregistration existed, so it is not a number that could have been re-rolled until the order looked tidy",
87+
"seed_hex": "c3068ed7fa880a70",
88+
"stratum_order": "linux then windows, fixed",
89+
"within_stratum_order": "ladder rungs ascending; within a rung the five runs are sequential; cell order within a run is the instrument's own fixed order and is not re-shuffled"
90+
},
91+
"ratified_by": "owner",
92+
"run_identity_required_fields": [
93+
"tree_sha",
94+
"tree_dirty",
95+
"python_reference_commit",
96+
"python_reference_tree",
97+
"workload_manifest_sha256",
98+
"harness_digest",
99+
"calibration_repetitions",
100+
"warmup_discards",
101+
"candidate_sha256",
102+
"candidate_bytes",
103+
"provenance.environment",
104+
"provenance.timestamp_utc",
105+
"raw samples per cell"
106+
],
107+
"stratification": {
108+
"across_strata": "observations, medians, envelopes and widths are never pooled, averaged or summed across strata",
109+
"cell_identity": [
110+
"rung",
111+
"engine",
112+
"workload",
113+
"regime"
114+
],
115+
"platform_is_a_cell_axis": false,
116+
"regime_rule": "cold and warm are distinct cells of the frozen four-part identity and pool within a stratum",
117+
"strata": [
118+
"linux",
119+
"windows"
120+
],
121+
"within_stratum": "every cell of U_p pools into one fit per ladder rung, by the frozen fitter unchanged"
122+
},
123+
"universe": {
124+
"applicability_rule": "the universe is exactly the cell set the frozen instrument emits for the six calibration workloads; a rung that does not apply to a workload is absent rather than N/A, and the count is asserted by a control rather than trusted",
125+
"calibration_workloads": [
126+
"cal-facts-tiny",
127+
"cal-facts-small",
128+
"cal-facts-medium",
129+
"cal-facts-refused",
130+
"cal-source-small",
131+
"cal-source-medium"
132+
],
133+
"decisive_workloads": "excluded, and the CALIBRATION_ONLY firewall refuses them structurally",
134+
"engines": [
135+
"python",
136+
"rust"
137+
],
138+
"expected_cells_per_stratum": 40,
139+
"pre_step6_data": "every --calibrate run recorded before this preregistration, including run 2036 and the four committed sizing halves, is excluded from U and remains historical hardware diagnostic only",
140+
"regimes": [
141+
"process-cold",
142+
"warm"
143+
],
144+
"rungs": [
145+
"core-usage",
146+
"core-parse-refused",
147+
"core-full-human",
148+
"core-full-sarif",
149+
"launcher-e2e"
150+
]
151+
}
152+
}

‎docs/notes/p022-263a-calibration-policy-proposal.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,8 @@ Status:
66
STEPS 1-3 COMPLETE.
77
STEP 4 DIGEST FREEZE ACCEPTED.
88
STEP 5 DESIGN CONSTANTS RATIFIED BY THE OWNER AND RECORDED.
9-
NO EMPIRICAL CONSTANT IS FITTED.
9+
STEP 6 TRAINING PREREGISTRATION RECORDED.
10+
NO TRAINING COLLECTION HAS RUN. NO EMPIRICAL CONSTANT IS FITTED.
1011
NO NUMERIC FITTING CORPUS EXISTS.
1112
NO TRAINING OR VALIDATION MEASUREMENT AUTHORISED.
1213
```

‎docs/notes/p022-263a-training-preregistration.md‎

Lines changed: 142 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,142 @@
1+
# P-022 / #263-A — training collection preregistration (step 6)
2+
3+
```text
4+
Status:
5+
TRAINING PREREGISTRATION RECORDED.
6+
NO TRAINING COLLECTION HAS RUN.
7+
NO EMPIRICAL CONSTANT EXISTS.
8+
STEP 7 COLLECTION NOT AUTHORISED.
9+
STEP 8 FIT NOT AUTHORISED.
10+
HOLDOUT NOT AUTHORISED.
11+
```
12+
13+
This document and `docs/evidence/calibration/p022-263a-training-preregistration.json`
14+
fix the protocol of a training collection **that has not happened**. Step 6 contains
15+
no clock, no observation, no fitted `A_abs` or `R_rel`, no selected `N` and no
16+
width. It changes neither the frozen policy nor the measurement harness.
17+
18+
## The gap this step closed, and where it was
19+
20+
The owner's review found that platform had fallen between two levels of the model
21+
rather than out of a report. Ratified §2.1 defines a cell as the four-part
22+
`(rung, engine, workload, regime)` and says *no finer*. §3.3 then pools every cell
23+
of the named universe into one fit per rung, and §3.4 turns that into one width and
24+
one selected `N`. Meanwhile the instrument requires both Linux and Windows, and
25+
§6.5 makes environment part of comparability while recording that a GitHub-hosted
26+
runner has already shown itself not measurement-grade.
27+
28+
Nothing in the ratified text said whether the universe spans platforms. Three
29+
readings all survived every sentence, and they disagree about whether an unstable
30+
machine widens the bound a stable one must then live under. That is a statistical
31+
model decision, so it went to the owner rather than being chosen here.
32+
33+
## The owner's ruling
34+
35+
**Platform is not a fifth cell axis.** The frozen four-part identity is unchanged.
36+
Platform is an **outer training stratum**, and there are exactly two universes,
37+
`U_linux` and `U_windows`.
38+
39+
Within a stratum, cold and warm remain distinct cells and every cell pools into one
40+
fit per rung by the frozen fitter, unchanged. Across strata, observations, medians,
41+
envelopes and widths are **never** pooled, averaged or summed. The result is
42+
`E_p,N` and `W_p(N)` per platform, rather than one envelope inflated by the noisier
43+
runner into a forgiving blanket.
44+
45+
**One global `N` all the same.** Each stratum computes its own admissible set
46+
47+
Q_p = { N in the ladder : W_p(N) <= (1 + G) * min_N W_p(N) }
48+
49+
and the selection is the smallest ladder rung in `Q_linux ∩ Q_windows`. Three
50+
alternatives are rejected by name. `max` of the per-stratum picks looks conservative
51+
for about twenty seconds and can select a rung admissible under neither stratum's
52+
margin. Averaging the widths lets an unstable platform buy a stable one a wider
53+
tolerance with its own instability. Treating one platform as the reference puts the
54+
other into D7 with no measurement model of its own.
55+
56+
**Empty intersection is a stop.** `NO_COMMON_N` is an **orchestration** refusal, not
57+
a fifth reproducibility verdict: the frozen four describe cells and run pairs and
58+
are untouched. There is no fallback rung, no stricter-platform tie-break, no second
59+
`G` and no second fit.
60+
61+
## The combiner, and the one thing it refuses to do
62+
63+
`scripts/training/scope.py` exists because `Q_linux ∩ Q_windows` is code that takes
64+
a statistical decision, and §3.6 requires such code to be written and
65+
provenance-bound **before** the data it will judge. It lives outside
66+
`scripts/calibration/` deliberately: the step 4 control proves the frozen root grew
67+
no new file, and it would stop proving that if this landed inside it.
68+
69+
**It does not re-implement the `(1 + G)` test.** The frozen `select_n` already
70+
computes the margin and reports `widths` and `limit` beside its own pick, so `Q_p`
71+
is read back off that result. A second copy of the margin arithmetic would only ever
72+
prove that the two copies agree, which is the defect this PR has spent thirteen
73+
rounds removing. `training-scope-admissible` proves the reading is real rather than
74+
incidental: it perturbs the `limit` the frozen function reported and requires the
75+
combiner's answer to move. A combiner that recomputed the margin itself would ignore
76+
that and pass.
77+
78+
On `NO_COMMON_N` the result carries **no** `selected_n` key at all, so a caller that
79+
reads it without checking the outcome raises `KeyError` rather than proceeding on a
80+
number the rule did not select.
81+
82+
### A control that could not tell a traceback from a finding, for the seventh time
83+
84+
`training-scope-refusals` was first written with a helper that caught bare
85+
`Exception` and scored that as a refusal. Under the mutation that deletes the
86+
strata-versus-selections check, the combiner then raised `KeyError` reaching for a
87+
stratum that was not there, and the helper read that crash as the guard working.
88+
The mutation came back MISSED, which is the only reason it was found.
89+
90+
This is the same family the ledger already records six times, in a file written
91+
after all six. The helper now returns empty only for a refusal raised **by name**,
92+
and reports anything else as the defect it is. Recorded rather than quietly fixed,
93+
because the interval between knowing this lesson by heart and writing it wrong
94+
again was, this time, about forty minutes.
95+
96+
## Three binding axes from here on
97+
98+
policy_implementation_digest c3068ed7fa88…
99+
measurement_harness_digest 562a7f7232da…
100+
training_scope_implementation_digest 614bf9efe6ba…
101+
102+
The third is new and is computed over every committed `*.py` under
103+
`scripts/training/` with the **same framing** step 4 froze, imported rather than
104+
restated. If the combiner changes after collection starts, the training corpus is
105+
stale, with no discussion about it having been only a small change to an
106+
intersection.
107+
108+
## The collection, fixed in advance
109+
110+
Per stratum, one preregistered environment identity covers the whole ladder: rungs
111+
5, 15 and 45, five full runs each, over the complete universe of that stratum.
112+
Fifteen runs per platform, thirty in total. Measuring one rung on one runner and
113+
another rung on a different one would leave `N` as something other than the only
114+
variable that changed.
115+
116+
Five runs per rung give exactly four observations per cell, from the consecutive
117+
pairs `(r1,r2)`, `(r2,r3)`, `(r3,r4)`, `(r4,r5)` — not five convenient pairs chosen
118+
once the outcome is visible. The universe is the 40 cells the frozen instrument
119+
emits for the six `cal-*` workloads across five rungs, two engines and both regimes;
120+
`training-prereg-universe` computes that from the manifest and the instrument rather
121+
than trusting the number written here.
122+
123+
Execution order and its seed are fixed now. The seed is `c3068ed7fa880a70`, the
124+
first sixteen hex digits of the policy digest the owner accepted at step 4 before
125+
this document existed, so it is not a number that could have been re-rolled until
126+
the order looked tidy.
127+
128+
**Exactly one collection.** Not the first satisfactory one of several. It carries a
129+
durable identity from its first run and ends `ADMISSIBLE` or `INVALID` by rules
130+
written here. There is no automatic retry because CI blinked; a repeat after an
131+
abort needs a rule already in this document or a new owner ruling.
132+
133+
**If either stratum invalidates, the collection yields no admissible step 8 input.**
134+
The surviving stratum's numbers remain durable evidence of what happened, and must
135+
never become a single-stratum fallback model. That would be selection on
136+
survivorship wearing a CI badge.
137+
138+
## What this does not authorise
139+
140+
Not the collection, not the fit, not the holdout. The validation pair remains a
141+
separate owner decision, and this document deliberately does not describe it, so
142+
that a training preregistration cannot quietly authorise a fresh validation run.

‎pyproject.toml‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -39,10 +39,11 @@ files = [
3939
"scripts/round7/runner.py",
4040
"scripts/round7/readout.py",
4141
"scripts/calibration/policy.py",
42+
"scripts/training/scope.py",
4243
"scripts/mutate_campaign.py",
4344
"scripts/shadow_compare.py",
4445
"scripts/render_checkpoint_status.py",
4546
]
46-
mypy_path = ["tests", "scripts", "scripts/round7", "scripts/calibration"]
47+
mypy_path = ["tests", "scripts", "scripts/round7", "scripts/calibration", "scripts/training"]
4748
explicit_package_bases = true
4849
strict = true

0 commit comments

Comments
 (0)