You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Accept any recorded or inferred ecosystem id and any mutation tool value - part 4 of 4 #16063
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #16355, #16356, #16357, #16358
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).
The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.
In the run's words:
host-only: every done-when item of this part needs a run this lane cannot make. (1) The full /testing:audit scan of this repository, before and after the #15710 change, needs the testing plugin's audit skill, and only the work-items plugin is loaded in this lane. (2) The two-version compare of the skill-text changes is an eval batch (opus and sonnet aliases, three runs per side, inside the 850-run budget) that #15710's Method says runs in the one local eval-runner session in the main checkout, and its record goes in the pull request body. AC7 and AC10 are statements about the sibling parts' work and need no edit. This part builds nothing, so the lane has nothing to commit.
What the run says it did not do:
Full /testing:audit scan of this repository before and after, showing zero changed findings (dogfood): needs the testing plugin's audit skill, which this lane does not load; a host run.
Two-version compare (opus and sonnet, three runs per side) of the skill-text changes, with its record in the pull request body: needs the local eval-runner session in the main checkout; a host run.
Each touched script's test file passes: the scripts were touched in sibling parts, and their suites run in the PR's CI and the host run.
Hub session (GitHub Support). Owner decision Q159, answered in the hub session on 2026-11-14 (US Eastern), by 2026-11-15T01:43:16Z.
Question. This issue's two-version compare was registered and not launched. Preparation found two things the scoping had missed: 24 case-and-model pairs have a recorded pass rate between 0.2 and 0.8, where 3 runs per side can show a false case drop that would read as a loss with no rerun allowed; and 12 mutation-testing setup cases load skill text that #16229 changed, each with a trace-judged grader. #15710's Method sets 3 runs per side and an 850-run budget. How large should the compare be?
Options shown.
(a) As the Method says: 3 runs per side, no setup cases: 792 runs, about 232 USD list price, 9.6 h.
(b) Recommended: 3 runs per side with the 24 unstable pairs at 6, no setup cases: 936 runs, about 288 USD, 10.3 h. The setup-skill text is recorded as not compared, with a follow-up issue.
(c) Option (b) plus the 12 setup cases: 1,080 runs, about 452 USD, 16.6 h.
(d) 6 runs per side for every case, with the setup cases: 1,872 runs, about 793 USD, 23 h.
Answer (owner, verbatim). "go w/ recommendations" That is option (b).
The 12 mutation-testing setup cases are not run; the record says the setup-skill text shipped without a compare, and a follow-up issue carries that compare.
Two cases that load no changed text (ck-notice-reads-checker, confirm-build-question-no-trigger) are left out and named in the record.
The record is posted here as a comment with the six verdicts, whatever they show, and no batch is rerun.
#16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)
Computed by the registered result.py and noise-report.py (--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).
Versions. toolchain and verification: BEFORE 7645e1c6ac, AFTER d87f47101e (#16228). mutation-testing: BEFORE 59d6f9d110, AFTER 08fcb9062e (#16229). Six plugin copies built with git archive; diff -rq showed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.
Not compared. The mutation-testing setup-skill text (setup/SKILL.md:73-76,96-98, setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12 se-* cases were not run. A follow-up issue carries them.
Verdicts
Compare
Under amendments B, C, D
By the registration's letter
toolchain sonnet
NOT MET: -0.01, 95% interval -0.06 to +0.04, not shown non-inferior
NOT MET (C's letter also drops ck-consumer-config-wins: -0.01, -0.06 to +0.05)
toolchain opus
MET: +0.00, -0.03 to +0.03, non-inferior (see the batch-225 note)
NOT MET: batch 225 void, ck-clean-tree-branch-diff dropped (18 cases: +0.01, -0.02 to +0.04)
verification sonnet
NOT MET: +0.03, -0.01 to +0.07 with case drop confirm-proof-level-user-strict-wins 0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0)
NOT MET: batches 39 and 40 void
verification opus
MET: +0.04, +0.00 to +0.07
same
mutation-testing sonnet
NOT MET: +0.04, -0.28 to +0.35, not shown non-inferior (3 cases)
same
mutation-testing opus
MET: +0.00, +0.00 to +0.00
same
Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.
Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep changed-read); BEFORE leaks 0.
New cases (by design, AFTER only): ck-recorded-filter-runs and ln-inferred-filter meet the bar on both models; every grader passed 3 of 3.
Amendments
B (owner, Q167 (a)): run 2 of batches 39 and 40 (confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.
C (hub, written after the event; confirmed by the owner, Q177 (a)): the batch-95 launch (sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, both error: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.
D (owner, Q173 (a)): run 3 of batch 225 (ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.
How B and D were applied: a result-step helper copied each affected file with only the errored run removed (it refuses a run with no error); unchanged copies of exclude.py (--runs 2 --min 2) and result.py then ran; all three files read VALID. Each exclusion was applied on the side where it occurred.
Run-level exclusions for sandbox denials (D299; each denied call read from its trace)
Batch
Compare, side
Case, run
Denied call
58
verification sonnet AFTER
confirm-prep-light-does-not-block r1
Glob **/setup-apply.mjs at the AFTER copy
97
toolchain sonnet BEFORE
ck-detects-changed-ecosystems r3
Read context/dotnet.md, context/python.md
106
toolchain sonnet AFTER
ck-skips-lint-only-surfaces r2
Read context/bash.md
162
verification opus BEFORE
confirm-intent-from-plan-brief r4
Bash sed -n ... reference/config.md
168
verification opus BEFORE
confirm-light-skips-verifier r4
Glob of the copy root (a Read of a mistyped path not counted at the copy)
189
verification opus AFTER
confirm-prep-light-does-not-block r2, r5
Glob setup-apply.mjs; Bash sed ... reference/config.md
210
verification opus BEFORE
confirm-retired-assumption-confirmed r3
Grep of the copy root
No case fell below its minimum (confirm-prep-light-does-not-block opus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.
Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)
toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.
toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).
verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.
verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.
mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.
Trace checks (tracegrep.py plus a bounded grep over all 936 kept traces): no is not a valid shim or Shell command permission check failed hits. The only command not found hits are cargo in ln-inferred-filter (sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.
Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.
Cost and time (file-level costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)
Model
Spent
Judge share
Expected
Worst
sonnet
63.70 USD
12.71
86.25
288.34
opus
153.50 USD
18.41
201.49
465.07
total
217.20 USD, plus 0.22 for the killed batch-95 attempt
287.74
753.41
Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.
Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.
Follow-ups.#16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
Activity
github-actions commentedon Nov 14, 2026
Size: Small
Complexity: Low
Uncertainty: Known
Flags: none
Blocked by: #16355, #16356, #16357, #16358
#16063 (comment)
This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.
Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 14, 2026
Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (
host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).melodic-automation-lanes commentedon Nov 14, 2026
{"v":1,"actor_id":337595936,"title_sha256":"c403775406470bfeca5bde4219ef8256e01fafcb209620635796475088871d96","body_sha256":"b9ff2193f8cd973eaf3a3c019a49d8cb678efe1db69545ed6ee720f2582c25e0"}
2 remaining items
melodic-automation-lanes commentedon Nov 14, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43650781221
To hand it back to the lane, remove
host-runand set Work status to agent-ready again.kyle-sexton commentedon Nov 15, 2026
Decision recorded.
Hub session (GitHub Support). Owner decision Q159, answered in the hub session on 2026-11-14 (US Eastern), by 2026-11-15T01:43:16Z.
Question. This issue's two-version compare was registered and not launched. Preparation found two things the scoping had missed: 24 case-and-model pairs have a recorded pass rate between 0.2 and 0.8, where 3 runs per side can show a false case drop that would read as a loss with no rerun allowed; and 12 mutation-testing setup cases load skill text that #16229 changed, each with a trace-judged grader. #15710's Method sets 3 runs per side and an 850-run budget. How large should the compare be?
Options shown.
Answer (owner, verbatim). "go w/ recommendations" That is option (b).
What it allows, as acceptance criteria.
ck-notice-reads-checker,confirm-build-question-no-trigger) are left out and named in the record.kyle-sexton commentedon Nov 15, 2026
#16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)
Computed by the registered
result.pyandnoise-report.py(--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).Versions. toolchain and verification: BEFORE
7645e1c6ac, AFTERd87f47101e(#16228). mutation-testing: BEFORE59d6f9d110, AFTER08fcb9062e(#16229). Six plugin copies built withgit archive;diff -rqshowed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.Not compared. The mutation-testing setup-skill text (
setup/SKILL.md:73-76,96-98,setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12se-*cases were not run. A follow-up issue carries them.Verdicts
-0.01, 95% interval -0.06 to +0.04, not shown non-inferiorck-consumer-config-wins:-0.01, -0.06 to +0.05)+0.00, -0.03 to +0.03, non-inferior (see the batch-225 note)ck-clean-tree-branch-diffdropped (18 cases:+0.01, -0.02 to +0.04)+0.03, -0.01 to +0.07with case dropconfirm-proof-level-user-strict-wins0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0)+0.04, +0.00 to +0.07+0.04, -0.28 to +0.35, not shown non-inferior (3 cases)+0.00, +0.00 to +0.00Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.
Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep
changed-read); BEFORE leaks 0.New cases (by design, AFTER only):
ck-recorded-filter-runsandln-inferred-filtermeet the bar on both models; every grader passed 3 of 3.Amendments
confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, botherror: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.exclude.py(--runs 2 --min 2) andresult.pythen ran; all three files read VALID. Each exclusion was applied on the side where it occurred.Run-level exclusions for sandbox denials (D299; each denied call read from its trace)
**/setup-apply.mjsat the AFTER copycontext/dotnet.md,context/python.mdcontext/bash.mdsed -n ... reference/config.mdsetup-apply.mjs; Bashsed ... reference/config.mdNo case fell below its minimum (
confirm-prep-light-does-not-blockopus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)
toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.
toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).
verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.
verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.
mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.
Trace checks (
tracegrep.pyplus a bounded grep over all 936 kept traces): nois not a valid shimorShell command permission check failedhits. The onlycommand not foundhits arecargoinln-inferred-filter(sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.
Cost and time (file-level
costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.
Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.
Follow-ups. #16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).
session: GitHub Support
kyle-sexton commentedon Nov 15, 2026
Decision recorded.
Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.
Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.
Acceptance criteria
session: GitHub Support