Repository navigation
Description
Parent
Refs #15703
Source: PLAN Phase 5, topic ecosystem-specifics
Depends on
- Add the ecosystem-resolution contract, its decision record and the review rule #15706
- Add the whole-page sensitivity-control clause to the four cut-proof rules #15707
Outcome
/testing:write no longer ships per-language property-testing pages; its property guidance applies to every language through the first synced copy of the generic resolution text. This is the pilot that proves the cutting route before any other cut depends on it.
Demo: the per-language property folder is gone, the shared-copy sync check passes, and the pull request body shows the control's predicted outcome (written before the first paid batch), the observed outcome and the compare's verdict line.
Order of work:
- Reproduce first: run the content-dependent cases with and without the plugin; keep only cases below ceiling without it.
- Before any paid batch, write the predicted sensitivity-control outcome into the pull request body. What each outcome means is fixed in advance: if the wrong page moves the score, the suite reads the page, and this cut and every later whole-page cut ships only on non-inferiority of the after version; if it does not move the score, the model ignores the page, that result is the evidence for this cut, and each later whole-page cut still runs its own control.
- Write and freeze the cases from the page lines: a dependent-input scenario, a past-bug scenario, and one TypeScript case expecting no recomputed-derived-value finding. Rewrite the existing Python-reference case to grade the test written; drop the no-read grader of the existing Go case and update its samples.
- Register the first carrier row (the generic text into the testing plugin) in the shared-copies list and regenerate.
- In the property context page: the per-language routing table and the Hypothesis pointer go; the any-language rule becomes the rule for every language through the carrier copy; one line stays naming the adapters'
property_markersand theextendkey. - Delete the three per-language property pages (Python, TypeScript, C#; 87 lines). Edit the matching row of the Trail of Bits upstream record.
- Run the two-version compare, then the sensitivity control.
Run budget: 340 runs (17 testing write cases plus 3 new, on the opus and sonnet aliases, three runs per side for the compare, three on one model for the sensitivity side, plus the with-and-without ceiling check and two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error. Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the record releases it.
If the owner disputes the control outcome as proof, the later deletion items stop until the proof rule is settled.
Decisions
- D7 (interview Q7, Q10): a whole-page deletion is proven with a sensitivity control; every other cut keeps the detection control.
- D1: the pilot runs before the volume work because the proof route is the riskiest unknown.
- Q42: testing is in the starting set of carriers of the generic resolution text.
- Q26: compares run on the
opusandsonnetaliases, three runs per side. - Load-bearing assumption: the sensitivity control gives a usable answer for a whole-page cut. If the recorded outcome is neither predicted outcome, stop the later whole-page cuts and bring the proof rule back to the owner.
Key interfaces
- "
carrier(a plugin holding a synced copy of a contract text)." - "The consumer's ecosystem (language, framework, tools): abstract now. Seam: the ecosystem-resolution contract and the consumer's own records."
- Conventions followed: a cut ships with eval evidence (the skills-and-agents review rules).
Done when
- AC1 (this plugin's share):
/testing:writeships no per-language property page or routing table. - AC7: this whole-page deletion carries its sensitivity-control result, predicted in a pull request body revision dated before the first batch's result file.
- The shared-copy sync check passes with the testing carrier row registered.
- The two-version compare's verdict line is in the pull request body, with this repository's dogfooding step.
- AC10: no new enforcement-placing skill is added here.
Out of scope
- Any other plugin's language pages.
- Carrier rows for other plugins.
Metadata
Metadata
Assignees
Labels
Type
Fields
Work status
Priority
Activity
github-actions commentedon Nov 13, 2026
The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).
github-actions commentedon Nov 13, 2026
Size: Medium
Complexity: Medium
Uncertainty: Known
Flags: none
Blocked by: none
Decision comment, quoted from #15709 (comment) (the later decision comment is the one that counts):
This decision settles the open questions about how the eval runs and whether the control prediction stands, so I'm asking nothing further.
What I understood: remove the per-language property pages (Python, TypeScript, C#) and the routing table from
/testing:write, register the testing carrier row for the generic resolution text, and prove the cut with the compare and sensitivity control. The issue's own listed dependencies (#15706, #15707) are not among the open issues, so I recorded no blocker. The eval batches need a host run, which thehost-runlabel already routes. I found no duplicate and no merged pull request that already fixes this.Automated triage. A maintainer reviews it.
melodic-automation-lanes commentedon Nov 13, 2026
Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (
.github/actions/resolve-config/admission-rules.json).1 remaining item
melodic-automation-lanes commentedon Nov 15, 2026
{"v":1,"actor_id":337595936,"title_sha256":"9166d950a7509418fc60e1ad0ac5d4b3bf7340cb5a39e000b75838f23a58ac05","body_sha256":"3f162cab3eaded09f27146ada3c31875c6323c36ec0e25af1ade33e7a3e67966"}
github-actions commentedon Nov 15, 2026
melodic-automation-lanes commentedon Nov 15, 2026
The work-resolve-issue lane stopped on this issue (
host-only): the run reports the work needs a host run the lane cannot make.It added
host-runand set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.In the run's words:
What the run says it did not do:
Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43719904878
To hand it back to the lane, remove
host-runand set Work status to agent-ready again.kyle-sexton commentedon Nov 15, 2026
kyle-sexton commentedon Nov 15, 2026
Decision recorded.
Owner, 2026-11-15, asked how the eval for this pilot should run (as registered, stage by stage; or the ceiling check only for now; or the compare on sonnet only): "go w/ recommended", which is as registered. Asked whether the posted control prediction should stand as posted: "go w/ recommended", which is yes.
Record (registered before any run; launched by the owner; every batch valid; no usage-limit stop; no rerun; Claude Code 2.1.325; 318 runs, 79.76 USD list price). Before = main at the branch's merge base; after = #16395's head da23783, with the eval cases frozen in their own commit before the cut.
wr-ts-async-property-markeddropped at ceiling (1.000 without the plugin).On sonnet the interval's lower end sits at the margin in the registered reading (runs where the skill loaded); the all-runs reading printed beside it falls just below the margin.
The control gave one of the two predicted outcomes, so the load-bearing assumption (the control gives a usable answer for a whole-page cut) held.