Skip to content

Cut the per-language property pages from testing:write as the pilot #15709

Description

@kyle-sexton

Parent

Refs #15703
Source: PLAN Phase 5, topic ecosystem-specifics

Depends on

Outcome

/testing:write no longer ships per-language property-testing pages; its property guidance applies to every language through the first synced copy of the generic resolution text. This is the pilot that proves the cutting route before any other cut depends on it.

Demo: the per-language property folder is gone, the shared-copy sync check passes, and the pull request body shows the control's predicted outcome (written before the first paid batch), the observed outcome and the compare's verdict line.

Order of work:

  1. Reproduce first: run the content-dependent cases with and without the plugin; keep only cases below ceiling without it.
  2. Before any paid batch, write the predicted sensitivity-control outcome into the pull request body. What each outcome means is fixed in advance: if the wrong page moves the score, the suite reads the page, and this cut and every later whole-page cut ships only on non-inferiority of the after version; if it does not move the score, the model ignores the page, that result is the evidence for this cut, and each later whole-page cut still runs its own control.
  3. Write and freeze the cases from the page lines: a dependent-input scenario, a past-bug scenario, and one TypeScript case expecting no recomputed-derived-value finding. Rewrite the existing Python-reference case to grade the test written; drop the no-read grader of the existing Go case and update its samples.
  4. Register the first carrier row (the generic text into the testing plugin) in the shared-copies list and regenerate.
  5. In the property context page: the per-language routing table and the Hypothesis pointer go; the any-language rule becomes the rule for every language through the carrier copy; one line stays naming the adapters' property_markers and the extend key.
  6. Delete the three per-language property pages (Python, TypeScript, C#; 87 lines). Edit the matching row of the Trail of Bits upstream record.
  7. Run the two-version compare, then the sensitivity control.

Run budget: 340 runs (17 testing write cases plus 3 new, on the opus and sonnet aliases, three runs per side for the compare, three on one model for the sensitivity side, plus the with-and-without ceiling check and two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error. Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the record releases it.

If the owner disputes the control outcome as proof, the later deletion items stop until the proof rule is settled.

Decisions

  • D7 (interview Q7, Q10): a whole-page deletion is proven with a sensitivity control; every other cut keeps the detection control.
  • D1: the pilot runs before the volume work because the proof route is the riskiest unknown.
  • Q42: testing is in the starting set of carriers of the generic resolution text.
  • Q26: compares run on the opus and sonnet aliases, three runs per side.
  • Load-bearing assumption: the sensitivity control gives a usable answer for a whole-page cut. If the recorded outcome is neither predicted outcome, stop the later whole-page cuts and bring the proof rule back to the owner.

Key interfaces

  • "carrier (a plugin holding a synced copy of a contract text)."
  • "The consumer's ecosystem (language, framework, tools): abstract now. Seam: the ecosystem-resolution contract and the consumer's own records."
  • Conventions followed: a cut ships with eval evidence (the skills-and-agents review rules).

Done when

  • AC1 (this plugin's share): /testing:write ships no per-language property page or routing table.
  • AC7: this whole-page deletion carries its sensitivity-control result, predicted in a pull request body revision dated before the first batch's result file.
  • The shared-copy sync check passes with the testing carrier row registered.
  • The two-version compare's verdict line is in the pull request body, with this repository's dogfooding step.
  • AC10: no new enforcement-placing skill is added here.

Out of scope

  • Any other plugin's language pages.
  • Carrier rows for other plugins.

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    The triage sweep dispatched intake triage for this issue (dispatch 1; 2 runs that reach intake, or 6 dispatches, hand it to a maintainer).

  4. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Medium
    Complexity: Medium
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision comment, quoted from #15709 (comment) (the later decision comment is the one that counts):

    Decision recorded.

    Owner, 2026-11-15, asked how the eval for this pilot should run (as registered, stage by stage; or the ceiling check only for now; or the compare on sonnet only): "go w/ recommended", which is as registered. Asked whether the posted control prediction should stand as posted: "go w/ recommended", which is yes.

    [quote truncated; the full decision is at the link]

    This decision settles the open questions about how the eval runs and whether the control prediction stands, so I'm asking nothing further.

    What I understood: remove the per-language property pages (Python, TypeScript, C#) and the routing table from /testing:write, register the testing carrier row for the generic resolution text, and prove the cut with the compare and sensitivity control. The issue's own listed dependencies (#15706, #15707) are not among the open issues, so I recorded no blocker. The eval batches need a host run, which the host-run label already routes. I found no duplicate and no merged pull request that already fixes this.

    Automated triage. A maintainer reviews it.

  5. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    Not admitted to the issue lane: it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  6. 1 remaining item

  7. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"9166d950a7509418fc60e1ad0ac5d4b3bf7340cb5a39e000b75838f23a58ac05","body_sha256":"3f162cab3eaded09f27146ada3c31875c6323c36ec0e25af1ade33e7a3e67966"}

  8. github-actions commented on Nov 15, 2026

    @github-actions
    Contributor
    No description provided.
  9. added
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
  10. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: the issue's order of work starts with paid eval batches before any cut: a with/without-plugin ceiling check, a predicted sensitivity-control outcome written into the PR body before the first batch result, the opus/sonnet two-version compare and the sensitivity control (340-run budget). It says these batches run in the one local eval-runner session in the main checkout. This lane can't run eval batches or edit the PR body before a batch, and docs/review/skills-and-agents.md says a cut to an existing skill ships only with eval evidence of no loss. So the per-language property pages can't be deleted here without breaking the issue's stated proof route (AC7) and the repository's review rule.

    What the run says it did not do:

    • All of the issue's work: the eval ceiling check, the predicted sensitivity-control outcome, the new and rewritten eval cases, the testing carrier row in scripts/shared-copies.txt, the property.md edits, deleting the three per-language pages, the Trail of Bits upstream record edit, the two-version…

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43719904878

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  11. self-assigned this
    on Nov 15, 2026
  12. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    MemberAuthor
    No description provided.
  13. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded.

    Owner, 2026-11-15, asked how the eval for this pilot should run (as registered, stage by stage; or the ceiling check only for now; or the compare on sonnet only): "go w/ recommended", which is as registered. Asked whether the posted control prediction should stand as posted: "go w/ recommended", which is yes.

    Record (registered before any run; launched by the owner; every batch valid; no usage-limit stop; no rerun; Claude Code 2.1.325; 318 runs, 79.76 USD list price). Before = main at the branch's merge base; after = #16395's head da23783, with the eval cases frozen in their own commit before the cut.

    1. Ceiling check (7 property cases with and without the plugin, sonnet): 6 kept; wr-ts-async-property-marked dropped at ceiling (1.000 without the plugin).
    2. Two-version compare (20 cases, 3 runs per side, margin 0.05): no loss on both models, no case drop.
    Model Paired change 95% interval Verdict
    sonnet -0.00 -0.05 to +0.04 non-inferior
    opus -0.00 -0.02 to +0.02 non-inferior

    On sonnet the interval's lower end sits at the margin in the registered reading (runs where the skill loaded); the all-runs reading printed beside it falls just below the margin.

    1. Sensitivity control (the 6 kept property cases, sonnet, wrong pages beside a fresh before copy). Predicted in refactor(testing): cut the per-language property pages from testing:write as the pilot #16395's body before the first batch, at low confidence: the wrong pages move the score. Observed: they did not. Before mean 0.861, wrong-page mean 0.843, a fall of 0.019 against the 0.15 bar, no case drop; 7 of the 18 wrong-page runs read a wrong page. By the meaning this issue fixed in advance, the model ignores the page, this result is the evidence for this cut, and each later whole-page cut still runs its own control.

    The control gave one of the two predicted outcomes, so the load-bearing assumption (the control gives a usable answer for a whole-page cut) held.

  14. removed
    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.

Type

Fields

Work status

confirmed

Priority

medium

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions