Skip to content

Accept any recorded or inferred ecosystem id and any mutation tool value - part 4 of 4 #16063

Description

@melodic-automation-lanes

Part 4 of 4 of #15710, split at intake (owner Q93). The done-when below is copied from #15710 as written.

Done when

  • A full /testing:audit scan of this repository shows zero changed findings before and after (dogfood).
  • Each touched script's test file passes; the two-version compare is non-inferior and its record is in the pull request body.
  • AC7: nothing is cut here, so no control result is owed.
  • AC10: no new enforcement-placing skill is added here.

Activity

  1. added
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 14, 2026
  2. github-actions commented on Nov 14, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: #16355, #16356, #16357, #16358

    Decision recorded.
    Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.

    • Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
    • Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
    • Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.

    Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.

    Acceptance criteria

    #16063 (comment)

    This decision settles the open questions. What remains is waiting on the four follow-up issues above, then re-reading the done-when against their records. The compare itself is a host run, already completed and recorded here.

    Automated triage. A maintainer reviews it.

  3. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 14, 2026
  4. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    ContributorAuthor

    Not admitted to the issue lane: its done-when asks for what an unattended lane run cannot perform: item 8 (tracker-output: "The record is posted here as a comment with the six verdicts, whatever they s..."); recommended split: keep the other items here as the admissible issue and move these to a child issue for the host queue (host-run); it carries host-run. Admission takes the issue once each of those is met, when intake triages it again (.github/actions/resolve-config/admission-rules.json).

  5. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    ContributorAuthor

    {"v":1,"actor_id":337595936,"title_sha256":"c403775406470bfeca5bde4219ef8256e01fafcb209620635796475088871d96","body_sha256":"b9ff2193f8cd973eaf3a3c019a49d8cb678efe1db69545ed6ee720f2582c25e0"}

  6. 2 remaining items

  7. melodic-automation-lanes commented on Nov 14, 2026

    @melodic-automation-lanes
    ContributorAuthor

    The work-resolve-issue lane stopped on this issue (host-only): the run reports the work needs a host run the lane cannot make.
    It added host-run and set Work status to confirmed: an agent run on a host builds it, and it waits on no decision from a person.

    In the run's words:

    host-only: every done-when item of this part needs a run this lane cannot make. (1) The full /testing:audit scan of this repository, before and after the #15710 change, needs the testing plugin's audit skill, and only the work-items plugin is loaded in this lane. (2) The two-version compare of the skill-text changes is an eval batch (opus and sonnet aliases, three runs per side, inside the 850-run budget) that #15710's Method says runs in the one local eval-runner session in the main checkout, and its record goes in the pull request body. AC7 and AC10 are statements about the sibling parts' work and need no edit. This part builds nothing, so the lane has nothing to commit.

    What the run says it did not do:

    • Full /testing:audit scan of this repository before and after, showing zero changed findings (dogfood): needs the testing plugin's audit skill, which this lane does not load; a host run.
    • Two-version compare (opus and sonnet, three runs per side) of the skill-text changes, with its record in the pull request body: needs the local eval-runner session in the main checkout; a host run.
    • Each touched script's test file passes: the scripts were touched in sibling parts, and their suites run in the PR's CI and the host run.

    Run: https://github.com/melodic-software/claude-code-plugins/actions/runs/43650781221

    To hand it back to the lane, remove host-run and set Work status to agent-ready again.

  8. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    Member

    Decision recorded.

    Hub session (GitHub Support). Owner decision Q159, answered in the hub session on 2026-11-14 (US Eastern), by 2026-11-15T01:43:16Z.

    Question. This issue's two-version compare was registered and not launched. Preparation found two things the scoping had missed: 24 case-and-model pairs have a recorded pass rate between 0.2 and 0.8, where 3 runs per side can show a false case drop that would read as a loss with no rerun allowed; and 12 mutation-testing setup cases load skill text that #16229 changed, each with a trace-judged grader. #15710's Method sets 3 runs per side and an 850-run budget. How large should the compare be?

    Options shown.

    • (a) As the Method says: 3 runs per side, no setup cases: 792 runs, about 232 USD list price, 9.6 h.
    • (b) Recommended: 3 runs per side with the 24 unstable pairs at 6, no setup cases: 936 runs, about 288 USD, 10.3 h. The setup-skill text is recorded as not compared, with a follow-up issue.
    • (c) Option (b) plus the 12 setup cases: 1,080 runs, about 452 USD, 16.6 h.
    • (d) 6 runs per side for every case, with the setup cases: 1,872 runs, about 793 USD, 23 h.

    Answer (owner, verbatim). "go w/ recommendations" That is option (b).

    What it allows, as acceptance criteria.

    • The compare runs as registered with the 24 unstable case-and-model pairs at 6 runs per side and every other pair at 3: 936 runs. This raises Accept any recorded or inferred ecosystem id and any mutation tool value #15710's 850-run budget to 936 for this compare.
    • The 12 mutation-testing setup cases are not run; the record says the setup-skill text shipped without a compare, and a follow-up issue carries that compare.
    • Two cases that load no changed text (ck-notice-reads-checker, confirm-build-question-no-trigger) are left out and named in the record.
    • The record is posted here as a comment with the six verdicts, whatever they show, and no batch is rerun.
  9. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    Member

    #16063 compare result: two-version eval of #16228 (toolchain, verification) and #16229 (mutation-testing)

    Computed by the registered result.py and noise-report.py (--margin 0.05 --threshold 0.8 --grader-agreement); verdict lines re-read by the hub from the result files. Done-when: not met (toolchain, verification and mutation-testing are NOT MET on Sonnet).

    Versions. toolchain and verification: BEFORE 7645e1c6ac, AFTER d87f47101e (#16228). mutation-testing: BEFORE 59d6f9d110, AFTER 08fcb9062e (#16229). Six plugin copies built with git archive; diff -rq showed the 12 registered lines before batch 1. Claude Code 2.1.325 on all 264 result files. Agent models sonnet and opus; judge claude-opus-5-5. 3 runs per side, the 24 band case-model pairs at 6 (Amendment A): 936 runs, 936 traces read, 0 missing.

    Not compared. The mutation-testing setup-skill text (setup/SKILL.md:73-76,96-98, setup/templates/config-template.md, the schema description) shipped without a compare, by the owner's choice (D303, Amendment A item 2); the 12 se-* cases were not run. A follow-up issue carries them.

    Verdicts

    Compare Under amendments B, C, D By the registration's letter
    toolchain sonnet NOT MET: -0.01, 95% interval -0.06 to +0.04, not shown non-inferior NOT MET (C's letter also drops ck-consumer-config-wins: -0.01, -0.06 to +0.05)
    toolchain opus MET: +0.00, -0.03 to +0.03, non-inferior (see the batch-225 note) NOT MET: batch 225 void, ck-clean-tree-branch-diff dropped (18 cases: +0.01, -0.02 to +0.04)
    verification sonnet NOT MET: +0.03, -0.01 to +0.07 with case drop confirm-proof-level-user-strict-wins 0.89 to 0.56, and a plugin-copy denial asymmetry (AFTER 1 run, BEFORE 0) NOT MET: batches 39 and 40 void
    verification opus MET: +0.04, +0.00 to +0.07 same
    mutation-testing sonnet NOT MET: +0.04, -0.28 to +0.35, not shown non-inferior (3 cases) same
    mutation-testing opus MET: +0.00, +0.00 to +0.00 same

    Batch-225 note: toolchain opus is MET on the owner's reading (Q179 (a)) that the weekly usage-limit stop at batch 225 is the registered pause and resume ("resuming later continues at the next batch number and is not a rerun"). Read literally, pass-rule item 5 ("no stop for that plugin and model") would make it NOT MET under D as well.

    Pass-rule item 4 holds in all six compares: the changed text appears in every counted AFTER run (tracegrep changed-read); BEFORE leaks 0.

    New cases (by design, AFTER only): ck-recorded-filter-runs and ln-inferred-filter meet the bar on both models; every grader passed 3 of 3.

    Amendments

    • B (owner, Q167 (a)): run 2 of batches 39 and 40 (confirm-live-app-run-fallback, sonnet, each side) died on "API Error: Sonnet 5.5's safeguards flagged this message"; excluded at run level, 2 of 3 counted per side.
    • C (hub, written after the event; confirmed by the owner, Q177 (a)): the batch-95 launch (sonnet-before-ck-consumer-config-wins), killed after 12 s by a session clear, is treated as no launch and its relaunch counts. Correction to C's text: the kept file holds 2 rows, both error: interrupted, scored 0.33 by the deterministic graders with no judge, and 0 finished runs; it billed 0.2226 USD. toolchain sonnet is NOT MET under both readings.
    • D (owner, Q173 (a)): run 3 of batch 225 (ck-clean-tree-branch-diff, opus AFTER) ended "You've hit your weekly limit"; excluded under B, 2 of 3 counted. Launches stopped and resumed at batch 226.
    • How B and D were applied: a result-step helper copied each affected file with only the errored run removed (it refuses a run with no error); unchanged copies of exclude.py (--runs 2 --min 2) and result.py then ran; all three files read VALID. Each exclusion was applied on the side where it occurred.

    Run-level exclusions for sandbox denials (D299; each denied call read from its trace)

    Batch Compare, side Case, run Denied call
    58 verification sonnet AFTER confirm-prep-light-does-not-block r1 Glob **/setup-apply.mjs at the AFTER copy
    97 toolchain sonnet BEFORE ck-detects-changed-ecosystems r3 Read context/dotnet.md, context/python.md
    106 toolchain sonnet AFTER ck-skips-lint-only-surfaces r2 Read context/bash.md
    162 verification opus BEFORE confirm-intent-from-plan-brief r4 Bash sed -n ... reference/config.md
    168 verification opus BEFORE confirm-light-skips-verifier r4 Glob of the copy root (a Read of a mistyped path not counted at the copy)
    189 verification opus AFTER confirm-prep-light-does-not-block r2, r5 Glob setup-apply.mjs; Bash sed ... reference/config.md
    210 verification opus BEFORE confirm-retired-assumption-confirmed r3 Grep of the copy root

    No case fell below its minimum (confirm-prep-light-does-not-block opus AFTER kept 4 of 6). No void file, dropped case or unfired run under the amendments. No shim or permission-check hits.

    Per-case results (counted runs BEFORE/AFTER where not 3/3 or 6/6, mean score BEFORE to AFTER, failing grader runs BEFORE/AFTER; cases not listed scored 1.00 on both sides)

    toolchain sonnet: ck-clean-tree-branch-diff .86 to .90 (meets-legacy-check-5 1/0, bash-row-pass 1/1, overall-pass 1/1); ck-detects-changed-ecosystems 2/3 1.00 to .92 (meets-expectations 0/1); ck-gate-fails-overall .91 to .94; ck-malformed-config-degrades 1.00 to .78 (warns-malformed 0/1, meets-expectations 0/1); ck-missing-tool-incomplete 1.00 to .83 (three graders 0/1); ck-skips-lint-only-surfaces 3/2 1.00 to .83 (meets-expectations 0/1); ln-check-mode-ts-bash .96 to 1.00; ln-missing-tool-incomplete .67 to 1.00; ln-names-unclaimed-source 1.00 to .97 (rs-named 0/1); ln-runs-yaml-and-cross-cutting .90 to .97.

    toolchain opus: ck-clean-tree-branch-diff 3/2 1.00 to .86 (bash-row-pass, overall-pass 0/1); ck-detects-changed-ecosystems .96 to 1.00; ck-malformed-config-degrades .83 to .89; ck-missing-tool-incomplete .83 to .83; ln-code-fix-file-cap .96 to 1.00; ln-code-fix-headless-stops 1.00 to .95 (says-rerun-with-yes 0/1); ln-missing-tool-incomplete .80 to 1.00; ln-no-changes-points-to-all .93 to .80 (no-changes-found 1/3).

    verification sonnet: app-control-absent-fallback-line .50 to .83; capture-blocked-not-verified .83 to 1.00; failed-check-observation-first .67 to .50 (names-observation-check 2/3); faster-claim-redirects-to-measure .78 to .89; lists-every-verifier-verdict .67 to .78; live-app-run-fallback 2/2 .33 to .83; names-what-checks-did-not-prove .33 to .33; new-endpoint-runs-e2e 1.00 to .87 (manifest-assertions 0/2); parser-and-storage-checks .78 to .89; prep-light-does-not-block 3/2 1.00 to 1.00; prep-light-mechanical-blocks .83 to .92; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .89 to .56 (report-line-strict 0/2, user-strict-boxes 1/2); retired-assumption-confirmed .44 to .44 (lists-retired 0/1); strict-docs-only-perf-na .00 to .11; strict-perf-merge-base-ab .33 to .33; strict-perf-no-baseline .67 to .67.

    verification opus: app-control-absent-fallback-line .83 to 1.00; failed-check-observation-first .67 to .67; falsified-assumption-withholds .94 to .67 (verdict-needs-work 0/3, barcode-falsified 1/3); faster-claim-redirects-to-measure 1.00 to .89; intent-from-plan-brief 5/6 .67 to .61 (couplings-table 3/6); light-skips-verifier 5/6 .90 to 1.00; lists-every-verifier-verdict .94 to 1.00; live-app-run-fallback .78 to .89; names-what-checks-did-not-prove .00 to .33; new-endpoint-runs-e2e .87 to .87 (manifest-assertions 3/4); prep-light-does-not-block 6/4 .67 to .75; prep-light-mechanical-blocks .88 to .96; proof-level-repo-live-wins .89 to 1.00; proof-level-unset-is-path .50 to .83; proof-level-user-strict-wins .78 to .89; refactor-large-reshape-equivalence .97 to .97 (verdict-not-confirmed 0/1); refactor-new-failure-surfaced .89 to 1.00; retired-assumption-confirmed 2/3 .33 to .56; strict-docs-only-perf-na .28 to .33; strict-perf-merge-base-ab .78 to 1.00; strict-perf-no-baseline .67 to .67; touched-feature-calls-refresh .83 to .67 (calls-refresh-for-cart-total 1/2); verifier-needs-work-not-upgraded .88 to .96.

    mutation-testing: sonnet au-light-survivors-unclassified .78 to .56 (no-triage-dispatched 0/1, writes-skip-token 1/2), au-no-test-command-asks .67 to 1.00; opus au-light-survivors-unclassified .89 to .89.

    Trace checks (tracegrep.py plus a bounded grep over all 936 kept traces): no is not a valid shim or Shell command permission check failed hits. The only command not found hits are cargo in ln-inferred-filter (sonnet run 3, opus run 1): the case expects cargo to be absent and both runs attempted the task, so neither is INVALID. skill-loaded and skill-body hits equal the run count on every side.

    Judges. Split votes: toolchain sonnet ck-missing-tool-incomplete / meets-legacy-check-2 (8 of 9); verification sonnet live-app-run-fallback / delegation-and-fallback (5 of 6), proof-level-user-strict-wins / user-strict-boxes (8 of 9); verification opus faster-claim-redirects-to-measure / redirect-no-claim (8 of 9), live-app-run-fallback (17 of 18), prep-light-mechanical-blocks / stops-before-outcome (17 of 18), verifier-needs-work-not-upgraded / verdict-kept (16 of 18). No judge was calibrated; the same uncalibrated judge scored both sides.

    Cost and time (file-level costUsd, which already includes the judge, so the judge is counted once; the registration's expected and worst figures counted it twice)

    Model Spent Judge share Expected Worst
    sonnet 63.70 USD 12.71 86.25 288.34
    opus 153.50 USD 18.41 201.49 465.07
    total 217.20 USD, plus 0.22 for the killed batch-95 attempt 287.74 753.41

    Summed eval time 5.80 h (sonnet 1.84, opus 3.96) against 10.3 h expected, excluding lock waits and the usage-limit pause.

    Stops that fired. The weekly usage limit at batch 225 (launches stopped, resumed at batch 226). The batch-95 session clear is covered by C.

    Follow-ups. #16355 (toolchain, Sonnet), #16356 (verification, Sonnet), #16357 (mutation-testing, Sonnet), #16358 (the 12 setup cases not compared).

    session: GitHub Support

  10. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    Member

    Decision recorded.
    Owner decisions Q177, Q178 and Q179, each option (a), in the hub session: Q177 by 2026-11-15T14:31Z ("go w/ recommednations"), Q178 and Q179 at 2026-11-15T15:04:43Z ("Go w/ recommendations"), each after the question was restated in full with its options and a recommendation.

    • Q177 (a): the three interrupted-launch amendments stand, including amendment C here (the batch-95 launch killed by a session clear with 0 finished runs is no launch; its relaunch counts). The record leads with that reading and gives the letter's reading beside it.
    • Q178 (a): this compare's done-when is not met (toolchain, verification and mutation-testing are NOT MET on Sonnet). The record is posted, this issue stays open, and each losing Sonnet compare has its own follow-up, as does the setup-skill text that was not compared.
    • Q179 (a): the weekly usage-limit stop at batch 225 is the registered pause and resume, not a loss under pass-rule item 5, so toolchain on Opus is MET; the literal reading is in the record.

    Earlier owner decisions this record applies: Q155 (a) run-level exclusion of sandbox-denied runs, Q159 (b) the setup cases not compared, Q167 (a) amendment B, Q173 (a) amendment D.

    Acceptance criteria

    session: GitHub Support

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    host-runNeeds an unattended run on a host; the host runner takes it, not a person or the issue lane.

    Fields

    Work status

    confirmed

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions