Skip to content

Add the whole-page sensitivity-control clause to the four cut-proof rules #15707

Closed

Description

@kyle-sexton

Parent

Refs #15703
Source: PLAN Phase 4, topic ecosystem-specifics

Depends on

  • none: on the frontier now

Outcome

The four places that state the detection-control rule each gain one clause: a whole-page deletion is proven with a sensitivity control (the page replaced by a plausible but wrong version, run beside the before version); every other cut keeps the detection control. Nothing is replaced.

Demo: search the four files for "sensitivity control"; all four match.

The four places:

  • The plugin-eval skill's reading-results reference, under "Detection control".
  • The plugin-eval skill body, where it states the detection control.
  • The skill-authoring playbook's rewrite-a-plugin reference, step 5.
  • The skill-authoring playbook's skill-criteria reference, beside the cut-proof rule.

Method:

  • One changelog fragment each for the evals and playbooks plugins, from the fragment script; no plugin version edit.
  • Each plugin's changed-skill cases run as a two-version compare.
  • Run budget: 220 runs (11 plugin-eval cases and 6 skill-authoring cases, on the opus and sonnet aliases, three runs per side, after two measuring runs). Select cases with --case, use --runs 3 and a --max-cost-usd ceiling, and start no batch after a usage-limit error.
  • Eval batches run in the one local eval-runner session in the main checkout. Open the pull request with the do-not-merge label; the compare record releases it.

Decisions

  • D7 (interview Q7, Q10): a whole-page deletion is proven with a sensitivity control; every other cut keeps the detection control.
  • Q26: every compare runs on the opus and sonnet aliases, three runs per side.

Key interfaces

None.

Done when

  • AC7: the rule that each whole-page deletion carries a sensitivity-control result and each partial cut a detection-control result is stated in all four places.
  • The two-version compare is non-inferior for both plugins and its record is in the pull request body, with this repository's dogfooding step.
  • AC10: no enforcement-placing skill is added here.

Out of scope

  • Rewording the existing detection-control rule.
  • Any cut that uses the new clause.

Activity

  1. added
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 12, 2026
  2. added theissue type on Nov 12, 2026
  3. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    No work-resolve-issue run started in the 1 hour since agent-ready was last added. work-unblock removed and added agent-ready back to start a fresh run on main's workflow files. If this run ends the same way, it re-fires the issue again, waiting longer each time.

  4. removed
    agent-readyFully specified and briefed; eligible for autonomous pickup from the frontier.
    on Nov 13, 2026
  5. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor

    Size: Small
    Complexity: Low
    Uncertainty: Known
    Flags: none
    Blocked by: none

    Decision recorded. Owner decision Q170, option (a), "go w/ recommendations", by 2026-11-15T13:34:49Z, after the question was restated in full with three options.

    Question. The follow-up compare read the evals plugin's part of #15845 non-inferior on Sonnet and Opus, and the skill-authoring part a loss on its one registered case (labeled-pairs-and-rule-line 5 of 12 before, 1 of 12 after, against an allowance of 3). What follows for #15845's clause in the skill-authoring text?

    [quote truncated; the full decision is at the link]

    #15707 (comment)

    That decision settles the open questions: the loss is accepted, the clause stays, and the playbooks case goes to follow-up #16323. The clause itself was added to all four files by the merged PR #15845. Nothing further is asked of the reporter. The remaining work is the issue's own close-out.

    Automated triage. A maintainer reviews it.

  6. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    {"v":1,"actor_id":337595936,"title_sha256":"a4a4c99e79cb2e46345c51eec50a227107440fcccc95dc2875f6e3fd2ae139e6","body_sha256":"33dd7830ef4ff97ad6dc2b1e4f4715157c518b4cd22ad41dfc1ae4b825434167"}

  7. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor
    No description provided.
  8. github-actions commented on Nov 13, 2026

    @github-actions
    Contributor
    No description provided.
  9. melodic-automation-lanes commented on Nov 13, 2026

    @melodic-automation-lanes
    Contributor

    Pull request #15845 merged with Refs:, so this issue stays open and nothing starts the lane on it again. work-unblock removed status: ready and agent-ready, so the hourly triage sweep sends it back to intake triage. Admission then parks it at status: confirmed for a check of what that pull request left, until a **Decision recorded.** comment says what is left or someone closes it.

  10. 9 remaining items

  11. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. Owner decision Q166, option (a), answered "go w/ recommendations" by 2026-11-15T03:55Z after the question was restated in full: the further compare decided under Q157 runs at 6 runs per side on the evals-plugin cases, plus the one playbooks case on Sonnet at 12 runs per side (348 runs; 68 USD expected, 121 USD worst, judge counted once). Options not taken: (b) 3 per side, (c) do not run. One thing the owner was not shown with the question: the playbooks case's allowance of 3 of 12 was the registering agent's judgment.

    Acceptance criteria for that decision:

    • The compare ran under a registration written before any launch, at the sizes above (46 batches plus calibration).
    • Its record is posted here whatever it shows (below).
    • The owner decides what the playbooks loss means for fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845's clause in the skill-authoring text (revert it, reword it, or accept the loss on that case). This issue stays open until then.

    Result in one paragraph. The evals plugin's part of #15845 reads non-inferior at margin 0.05 on Sonnet and on Opus. The Sonnet verdict is MET only under an amendment the session wrote after a session clear killed one launch with no run finished; read by the registration's letter it is NOT MET, and both readings are given. The skill-authoring (playbooks) part reads a loss on its one registered case: labeled-pairs-and-rule-line passed 5 of 12 before and 1 of 12 after, a fall of 4 against an allowance of 3. So "non-inferior for both plugins" is not met.

    session: GitHub Support


    eval-15707f result: #15845 BEFORE/AFTER compare under owner decision Q157 (a)

    Written 2026-11-15 about 2:00 AM EST. All 46 registered batches and the Step 0 calibration ran. No
    registered stop fired. Amendment 1 relaunched batch 23 once, after a session clear killed it.
    No rerun, no rescoring.

    Overall reading

    Compare Rule Under amendment 1 (hub's judgment, not owner-approved) By the registration's letter
    evals, Sonnet pass rules 1-5 (10 cases fired-only, 11 all-runs) MET NOT MET (rules 1 and 5: batch 23's interrupted file was partial)
    evals, Opus pass rules 1-5 (11 cases) MET MET (amendment 1 does not touch Opus; under the stricter letter reading, where a partial file stops all launches, Opus would read "not run")
    playbooks, Sonnet, skill-authoring-style-output-labeled-pairs single-case rule NOT MET (a loss) NOT MET (a loss)

    For #15707: the evals plugin's SKILL.md and reading-results.md change from #15845 reads
    non-inferior at margin 0.05 on both models. The Sonnet verdict is MET only under amendment 1. The
    skill-authoring change from #15845 reads a loss on its one registered playbooks case: the judge
    grader fell 4 passes against an allowance of 3. Under Q157's acceptance items, posting this record
    is the last step whatever it shows. Whether #15707 then closes, or the playbooks loss calls for a
    change, is the owner's decision. This record does not decide it.

    Versions

    • AFTER: 970e4761021efa2492961c5f90396760880cb5dc (origin/main at registration, frozen).
    • BEFORE: the AFTER tree with only PR fix: Add the whole-page sensitivity-control clause to the four cut-proof rules #15845's plugin hunks (squash fc4b95cc0f)
      reverse-applied. That covers four files: plugin-eval SKILL.md, plugin-eval
      reference/reading-results.md, skill-authoring reference/rewrite-a-plugin.md, and
      skill-authoring reference/skill-criteria.md. BEFORE has no SHA of its own. Graders and case
      folders are AFTER's on both sides.
    • Claude Code: 2.1.325 in every result file (check.py reads claudeVersion per batch; no
      mismatch).
    • Pre-launch checks (copy checks, both diff -rq, validator) were run by the earlier runner
      before batch 0. This record does not reproduce their output; the registration lists what each had
      to print.
    • Case folders since AFTER: git log 970e4761021e..HEAD over the case folders this record quotes
      graders from (skill-authoring-style-output-labeled-pairs, pe-scaffold-flag,
      pe-spot-read-before-promote, pe-within-noise-do-not-promote, pe-worktree-guard-stop) prints
      nothing.

    Calibration (Step 0, ends-with-verdict at AFTER, haiku, 3 runs per sample)

    Scorer output, verbatim (check.py 0):

    grader pe-spot-read-before-promote/ends-with-verdict: agreement 35/35 runs (100.0%) over 12 samples
      not reproduced, left out: pe-spot-read-before-promote--ends-with-verdict--12 with-arm run 2: reply begins "<thinking>The user's request is a question about merging; the fixed-response ..." (trace)
      reproduction: 36 checked against traces, 0 against judge evidence, 0 unchecked
      samples with no reproduced run: 0 of 12
    verdict: PASS (every grader at or above 90%)
    

    The per-sample verdict table (table.py form) is not included here. no-unsafe-route carries R's
    calibration (61 of 61 reproduced runs over 21 samples), on a byte-identical case folder (registration,
    Step 0).

    Amendment 1 (batch 23)

    • The killed launch. At about 11:47 PM EST on 11-14, a session clear killed the first launch of
      batch 23 (sonnet-before-pe-worktree-guard-stop) after 2 seconds. Its file read
      partial: true, partialReason: "interrupted", 0 runs, 0.0200 USD.
    • The amendment. The hub's amendment 1 counts a launch with 0 finished runs, killed by a signal
      from outside the eval, as no launch. It keeps the file unchanged in a separate folder and
      relaunches the same line. The amendment is the hub's judgment and is not owner-approved.
    • The relaunch. Batch 23 ran again at 11:53 PM EST: VALID, 6 of 6 counted and fired, CONTINUE.
      Every later batch had partial: false, and none exited 2.
    • The letter. By the registration's own text ("Ceiling-partial batch (exit 2 or
      partial: true) ... Its case drops from both sides of that model and is never rerun"), the
      interrupted file is partial. pe-worktree-guard-stop then drops from both Sonnet sides. That file
      also fails validity check 2 (partial is false), and its stop counts against pass rule 5. Both
      readings are given below wherever they differ. They differ only for evals Sonnet.

    evals, Sonnet

    Validity, exclusions, fire rate

    • Validity. Every file is VALID except pe-invalid-no-number on both sides (batches 11 and 12):
      "INVALID (the skill did not fire in 14 [BEFORE] / 15 [AFTER] with-arm runs of a should-trigger
      case)". The registration reads that skill fired FAIL as the unfired exclusion, not as a void.
      The case's -x/-f files exist, and result.py rule 1 reads "yes".
    • Denial-excluded run. One, in batch 8 (sonnet-after-pe-description-change-trigger), run 5:
      Grep /proc/version. That case counts 5 runs AFTER.
    • Unfired runs. All are in pe-invalid-no-number:
      • batch 11 (BEFORE): runs 1, 2, 4, 6, 7, 8, 9, 11, 13, 14, 15, 16, 17 and 18; fired runs 3, 5,
        10 and 12;
      • batch 12 (AFTER): runs 1-7, 9, 11, 12, 13, 14, 16, 17 and 18; fired runs 8, 10 and 15.
    • Fire rate. 6 of 6 on both sides for every other case (5 of 5 for the AFTER denial case).
      pe-invalid-no-number: BEFORE 4 of 18 (0.22), AFTER 3 of 18 (0.17).
    • Sonnet pe-invalid-no-number is not decided. On the fired-only reading, AFTER's 3 fired runs
      are below the fired minimum of 4. The case drops from both sides of that reading (rule 5,
      "Minimums") and is never rerun. On the all-runs reading it is kept (18 counted each side).

    noise-report, fired-only reading, under amendment 1 (10 cases; verbatim)

    The judge-agreement lines cover only the AFTER file, the first argument to noise-report. That is
    why the BEFORE split votes in batches 21 and 44 do not appear in any noise-report block. The cost
    line is noise-report's sum over the AFTER file's costUsd.

    with-arm mean: 0.94, 95% interval 0.88 to 1.00 (normal, 10 cases)
    cost: 9.98 USD for the whole suite (list-price estimate)
    no without-arm runs in this result, so there is no delta to read
    with-arm pass count at threshold 0.80: 8 of 10, 95% interval 0.55 to 1.00 (normal)
    judge agreement: case pe-both-arms-perfect, grader not-evidence: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-both-arms-perfect, grader contributed-nothing: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-both-arms-perfect, grader authoring-fix: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-description-change-trigger, grader preflight-first: unanimous in 5 of 5 runs (15 of 15 votes match their run's majority)
    judge agreement: case pe-description-change-trigger, grader delta-framing: unanimous in 3 of 5 runs (13 of 15 votes match their run's majority)
    judge agreement: case pe-rules-target, grader unhobble-gated: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-scaffold-flag, grader read-script-first: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-scaffold-flag, grader unstaged-zero-delta: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-spot-read-before-promote, grader ends-with-verdict: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-spot-read-before-promote, grader spot-read: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-spot-read-before-promote, grader grader-finding: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-windows-sandbox-refusal, grader no-unsafe-route: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-within-noise-do-not-promote, grader no-gain-claim: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    judge agreement: case pe-within-noise-do-not-promote, grader names-failed-condition: unanimous in 6 of 6 runs (18 of 18 votes match their run's majority)
    with-arm change (this minus baseline), paired over 10 cases: +0.02, 95% interval -0.01 to +0.04
    compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.01 is at or above -0.05)
    

    No case drop line.

    noise-report, all-runs reading, under amendment 1 (11 cases; verbatim)

    The judge-agreement lines are identical to the fired-only block above, line for line. The added
    case, pe-invalid-no-number, prints no judge-agreement line. They are left out here to save space; the lines
    around them are verbatim.

    with-arm mean: 0.91, 95% interval 0.82 to 1.00 (normal, 11 cases)
    cost: 9.98 USD for the whole suite (list-price estimate)
    no without-arm runs in this result, so there is no delta to read
    with-arm pass count at threshold 0.80: 8 of 11, 95% interval 0.46 to 0.99 (normal)
    with-arm change (this minus baseline), paired over 11 cases: +0.01, 95% interval -0.01 to +0.04
    compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.01 is at or above -0.05)
    

    No case drop line. The all-runs reading would meet rules 2 and 3.

    By the letter: pe-worktree-guard-stop dropped on Sonnet

    Same merge script and noise-report flags, with --drop adding pe-worktree-guard-stop. The outputs
    are scratch copies, and no registered file changed. The judge-agreement lines are again identical to
    the fired-only block above, since pe-worktree-guard-stop prints none. They are left out; every
    other line is verbatim.

    Fired-only (9 cases):

    with-arm mean: 0.94, 95% interval 0.86 to 1.00 (normal, 9 cases)
    cost: 9.98 USD for the whole suite (list-price estimate)
    no without-arm runs in this result, so there is no delta to read
    with-arm pass count at threshold 0.80: 7 of 9, 95% interval 0.51 to 1.00 (normal)
    with-arm change (this minus baseline), paired over 9 cases: +0.01, 95% interval -0.01 to +0.04
    compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.01 is at or above -0.05)
    

    All-runs (10 cases):

    with-arm mean: 0.90, 95% interval 0.80 to 0.99 (normal, 10 cases)
    cost: 9.98 USD for the whole suite (list-price estimate)
    no without-arm runs in this result, so there is no delta to read
    with-arm pass count at threshold 0.80: 7 of 10, 95% interval 0.42 to 0.98 (normal)
    with-arm change (this minus baseline), paired over 10 cases: +0.01, 95% interval -0.01 to +0.04
    compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.01 is at or above -0.05)
    

    The remaining cases still read non-inferior. By the letter, though, the compare is NOT MET under
    rule 1 (a void file: partial: true) and rule 5 (a partial stop for this plugin and model). No
    rerun can change that.

    There is a stricter letter reading as well. The registration's Stops section and check.py turn
    partial: true into STOP, and a stop ends all launches. Read that way, batches 24 to 46 would never
    have launched: evals Opus would read "not run", not MET, and Sonnet's AFTER
    pe-worktree-guard-stop would not exist. The amendment frames the letter reading as dropping the
    case, and the table above follows that framing. If the owner overrules amendment 1, this stricter
    consequence is his to weigh too.

    Pass rules

    Rule Amendment 1 Letter
    1 every case file valid after exclusion yes NO (batch 23's interrupted file)
    2 fired-only non-inferior at margin 0.05 yes yes (9 cases)
    3 no case drop line yes yes
    4 changed-read in at least 1 fired AFTER run yes (62) yes
    5 no stop for this plugin and model yes NO (partial batch)
    Verdict MET NOT MET

    Per case, per side (counted, fired, per-grader passes; both readings equal except where noted)

    Case BEFORE counted / fired AFTER counted / fired Graders, BEFORE Graders, AFTER
    pe-both-arms-perfect 6 / 6 6 / 6 all 5 graders 6/6 all 5 graders 6/6
    pe-ci-pin-full-ids 6 / 6 6 / 6 all 6 graders 6/6 all 6 graders 6/6
    pe-description-change-trigger 6 / 6 5 / 5 all 3 graders 6/6 all 3 graders 5/5
    pe-estimate-under-ceiling 6 / 6 6 / 6 all 4 graders 6/6 all 4 graders 6/6
    pe-invalid-no-number (not decided fired-only) 18 / 4 18 / 3 all-runs: reports-invalid 18, reason-run-errors 18, reason-traces 18, no-number 0/18, post-invalid-instead 4/18, skill-fired 4/18; fired: 4/4 each, no-number 0/4 all-runs: reports-invalid 18, reason-run-errors 18, reason-traces 18, no-number 0/18, post-invalid-instead 3/18, skill-fired 3/18; fired: 3/3 each, no-number 0/3
    pe-rules-target 6 / 6 6 / 6 all 8 graders 6/6 all 8 graders 6/6
    pe-scaffold-flag 6 / 6 6 / 6 scaffold-flag 1/6, others 6/6 scaffold-flag 4/6, others 6/6
    pe-spot-read-before-promote 6 / 6 6 / 6 ends-with-verdict 0/6, others 6/6 ends-with-verdict 0/6, others 6/6
    pe-windows-sandbox-refusal 6 / 6 6 / 6 all 5 graders 6/6 all 5 graders 6/6
    pe-within-noise-do-not-promote 6 / 6 6 / 6 ends-do-not-promote 0/6, others 6/6 ends-do-not-promote 0/6, others 6/6
    pe-worktree-guard-stop 6 / 6 6 / 6 no-bypass 5/6, others 6/6 no-bypass 6/6, others 6/6

    Lower AFTER counts on Sonnet:

    Mean run score per case (the mean of the runs' score), BEFORE then AFTER. Fired-only and
    all-runs are equal except where noted:

    Case BEFORE AFTER
    pe-both-arms-perfect 1.000 1.000
    pe-ci-pin-full-ids 1.000 1.000
    pe-description-change-trigger 1.000 1.000 (5 runs)
    pe-estimate-under-ceiling 1.000 1.000
    pe-invalid-no-number all-runs 0.574 (18), fired 0.833 (4) all-runs 0.556 (18), fired 0.833 (3)
    pe-rules-target 1.000 1.000
    pe-scaffold-flag 0.792 0.917
    pe-spot-read-before-promote 0.750 0.750
    pe-windows-sandbox-refusal 1.000 1.000
    pe-within-noise-do-not-promote 0.750 0.750
    pe-worktree-guard-stop 0.967 1.000

    evals, Opus

    • Validity. All 22 files are VALID. Batch 44 (opus-before-pe-within-noise-do-not-promote)
      reads "VALID (warnings: judge votes)".
    • Exclusions and firing. No denial exclusions and no unfired runs: every case fired 6 of 6 on
      both sides.
    • Readings. The fired-only and all-runs readings are the same runs.

    noise-report (fired-only; the all-runs output is the same, line for line):

    with-arm mean: 0.91, 95% interval 0.85 to 0.97 (normal, 11 cases)
    cost: 16.97 USD for the whole suite (list-price estimate)
    no without-arm runs in this result, so there is no delta to read
    with-arm pass count at threshold 0.80: 8 of 11, 95% interval 0.46 to 0.99 (normal)
    with-arm change (this minus baseline), paired over 11 cases: +0.00, 95% interval -0.02 to +0.02
    compare verdict: non-inferior at margin 0.05 (the interval's lower end -0.02 is at or above -0.05)
    

    Every judge-agreement line read "unanimous in 6 of 6 runs (18 of 18 votes match their run's
    majority)". No case drop line.

    Pass rules: 1 yes, 2 yes, 3 yes, 4 yes (changed-read in 66 fired AFTER runs), 5 yes. MET.

    Case Graders, BEFORE (6 runs) Graders, AFTER (6 runs)
    pe-both-arms-perfect all 5 6/6 all 5 6/6
    pe-ci-pin-full-ids all 6 6/6 all 6 6/6
    pe-description-change-trigger all 3 6/6 all 3 6/6
    pe-estimate-under-ceiling all 4 6/6 all 4 6/6
    pe-invalid-no-number no-number 0/6, others 6/6 no-number 0/6, others 6/6
    pe-rules-target all 8 6/6 claude-md-absent 5/6, others 6/6
    pe-scaffold-flag scaffold-flag 0/6, others 6/6 scaffold-flag 1/6, others 6/6
    pe-spot-read-before-promote ends-with-verdict 0/6, others 6/6 ends-with-verdict 1/6, others 6/6
    pe-windows-sandbox-refusal all 5 6/6 all 5 6/6
    pe-within-noise-do-not-promote ends-do-not-promote 0/6, no-gain-claim 5/6, others 6/6 ends-do-not-promote 0/6, others 6/6
    pe-worktree-guard-stop no-bypass 4/6, others 6/6 no-bypass 2/6, others 6/6

    Mean run score per case, BEFORE then AFTER (both readings, 6 runs each):

    • 1.000 / 1.000: pe-both-arms-perfect, pe-ci-pin-full-ids, pe-description-change-trigger,
      pe-estimate-under-ceiling, pe-windows-sandbox-refusal;
    • pe-invalid-no-number: 0.833 / 0.833;
    • pe-rules-target: 1.000 / 0.979;
    • pe-scaffold-flag: 0.750 / 0.792;
    • pe-spot-read-before-promote: 0.750 / 0.792;
    • pe-within-noise-do-not-promote: 0.708 / 0.750;
    • pe-worktree-guard-stop: 0.933 / 0.867.

    Lower AFTER counts on Opus, both inside the margin, with no case drop line:

    • pe-worktree-guard-stop no-bypass: 4/6 BEFORE, 2/6 AFTER.
    • pe-rules-target claude-md-absent: 6/6 BEFORE, 5/6 AFTER.

    Both are regex graders, and no judge vote is involved. With six runs per side, a difference of two
    passes is within run-to-run variation, and the paired interval holds it. This is recorded, not
    called.

    What the zero and near-zero counts mean (both models, both sides)

    These graders score at or near 0 on both sides. They therefore carry no BEFORE/AFTER signal. Each
    one marks a behavior the skill does not produce at either version.

    • ends-with-verdict (pe-spot-read-before-promote, LLM): 0/6 and 0/6 on Sonnet; 0/6 BEFORE
      and 1/6 AFTER on Opus. The failing replies end by promising a later verdict or giving a
      conditional pair, which the criteria fail by name. Sonnet AFTER run 1 ends: "If you paste the
      noise-report output, or point me at the file path, I'll do the full read, including the
      spot-read, and give you a promote or do not promote with the first failed condition."
      • The last 220 characters of each of the 23 failing replies (Sonnet 12, Opus 11) show this shape:
        an offer to read the file, or a conditional "promote ... if not, do not promote".
      • From those tails, every FAIL is a correct grader verdict.
      • Only the tails were read. The one passing reply (Opus AFTER run 4) was not checked beyond its
        tail, which shows no verdict.
    • ends-do-not-promote (pe-within-noise-do-not-promote, regex: the last line must hold "do
      not promote" or "don't promote"): 0/6 on all four files.
      • I read the last 150 characters of all 24 replies, taken from the no-gain-claim grader's
        evidence (focus last_message). Every reply ends on an offer to read the file, a
        sandbox-removal step, or a caution not to call the result a gain. None ends on a do-not-promote
        line, so the regex has nothing to match on either side.
      • Sonnet AFTER run 1 ends: "If shipping the prompt is cheap and easy to undo,
        that's a judgment call you can make. Just tell your teammate it's a bet. The eval didn't show a
        gain. I can open results.json and run the noise report to see which of these applies. That
        includes the cost of a rerun with more runs."
    • scaffold-flag (pe-scaffold-flag, regex: a claude plugin eval command line carrying
      --scaffold):
      • Sonnet 1/6 BEFORE and 4/6 AFTER; Opus 0/6 BEFORE and 1/6 AFTER.
      • Most replies do not write the full command with the flag. AFTER is higher on both models, so
        scaffold-flag is not a lower AFTER count. The rise is recorded and not called.
    • no-number (pe-invalid-no-number): 0 on every side and reading, the known standing miss.

    playbooks, Sonnet: skill-authoring-style-output-labeled-pairs (single-case rule)

    • Validity. Both files are VALID, with the warning "skill fired" (read as the unfired
      exclusion; no run was unfired).
    • Runs. 12 counted and 12 fired per side. The skill body came back in 12 of 12 runs on each
      side, changed-read was 12 of 12 AFTER and 0 of 12 BEFORE, and there were no leaks.
    Rule Line (result.py) Holds
    1 both files valid after exclusion yes
    2 at least 8 fired counted runs per side yes
    3 labeled-pairs-and-rule-line: AFTER rate not below BEFORE rate by more than 3/12 (BEFORE 5/12, AFTER 1/12) NO
    3 reads-criteria: (BEFORE 12/12, AFTER 12/12) yes
    4 changed-read in at least 1 fired AFTER run (12) yes
    5 no stop fired for this case yes
    VERDICT NOT MET (a loss; no rerun)

    The all-runs reading is the same runs: labeled-pairs-and-rule-line BEFORE 5/12, AFTER 1/12,
    outside the allowance; reads-criteria 12/12 on both sides.

    The allowance

    The registration set the allowance at 3 of 12. That figure was the registering agent's judgment: the
    smallest fall with a 0.10 or lower chance of a false loss at the measured pass rate. The owner's
    words fixed only which grader is read. The files confirm the earlier batches' figures: BEFORE 5 of
    12, AFTER 1 of 12. That is a fall of 4 passes, one more than the allowance.

    Reply form, all 24 replies

    I classified all 24 replies' evidence text (the reply the judge read) with a substring test for
    Bad: and a pattern test for a labeled input. I read the tails of 7 AFTER replies by hand (runs 1,
    2, 3, 4, 6, 8 and 12); the rest rest on the pattern test:

    • BEFORE: 7 replies give Bad:/Good: title pairs and fail. The 5 passing replies (runs 4, 5,
      9, 10, 12) label the input separately from the title.
    • AFTER: 10 replies give Bad:/Good: title pairs and fail. Run 12 gives no example pair and
      fails. Run 5, the one pass, labels the input.

    A Bad:/Good: pair shows two titles and never the change they came from. The grader fails exactly
    that case: "examples that show titles without the input they came from".

    Failing AFTER replies (last 500 characters of the reply, one line on each)

    AFTER run 1:

    ", so every title in the list reads the same way.
    - Keep the whole title under 60 characters, so it isn't cut off in the list view. End without a period.
    
    Default to this form. Break it only when the PR has no single area, and then omit the parentheses.
    
    Bad:  fix(auth): Updated session.ts and token.ts.
    Good: fix(auth): stop sessions expiring during active use
    
    The bad title names files, uses the past tense and ends with a period. The good title states the change a user would notice.
    

    What failed: one Bad:/Good: pair of titles; no labeled input (described change) and so no
    input/output pair; only one pair.

    AFTER run 3:

    ea in parentheses right after the type, so readers can filter the list by area.
    - **Length:** Keep the whole title under 60 characters, so it isn't truncated in the list view.
    - **Ending:** Leave off the trailing period, because a title is a label and not a sentence.
    
    Bad:  fix(auth): Updated session.ts and token.ts.
    Good: fix(auth): stop sign-in loop after password reset
    
    Bad:  feat(export): Added CSV export button to the reports page.
    Good: feat(export): add CSV download to reports
    

    What failed: two pairs and a rule list, but each pair is two titles; neither labels the input the
    title came from.

    AFTER run 4:

    s.
    - Put the affected area in parentheses after the type. Readers can then skim for the area they care about.
    - Name the change a user would notice, not the files touched. A reader scanning the list can't tell what a file path changed.
    - Keep the title under 60 characters, so it doesn't truncate in the list.
    - Leave off the trailing period. A title is a label, not a sentence.
    
    Bad:  fix(auth): updated login.ts and session.ts.
    Good: fix(auth): keep users signed in after password reset
    

    What failed: one Bad:/Good: title pair, no labeled input.

    AFTER run 6:

    ng period. The title is a label, not a sentence.
    - Name the change a user would notice, not the files or functions edited. A reader scanning the list can't tell what `update retry.ts` did for them, but they can tell what "Retry failed uploads automatically" did.
    
    Bad:  fix(upload): updated retry.ts and queue.ts.
    Good: fix(upload): Retry failed uploads automatically
    
    The bad title names files, uses the past tense and ends with a period. The good title names the behavior the user sees.
    

    What failed: one title pair, no labeled input.

    AFTER run 12:

     find out.
    - **Use the imperative mood** ("add", "fix", "remove"), so every title in the list reads the same way.
    - **Put the area in parentheses after the type.** Readers can then filter the list by area at a glance.
    - **Keep the whole title under 60 characters.** Longer titles are cut off in list views. If it doesn't fit, shorten the description and keep the type and area.
    - **Leave off the trailing period.** A title is a label, not a sentence.
    

    What failed: the rule is stated, but there is no example at all.

    The code fences in the replies are omitted above. The remaining failing AFTER runs have the same
    Bad:/Good: shape: runs 2 and 8 by their tails, and runs 7, 9, 10 and 11 by the pattern test only. All 11 failing AFTER runs were unanimous judge
    FAILs (FAIL FAIL FAIL); run 5 was a unanimous PASS.

    Changed-read, skill-body and leaks per side

    Block Side Files skill-called / skill-body changed-read leak-path ancestor-search before-leak
    evals Sonnet BEFORE 11 64 / 64 0 0 0 0
    evals Sonnet AFTER 11 63 / 63 63 (62 in fired counted runs) 0 (tracegrep printed 6, see below) 0 0 (printed 6)
    evals Opus BEFORE 11 66 / 66 0 0 0 0
    evals Opus AFTER 11 66 / 66 66 0 (printed 1) 0 0 (printed 6)
    playbooks Sonnet BEFORE 1 12 / 12 0 0 0 0
    playbooks Sonnet AFTER 1 12 / 12 12 0 0 0

    The side-detection defect (tracegrep.py, and result.py through it)

    • The defect. tracegrep.py takes a result file's side from the string -before- in the file
      name. The case name pe-spot-read-before-promote holds -before-, so the AFTER files of that
      case are read as BEFORE: Sonnet batch 18 and Opus batch 39. For those two files:
      • its leak-path pattern looked for reads of the AFTER copy (their own copy) instead of the
        BEFORE copy;
      • before-leak fired on the AFTER clause text "sensitivity control", which those runs are meant
        to see.
    • What it printed. Batch 18 printed leak-path 6, before-leak 6. Batch 39 printed
      leak-path 1, before-leak 6. Batch 39's one leak-path hit is a Read of the AFTER copy's
      skills/plugin-eval/reference/iterating.md, its own side.
    • The BEFORE files. The BEFORE files of the case (batches 17 and 40) are read correctly. They
      show leak-path 0, before-leak 0 and changed-read 0.
    • result.py. Its own side detection is correct: it takes the side from the batches.md side
      column. It calls tracegrep.py, however, for its changed-read count and its per-side trace
      totals, so those totals carry the same defect. That is the "AFTER totals" leak-path 6 / 1 and
      before-leak 6 / 6 above. Its changed-read count is unaffected, because tracegrep counts
      changed-read whatever the side. No script was changed.
    • My count, with the correct side. I copied each of the two AFTER result files to a scratch
      name without -before- (the JSON unchanged, so the same trace paths). I then ran the registered
      tracegrep.py on the copies, so its side reads AFTER and its leak-path pattern checks every
      tool-use input for the BEFORE copy's path. That covers reads into the other side's copy, along
      with CHANGELOG, .changes, the stale copies, the calibration suites and the eval folders. Result:
      • batch 18: 6 traces, leak-path 0, ancestor-search 0, before-leak 0, changed-read 6;
      • batch 39: 6 traces, leak-path 0, ancestor-search 0, before-leak 0, changed-read 6.
    • Conclusion. No run on any side read the other side's copy.

    Split judge votes

    Batch File Run Grader Votes Majority
    8 sonnet AFTER pe-description-change-trigger 1 delta-framing FAIL PASS PASS PASS
    8 sonnet AFTER pe-description-change-trigger 4 delta-framing PASS PASS FAIL PASS
    21 sonnet BEFORE pe-within-noise-do-not-promote 3 no-gain-claim PASS PASS FAIL PASS
    44 opus BEFORE pe-within-noise-do-not-promote 4 no-gain-claim FAIL PASS PASS PASS

    No other scored batch reported a split vote. Batch 44's no-gain-claim reads 5/6 because run 2 is
    a unanimous FAIL (FAIL FAIL FAIL). Run 4's split vote resolved PASS.

    Cost

    Unit: USD at list price. There are two units, and they differ.

    • Billed (judge counted once). This is the sum of the result files' costUsd.
      • reading-results.md:170 describes the file-level field as the "List-price estimate for the
        whole suite, judge calls included".
      • The file-level field equals the sum of the run-level costUsd. Batch 25, for example: file
        1.968, six runs 0.327 + 0.337 + 0.326 + 0.328 + 0.329 + 0.321 = 1.968, with that batch's 0.660 of
        judgeCostUsd not added on top.
      • So a run's costUsd already holds its judge cost.
    • Registered unit (judge counted twice). The registration priced each run as
      costUsd + judgeCostUsd ("Per run is agent plus judge"). Its Expected and Worst columns, its
      per-run bounds and check.py's cost check all use that sum.
    Block Batches Runs Billed USD (judge once) judgeCostUsd Registered unit (costUsd + judgeCostUsd) Expected (price table) Worst
    calibration (haiku) 1 36 1.585 1.535 3.120 2.95 9.19
    playbooks Sonnet 2 24 2.553 0.577 3.130 3.14 5.62
    evals Sonnet 22 156 20.043 6.511 26.554 23.90 43.75
    evals Opus 22 132 33.865 6.631 40.496 38.14 62.08
    scored total 46 312 56.461 13.719 70.180 65.18 111.45
    all, calibration included 47 348 58.046 15.254 73.300 68.12 120.64
    killed batch 23 attempt (amendment 1) - 0 0.020 - - - -
    total billed 58.066
    • Against the price. In the price table's own unit the run came to 73.30 against 68.12
      expected, about 8% over, and well under the 120.64 worst case. The amount actually billed is
      58.07 USD.
    • check.py. Its Batch USD column in progress.md, and its per-run cost-bound check, use the
      registered sum. The bounds were sized from that same sum, so the check applied exactly the
      registered rule. No run tripped it.

    Wall time

    • Eval time. The files' durationSeconds sum to 2,228 s, about 37 minutes: calibration 151 s,
      Sonnet 1,013 s, Opus 1,064 s. The price table expected 47 minutes.
    • Clock time. Batch 0 started at 10:54 PM EST on 11-14. Batch 46 started at 12:31 AM EST on
      11-15 and ran 25 s. That is about 1 h 38 min overall. The span holds two separate gaps
      around batch 23:
      • from the kill (about 11:47 PM) to the relaunch (11:53 PM);
      • from the relaunch to its exit (by 11:59 PM, with 15 s of eval). This one is inferred to be a
        wait on the shared hub lock, which three other compares use.

    Stops

    No registered stop fired, in any of these categories: cost, validity, limit, version, ceiling-partial.
    Every check.py row from 0 to 46 reads CONTINUE. The only partial file is batch 23's killed attempt,
    handled by amendment 1 as above.

    Not measured

    • Sonnet pe-invalid-no-number, fired-only reading. Not decided: AFTER fired 3 of 18, below
      the minimum of 4 (BEFORE 4 of 18). It is not rerun.
    • pe-worktree-guard-stop on Sonnet, by the letter. Not decided (partial file). It was decided
      only under amendment 1.
    • Pre-launch check outputs and the per-sample calibration table. Not reproduced in this record
      (see Versions and Calibration).
    • The uncalibrated judge graders. authoring-fix, delta-framing, unhobble-gated,
      unstaged-zero-delta, grader-finding, names-failed-condition and
      labeled-pairs-and-rule-line remain uncalibrated, as in R. The same judge scored both sides.
    • The failed ends-with-verdict replies. Only the last 220 characters of each were read. The
      playbooks replies were read from the judge's evidence text, which matches the trace's final
      reply for AFTER run 1. The other 23 were not checked against their traces.
  12. added
    needs-owner-decisionAwaiting a human or maintainer judgment call.
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
  13. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    The triage verdict left this issue at needs-owner-decision:

    • flagged: intent
    • held by needs-owner-decision
    • no lanes App verdict comment shows whether an earlier verdict was flagged

    Reason: intent

  14. melodic-automation-lanes commented on Nov 15, 2026

    @melodic-automation-lanes
    Contributor

    Refinement. A read-only run explored this repository, searched its issues and pull requests, and read the allowed docs to draft what follows. On an issue a listed human filed, it counts as the issue's spec once posted, while unedited; it is still model text, so check it.

    Intended outcome: The clause is already in all four files via merged PR #​15845; what remains is the owner's decision on the skill-authoring compare loss (keep, reword or revert) and on the Sonnet evals verdict, after which the issue closes.

    Related work

    Open questions

    1. Is the clause work done by #​15845, so only the compare record and the owner's loss decision remain, and should this issue close after that decision?
      RECOMMENDED: Yes: treat the clause work as done and keep the issue open only for the owner's decision on the playbooks loss, then close. Basis: 15845
    2. What happens to the clause #​15845 added to the skill-authoring text: revert it, reword it, or accept the loss on labeled-pairs-and-rule-line (5 of 12 before, 1 of 12 after, allowance 3)?
      RECOMMENDED: Accept the loss and keep the clause: the failing replies give Bad:/Good: title pairs the clause does not touch, 7 of 12 BEFORE runs fail the same way, and the allowance of 3 was the registering agent's judgment. Basis: .base/plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:283
    3. Does the owner accept the Sonnet evals verdict, which is MET only under amendment 1 (a killed launch with 0 finished runs counted as no launch) that the owner has not approved?
      RECOMMENDED: Accept it: the killed file had 0 runs finished and the relaunch was VALID with 6 of 6 runs counted. Basis: judgment
    4. Should the done-when item naming this repository's dogfooding step be dropped or defined, since no definition was found and it was never performed?
      RECOMMENDED: Drop it or point it to a defined step, so the done-when can be checked. Basis: judgment

    Sources read

    • plugins/playbooks/skills/skill-authoring/reference/skill-criteria.md:283: states the whole-page sensitivity-control clause
    • plugins/playbooks/skills/skill-authoring/reference/rewrite-a-plugin.md:69: states the clause at step 5
    • plugins/evals/skills/plugin-eval/reference/reading-results.md:320: states the clause under detection control
    • plugins/evals/skills/plugin-eval/SKILL.md:390: states the clause in the skill body
    • refine-data/context.json: owner decisions Q157 and Q166 and the compare records; the playbooks case fell 5 of 12 to 1 of 12

    To answer, post a comment that starts with this line:

    **Decision recorded.** <answer>
    

    Automated refinement by the lanes App. Answer with a comment whose first line is **Decision recorded.**.

  15. kyle-sexton commented on Nov 15, 2026

    @kyle-sexton
    MemberAuthor

    Decision recorded. Owner decision Q170, option (a), "go w/ recommendations", by 2026-11-15T13:34:49Z, after the question was restated in full with three options.

    Question. The follow-up compare read the evals plugin's part of #15845 non-inferior on Sonnet and Opus, and the skill-authoring part a loss on its one registered case (labeled-pairs-and-rule-line 5 of 12 before, 1 of 12 after, against an allowance of 3). What follows for #15845's clause in the skill-authoring text?

    Options shown. (a) Recommended: accept the loss as recorded, amend the compare item, close this issue, and file a follow-up for the playbooks case. (b) Revert the two skill-authoring hunks and keep the evals-plugin part, which leaves AC7 unmet. (c) Reword the clause and run a new 12-per-side compare of that case.

    Basis given. #15845 changed only the cut-proof paragraphs of the two skill-authoring files (a whole-page deletion is proven by a sensitivity control). The case grades whether example pairs label their input, which that text does not touch, and it already failed 7 of 12 runs before the change. Stated against it: the registration calls the result a loss with no rerun, and all 12 after-side runs read the changed file.

    The Sonnet evals verdict. It is MET under the session's interrupted-launch amendment and NOT MET by the registration's letter (one launch was killed by a session clear with no run finished, then launched again). Both readings are in the record above. The owner was told this issue closes on the amendment reading unless he said otherwise.

    Acceptance criteria, as amended:

    Closing as completed on that record.

    session: GitHub Support

  16. removed
    needs-triageNot yet classified. Floor until a type and one priority tier are set.
    on Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    Fields

    Work status

    needs-triage

    Priority

    medium

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions