diff --git a/docs/skill-eval-status.md b/docs/skill-eval-status.md index 13f5ca8e..686d216e 100644 --- a/docs/skill-eval-status.md +++ b/docs/skill-eval-status.md @@ -153,7 +153,7 @@ suites still assert on final text only). | `establishing-ubiquitous-language` | 3 | 6 | none | no | [evals/establishing-ubiquitous-language/eval-status.md](../evals/establishing-ubiquitous-language/eval-status.md) | | `evaluating-context-channel-maturity` | 3 | 13 | none | no | [evals/evaluating-context-channel-maturity/eval-status.md](../evals/evaluating-context-channel-maturity/eval-status.md) | | `evaluating-deterministic-gate-quality` | 3 | 40 | none | no | [evals/evaluating-deterministic-gate-quality/eval-status.md](../evals/evaluating-deterministic-gate-quality/eval-status.md) | -| `evaluating-skill-quality` | 3 | 96 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) | +| `evaluating-skill-quality` | 3 | 98 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) | | `executing-a-branch-plan` | 3 | 10 | none | no | [evals/executing-a-branch-plan/eval-status.md](../evals/executing-a-branch-plan/eval-status.md) | | `explaining-the-work` | 3 | 14 | none | no | [evals/explaining-the-work/eval-status.md](../evals/explaining-the-work/eval-status.md) | | `grounding-in-primary-sources` | 3 | 23 | none | no | [evals/grounding-in-primary-sources/eval-status.md](../evals/grounding-in-primary-sources/eval-status.md) | diff --git a/docs/superpowers/plans/2026-09-02-claude-pr-1676-merge-prep-1a0cui.md b/docs/superpowers/plans/2026-09-02-claude-pr-1676-merge-prep-1a0cui.md new file mode 100644 index 00000000..3690f123 --- /dev/null +++ b/docs/superpowers/plans/2026-09-02-claude-pr-1676-merge-prep-1a0cui.md @@ -0,0 +1,85 @@ +# Add a Mixed-portability closure for a Dimension-5-exempted every-use non-portable target + +**Goal:** `evaluating-skill-quality/references/rubric.md`'s Mixed-portability +rule (Step 4's Mixed bullet) requires a physical file-level split for a +skill's repository/platform-specific content. Dimension 5's own +cohesion-confirmed sequential-pipeline exemption (issue #1662) grades a +qualifying target on minimizing an already-irreducible common-case file +count, but does not mention the Mixed split rule at all -- so a +Dimension-5-exempted target whose non-portable content is itself every-use +(`skills/executing-a-branch-plan` is exactly this shape) has no +satisfiable, textually-compliant way to close a Mixed-portability finding: +splitting it out adds a 4th every-use file (undermining the exemption's own +minimized floor); folding it into the skill's non-every-use file +(`failure-and-recovery.md`) destroys that file's own "never read on an +ordinary clean run" contract and reopens dimension 5 outright. Add one +narrow, loophole-resistant substitute to the Mixed bullet, gated on two +independently-verifiable conditions, without touching the Mixed rule's +existing file-level-split requirement for the ordinary case, and without +reopening issue #1662's own exemption. Source: +https://github.com/tvna/gitapex/issues/1676. + +**Authorization record:** No approving comment on issue #1676 at plan time +(checked via `github:issue_read` method `get_comments`, empty result -- +opened the same session, by the repository owner). Branch 2 of the +Authorization gate applies: the active human operator's own opening turn +in this session instructed executing issue #1676's PR through to +just-before-merge, then explicitly confirmed (via in-session +`AskUserQuestion` exchanges) the specific scope: rubric.md substitute + +`executing-a-branch-plan` re-grade only, no code change to +`executing-a-branch-plan` itself, and no PR #1632 work bundled into this +session. + +**Structural precondition (issue #1306's own gate):** +`planning-a-branch-from-an-issue`'s Step 5 re-verification marker was +written to issue #1676's own body at 2026-09-02T15:04:52Z and confirmed +present via `gitapex_check_branch_plan_reverified.py` (PASS) before this +file was authored. + +**Threat-model triage (step 2):** Issue #1676 was read in full and its ACM +independently re-verified against current repo state (see the +re-verification marker's own findings on the issue). It is a well-formed, +professionally-scoped ACM issue authored by the repository owner +(`author_association: OWNER`), citing concrete line numbers and prior +issues/PRs (#1662, #1632, #1648, #730) throughout. Every ACM row's Planned +ops column describes a change to a named file or a verification step, not +an instruction directed at the executing agent. Clean. + +**Architecture:** Two tasks, two waves (sequential -- task B has an +interface dependency on task A's own edited rubric text). + +- Task A -- `skills/evaluating-skill-quality/references/rubric.md`: insert + one new nested Mixed-portability substitute bullet under the existing + Mixed bullet (after line 785), plus a one-clause cross-reference inside + the Dimension-5 exemption's own "still apply in full" parenthetical + (~1703-1706). `evals/evaluating-skill-quality/tasks/*.yaml` (two new + selection fixtures: a qualifying positive case and an + anti-loophole false-positive-attempt negative case). + `evals/evaluating-skill-quality/split.json` (register both, update + partition arithmetic). `evals/evaluating-skill-quality/results/- + issue-1676-*/manifest.json` (new gate run record, schema-validated). + `evals/evaluating-skill-quality/split.md` (Kept-edit-log entry). + `docs/skill-eval-status.md` (regenerated). `tests/ + test_gitapex_gate_split_fixture_coverage.py` (pinned partition-count + fixed). `skills/evaluating-skill-quality/metadata/gitapex.yaml` + (decision-log entry). +- Task B -- isolated re-grade of `executing-a-branch-plan`'s + Mixed-portability status against the new rubric (verification only, no + code change to that skill); records the verdict in + `skills/executing-a-branch-plan/metadata/gitapex.yaml`'s own decision + log. + +**Interface dependencies:** Task B reads Task A's own edited rubric.md +text before it can grade anything -- sequenced after it, never +co-assigned to the same wave. + +**Wave assignment:** wave 1 -- Task A. wave 2 -- Task B. + +**Proof method:** `scorer-gated-skill-edits`'s own held-out gate +(selection-split mean strictly increases before -> after on the primary +positive fixture; the negative fixture confirms no regression/loophole) +plus `gitapex_check_skill_shape.py` full run (70/70) plus +`.github/scripts/gitapex_gate_split_fixture_coverage.py` plus +`gitapex_scan_eval_suite_schema.py`. Task B's own proof method is the +isolated dispatch verdict itself, recorded structurally in the decision +log per this repository's own established convention. diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-after.md b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-after.md new file mode 100644 index 00000000..215ba0c5 --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-after.md @@ -0,0 +1,13 @@ +**5. Progressive disclosure.** Two sub-findings: + +*(a) Sequential-pipeline exemption — given, cleared.* Reused verbatim per the task's stipulation; not re-derived. + +*(b) Mixed-portability substitute — independently checked, fails.* The rubric's substitute text requires both: "(1) this same dimension-5 walk has already granted the target the cohesion-confirmed sequential-pipeline exemption immediately above ... and (2) the non-portable content is demonstrably reached and acted on, not merely read as inert text, on every ordinary origin-environment run, established by inspecting which procedure steps actually execute it, never accepted from the target's own 'every-use'/'interleaved' self-characterization -- a step whose own action is itself conditional (skipped on some runs, gated on a runtime value) fails this condition even when its surrounding prose is read every time." + +Condition (1) is satisfied (given). Condition (2) is not: the target's own Procedure step 4 reads, verbatim, "Only when the destination is ProviderX (this deployment's own default, but not its only configured destination), call ProviderX's proprietary providerx-ship binary to deliver the batch, using ProviderX's own authentication token format. For any other configured destination, this step is skipped entirely and the batch is delivered through the portable Procedure's own generic HTTP-POST fallback instead." That is a step whose own action is explicitly gated on a runtime value (the deployment's configured destination) and explicitly skipped on some runs -- the textbook fail case the rubric names. The Notes section's countervailing claim is exactly the "every-use"/"interleaved" self-characterization the rubric instructs must "never [be] accepted." + +Because only condition (1) is met, the rubric routes this target back to the ordinary Mixed rule. Checking that rule against the actual content: the ProviderX-specific instructions sit inline inside step 4 of the main numbered Procedure section of SKILL.md itself -- not isolated in a dedicated reference file. + +This matches dimension 5's own Fail bullet directly: "a Mixed-declared target claiming the Mixed-portability substitute above whose non-portable step is in fact conditional rather than executed every run (failing that substitute's own condition 2)." + +**Dimension 5 does not clear.** diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-before.md b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-before.md new file mode 100644 index 00000000..2c0f534f --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-false-positive-selection-before.md @@ -0,0 +1,92 @@ +# Review: `log-shipping-connector` SKILL.md (draft excerpt) — per `evaluating-skill-quality` + +**Mode note (read once, before the walk):** Per the task framing this is a bare-mode, single-file simulation — I have `combined.md` (SKILL.md + every `references/*.md` for `evaluating-skill-quality`) and the target excerpt only, no live subagent dispatch, no `scripts/gitapex_check_skill_shape.py` execution, no target `metadata/gitapex.yaml`, no target `references/*.md` content, and the target is explicitly labelled an **excerpt** (not confirmed to be the complete `SKILL.md`). Every gap this causes is named explicitly below rather than silently assumed away. + +Per the task's explicit instruction, two findings are taken as **already established, not re-derived**: (1) the cohesion check at Procedure step 2 returned single-outcome sequential cohesion confirmed; (2) dimension 5's own walk already confirmed the cohesion-confirmed sequential-pipeline exemption clears on the reference-file/`BODY_MAX_LINES` axis. Both are folded into the walk below at their normal positions, without re-litigation. + +## Precondition (Procedure steps 1 & 3) + +**Step 1 (read target).** Read in full: frontmatter, both Procedure sections (portable, steps 1-3; ProviderX-specific/non-portable, steps 4-5), and Notes. `references/batching-rules.md`, `references/retry-policy.md`, `references/delivery-confirmation.md`, and `references/porting-boundary-map.md` are cited but their content was not provided — findings below that would depend on their actual text are named as unverified, not assumed. + +**Step 3 (deterministic shape, hand-applied — script unavailable).** +- `description-present`: pass (non-empty). +- `description-no-xml`: pass (no tags). +- `name-pattern`: pass — `log-shipping-connector` is lowercase-hyphenated. +- `name-not-reserved`: pass. +- Forward slashes used throughout. +- Every sidecar-dependent check: not measured — no script, no confirmed-complete body, no sidecar, no reference-file content. Recorded as not-applicable/unmeasured, not as failures. + +No shape violation is observed in the checkable subset. This does not by itself license "well-formed." + +## Step 2 — Agentic operation mechanism-fit (headline-eligible) + +### Whole-artifact: Skill vs. hook — flag, candidate wrong-mechanism finding + +Walking the five numbered steps: (1) read local config, (2) apply batching-rules.md's "size/time-window rules," (3) compute a retry/backoff schedule "per its own rules," (4) a fixed binary-or-HTTP-POST branch keyed purely on a config value ("Only when the destination is ProviderX... For any other configured destination..."), (5) record the outcome. Every step is fully deterministic — none asks for judgment, interpretation, or open-ended reasoning. This is the exact shape SKILL.md's own Agentic operation mechanism-fit section warns about: "'Every time X, always do Y' in CLAUDE.md[, or a skill]. If the behavior should happen reliably ... use a hook ... instead." Reported as the review's headline finding per Stop boundaries: resolve why this must be a skill rather than a hook/script before treating any dimension score below as sufficient for shipping. + +### Step-level: Skill-step vs. bundled script + +Steps 2 and 3 are deterministic, multi-rule, error-prone-for-a-model shape the break-even test names. Step-level finding, not a headline blocker. + +### Step-level: Invocation-mode fit + +No `disable-model-invocation`/`user-invocable` declared. Effective mode is invocable by both. Step 4's ProviderX path transmits data externally using "ProviderX's own authentication token format" — outward-facing behavior with no stated rationale for leaving it open to autonomous model invocation. Flagged as a step-level finding. + +### Whole-artifact: cohesion — as given, not re-derived + +Per the task's stated fact: single-outcome sequential cohesion confirmed. No split finding. Taken as established. + +### Blind spot pass + +No dimension asks whether a data-shipping skill's retry logic is idempotent — a genuine gap named per the Unknowns framework. + +## Step 4 — Portability, capability assumption, dependency policy, compatibility, confidentiality + +### Portability level — independently established, not taken from the Notes at face value + +Reading the Procedure text directly: steps 1-3 and 5 resolve inside the skill's own folder against generic config/reference files; step 4 depends on ProviderX's proprietary binary and auth format. A real mix is exactly Mixed. I confirm Mixed, independently derived from content. + +But the Notes' supporting claim does not survive comparison against the Procedure text, exactly as the task warns. The Notes states: "Step 4's `providerx-ship` invocation is ProviderX-specific ... interleaved with the portable steps around it -- there is no caller-selectable narrower path that skips it." Step 4 itself states the opposite: "Only when the destination is ProviderX ... For any other configured destination, this step is skipped entirely and the batch is delivered through the portable Procedure's own generic HTTP-POST fallback instead." This is a direct, citable contradiction within the same document — the destination configuration is a caller-selectable narrower path that skips step 4. + +### Dimension 5 (Progressive disclosure) — consuming the Mixed precondition just established + +The given, not-re-derived finding clears one specific sub-question: the three mandatory-every-run reference files exceed BODY_MAX_LINES with no rearrangement able to lower that floor, so the cohesion-confirmed sequential-pipeline exemption licenses the multi-file read for the common case. That exemption answers only the file-count/length question. It does not touch the separate rule this precondition step's Mixed classification triggers: "Mixed -- dimension 5 (progressive disclosure) requires the actual split, not just the intent to split: the repository-specific part belongs in a clearly named reference file ... not blended into the portable core" (rubric.md, Portability level). As the task states, this rubric snapshot carries no substitute or exemption that lets a Dimension-5-exempted target off this separate requirement. + +Applying it: the vendor-specific detail (the providerx-ship binary name and "ProviderX's own authentication token format") is written inline in SKILL.md's own Procedure, under a heading that itself stays inside SKILL.md ("## Procedure (ProviderX-specific, non-portable)"), not delegated to a dedicated reference file. The one file that is pushed out, references/porting-boundary-map.md, is read "only when vendoring this skill elsewhere" and carries "its portable HTTP-POST substitute," i.e. the portable fallback, not the vendor-specific detail. + +**Dimension 5 verdict, stated explicitly: does not clear.** The sequential-pipeline sub-issue is cleared (given, accepted); the Mixed-portability blending sub-issue is not, and independently blocks a clean dimension-5 pass. Fails to clear. + +### Capability assumption, dependency policy, compatibility, confidentiality + +- **Capability assumption**: no sidecar; graded at the ungraded, no-declaration default. +- **Dependency policy**: not applicable. +- **Compatibility awareness**: NO_COMPATIBILITY_WARNING. +- **Confidentiality awareness**: fires. Application logs routinely carry secrets/PII, and the procedure ships them externally with no stated safeguard. PROPOSE_CONFIDENTIALITY_SAFEGUARD. + +## Step 5 — Nine-dimension walk + +**1. Discovery.** Pass, with a minor nit — near-restatement between description sentences. + +**2. Conciseness.** Pass on the visible text. + +**3. Degree of freedom.** Pass. Exact steps, exact reference reads, a binary-branch dispatch. + +**4. Clarity and structure.** Does not clear — the Notes section flatly contradicts step 4's own text on whether a caller-selectable narrower path exists. Secondary structural nit: step 5 (destination-independent) is filed under the "ProviderX-specific, non-portable" heading. + +**5. Progressive disclosure.** Does not clear, per the analysis above: cleared on the given file-count/BODY_MAX_LINES exemption; fails the separate, still-applicable Mixed-portability blending requirement. + +**6. Durability.** Mostly clean on classic bullets, but the Mixed portability Notes-disclosure expectation is undermined by the Notes' factual inaccuracy about step 4's conditionality — this dimension does not cleanly clear either pending that fix. + +**7. Bundled scripts.** Not applicable. + +**8. Behavioural evidence.** Unmeasured. + +**9. Cross-model robustness.** Unmeasured, qualitative read only. + +## Step 6 — Verdict + +**Agentic operation mechanism-fit:** headline concern raised, not resolved — this procedure's every step is deterministic, matching the rubric's own "should have been a hook" anti-pattern. + +**Well-formed:** not fully determinable in this bare-mode run; no violation found in the hand-checkable subset. + +**Mature: no.** Dimensions 4 and 5 both carry named, uncleared gaps (Notes/Procedure self-contradiction; Mixed-portability content blended into the portable core rather than split), which alone blocks Mature. Cohesion (given) and the dimension-5 file-count sub-issue (given) are not what's holding this below Mature; the independent Mixed-blending finding and the Notes/Procedure contradiction are. diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-after.md b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-after.md new file mode 100644 index 00000000..404ae01c --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-after.md @@ -0,0 +1,92 @@ +# Review: vault-secret-rotation (excerpt) — via evaluating-skill-quality + +**Run-mode disclosure:** This is a bare-mode, zero-further-tool-access simulation. I read exactly one file (`evaluating-skill-quality`'s combined SKILL.md + references/*.md) and applied its Procedure to the task text given. + +## Procedure step 1 — target read + +Read in full: the `vault-secret-rotation` SKILL.md excerpt (frontmatter, five-step Procedure across three headings, Notes) and the supplied full content of `references/porting-boundary-map.md`. No traversal/symlink/execution risk applies to pasted text. + +## Procedure step 2 — Agentic operation mechanism-fit, cohesion, Blind spot pass + +**Mechanism-fit:** A multi-step procedure with fragile, irreversible content (credential rotation) — correctly a skill, not CLAUDE.md content. + +**Cohesion:** *Given, not re-derived per the task's instruction*: single-outcome sequential cohesion confirmed. + +**Blind spot pass:** No rubric gap specific to this target's domain was found beyond what Confidentiality awareness already covers. + +## Procedure step 4 — Portability, capability assumption, dependency policy + +- **Portability, read from content: Mixed.** The excerpt's own Notes states this explicitly, and the content supports it. +- **Capability assumption:** undeclared. Defaults to Frontier-level strictness for dimensions 2/3/5/9. +- **Dependency policy:** not applicable. + +## Procedure step 5 — Nine dimensions + +**1. Discovery.** Clears. + +**2. Conciseness.** Not a Fail. Clears. + +**3. Degree of freedom.** Pass. + +**4. Clarity and structure — one genuine finding.** Step 4's own text is internally inconsistent about invocation count. This is a genuine, citable minor imprecision — this dimension does not cleanly clear. It does not, however, undermine the conditionality finding in dimension 5 below. + +**5. Progressive disclosure — the requested focus.** + +*Common-case file count / cohesion-confirmed sequential-pipeline exemption*: given, not re-derived, already cleared per the task. + +*Mixed-portability substitute — independently verified against the excerpt text, not accepted from the target's self-characterization*: + +- **Condition 1** (dimension-5 already granted the cohesion-confirmed sequential-pipeline exemption): satisfied — reused from the given finding above. +- **Condition 2** (the non-portable content is reached and acted on, unconditionally, on every ordinary run — not merely read as inert text, and not itself gated on a runtime value). I checked which steps actually execute `vaultco-cli`: only step 4 — step 5 explicitly disclaims it. I then checked whether step 4's own execution is conditional. The text's "if any dependent service fails to acknowledge within VaultCo's own webhook timeout" clause gates only the internal outcome of an already-issued call, not whether the call is issued. No branch in the five-step Procedure skips step 4 on any stated runtime condition. **Condition 2 holds** — the invocation itself is unconditional. +- **Third positive requirement — dedicated reference file, independently confirmed rather than accepted on claim.** The task supplied `references/porting-boundary-map.md`'s actual content, which I inspected directly: it states "Read only when vendoring vault-secret-rotation outside a VaultCo deployment -- never on an ordinary run", and its one-row table names the single touchpoint that exists against a concrete, non-vacuous portable substitute — genuinely enumerating the one touchpoint condition-2 verification above found exists, not a placeholder. +- **First positive requirement — isolation under distinct headings, not blended.** Three headings correctly separate the content. +- **Second positive requirement — Notes declaration naming the executing step and its fallback.** The Notes names step 4 by number and states the fallback in substance. + +**Verdict for this check: the Mixed-portability substitute's two gating conditions and all three positive requirements independently hold against the actual excerpt text.** This target is not required to physically relocate step 4's content into an every-use split — dimension 5 clears on this specific question, graded here (not at the Portability-level classification step) per the rubric's own routing. + +**6. Durability.** Clears. + +**7. Bundled scripts.** Not applicable. + +**8. Behavioural evidence.** Unmeasured. + +**9. Cross-model robustness.** Unmeasured. + +## Compatibility and Confidentiality awareness + +- **Compatibility awareness: NO_COMPATIBILITY_WARNING.** +- **Confidentiality awareness: PROPOSE_CONFIDENTIALITY_SAFEGUARD.** + +## Verdict + +- **Well-formed:** not confirmable with certainty in this bare-mode run. +- **Mechanism-fit / cohesion:** no wrong-mechanism or split finding. +- **Dimension-5 Mixed-portability substitute:** passes, both conditions and all three positive requirements independently verified against the actual text rather than accepted on the target's self-characterization. +- **Mature:** blocked — not by the Mixed-portability substitute (which clears), but by the dimension-4 "exactly once" vs. per-credential invocation-count inconsistency, a genuine minor gap. +- **Indeterminate:** does not apply. + +```json +{ + "target": "vault-secret-rotation (SKILL.md excerpt)", + "dimension_5_mixed_portability_substitute": { + "condition_1_cohesion_exemption_reused": "satisfied (given)", + "condition_2_unconditional_execution": "verified true", + "positive_requirement_isolation": "satisfied", + "positive_requirement_notes_declaration": "satisfied", + "positive_requirement_dedicated_reference_file": "satisfied", + "result": "pass" + }, + "dimensions": { + "1_discovery": "clears", + "2_conciseness": "clears", + "3_degree_of_freedom": "clears", + "4_clarity_and_structure": "does not clear", + "5_progressive_disclosure": "clears", + "6_durability": "clears", + "7_bundled_scripts": "not applicable", + "8_behavioural_evidence": "unmeasured", + "9_cross_model_robustness": "unmeasured" + }, + "verdict": "Mature: blocked by dimension 4 minor gap; Well-formed: unconfirmed; Mixed-portability substitute at dimension 5: pass" +} +``` diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-before.md b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-before.md new file mode 100644 index 00000000..4cb02bff --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/mixed-portability-dimension5-substitute-selection-before.md @@ -0,0 +1,121 @@ +# Review: `vault-secret-rotation` (evaluating-skill-quality applied) + +## Scope and run-mode caveats (stated up front, per Contract discipline) + +This is a bare-mode, zero-further-tool-access run reviewing a **draft excerpt**, not a full skill directory. Before walking the Procedure I record what this run cannot do, rather than silently filling the gaps: + +- **Procedure step 3 (deterministic shape checker)** cannot be executed — no script/file-system access to `gitapex_check_skill_shape.py`, and the excerpt gives no full frontmatter/line counts to hand-apply its rules against. Reported as **shape-check status: unavailable in this run**, not fabricated as PASS or FAIL. +- **Subagent dispatch / isolation verification** (`SKILL.md`'s Subagent dispatch section, `adversarial-self-audit.md`'s Isolation verification) cannot actually be performed — there is no dispatch mechanism available in this simulation. Noted as a limitation, not silently assumed clean. +- No `metadata/gitapex.yaml` sidecar was given. Per rubric.md's Capability assumption / Procedure step 4: *"When the target has no sidecar … establish portability and capability assumption by reading the target's content instead … and note the sidecar's absence as context, not as a finding."* Done below. +- Given, not re-derived (per task instructions): the **cohesion finding** (single-outcome sequential cohesion) and the **dimension-5 sequential-pipeline exemption** for the three mandatory reference files. Both are treated as established facts from here on. + +## Step 1 — Read + +Target read in full as given: frontmatter (`name`, `description`), a five-step Procedure split across three headed subsections, a `## Notes` section declaring portability, and the full content of `references/porting-boundary-map.md`. No other `references/*.md` content (e.g. `rotation-policy.md`) was supplied, so any claim about *their* content is explicitly unverifiable and not asserted. + +## Step 2 — Agentic operation mechanism-fit + +**Whole-artifact wrong-mechanism check:** nothing in the excerpt suggests this should have been a hook, subagent, or CLAUDE.md content — it is a multi-step judgment+external-call procedure, appropriate for a skill. No finding. + +**Cohesion (Skill vs. multiple skills):** per the task's own preamble, already confirmed at this step as single-outcome sequential cohesion converging on *"every credential due for rotation is rotated in the vault and every dependent service is updated to the new value, with no service left holding a stale credential."* Not re-derived here. + +**Step-level checks:** + +- **Invocation-mode fit — Fail, unguarded side effects (step-level).** No `disable-model-invocation` or `user-invocable` field appears anywhere in the given frontmatter, so the effective mode is the default: invocable by both. The procedure's outward-facing/irreversible action is exactly the shape rubric.md names. Rotating every due credential across an org's vault and pushing new values to every dependent service is materially the same blast-radius class as `/deploy`. Auto-rollback on webhook-timeout mitigates the *failure* case but does not remove the outward-facing write itself. No justification for open model-invocation is stated anywhere in the excerpt. **Propose:** `disable-model-invocation: true`, or an explicit stated reason for leaving it open. + +- **Model/effort tier fit, Subagent delegation scope, Tool-capability verification:** not applicable — no model/effort pin, no subagent-dispatch instruction, and `vaultco-cli` is an externally-invoked CLI, not a harness-surfaced tool/MCP subcall with a schema this check is scoped to. Noting by reasoned analogy, not as this check's literal Fail: the claim that `--auto-rollback` *"atomically reverts to the prior value, internally, if any dependent service fails to acknowledge"* is asserted as flat fact about a third-party binary's behavior with no citation to VaultCo's own docs — worth a hedge on the same principle even though the check's stated applicability trigger doesn't literally cover it. + +- **Skill-step vs. bundled script — candidate, unconfirmed.** Step 2's *"compute each credential's own new value per its rotation policy"* is exactly the shape this check flags as a delegation candidate. If `rotation-policy.md`'s rules are mechanical this is a real candidate; its content was not supplied, so this is flagged as **worth checking against that file**, not a confirmed Fail. + +**Blind spot pass:** naming a gap, not folding it into an existing dimension. None of the nine dimensions, mechanism-fit, or portability asks whether the procedure verifies that each dependent service's *own* credential store actually converged on the new value, as opposed to merely acknowledging VaultCo's webhook — webhook ack and actual downstream consistency are different facts. + +## Step 3 — Deterministic shape + +Not run (see Scope caveats). Reported as **unavailable**, not PASS/FAIL. + +## Step 4 — Portability / Capability / Dependency preconditions; Compatibility & Confidentiality + +**Portability — read as Mixed** (declared directly in `## Notes`: *"Portability: **Mixed**."*), consistent with `SKILL.md`'s own three-level definition: *"Mixed: a portable core plus repo-specific detail should split the two into a clearly named reference file, not blend them."* Graded in full below (dimension 5 is where this gets checked). + +**Capability assumption — undeclared / not established** from the given excerpt (no sidecar, no stated Broad/Frontier/Adaptive). Per rubric.md dimension 2's own rule, this means dimensions 2/3/5/9 grade at *"the ungraded, no-declaration default — equivalent to Frontier-level strictness."* Applied throughout step 5. + +**Dependency policy:** not applicable — no `scripts/` shown. + +**Declaration-vs-pin consistency:** no pin exists (step 2 found none), so no contradiction to check. + +**Compatibility awareness — `NO_COMPATIBILITY_WARNING`.** The given frontmatter carries only `name`/`description`; no runtime-specific field is present in what was supplied, so no runtime-specific dependency is established from this excerpt. + +**Confidentiality awareness — `PROPOSE_CONFIDENTIALITY_SAFEGUARD`.** Applicability is squarely met — this skill's entire procedure is credential handling: enumerating credentials, computing new values, writing them, and recording rotation outcomes. No step in the excerpt states any safeguard. **Proposed fix**, targeting step 5 specifically: state explicitly that the audit trail records the credential's identifier and outcome status only, and must never write the new credential value itself. + +## Step 5 — Nine dimensions + +**1. Discovery.** `description` states both what and when in concrete, vault/credential-specific terms unlikely to collide with an unrelated sibling. `name` reads as a noun phrase, acceptable per rubric. Grading the trigger against the invocation mode established at step 2 (invocable by both, no dead-trigger issue): trigger is reachable. **Clears** — the separate unguarded-side-effects concern is Agentic-operation-mechanism-fit's finding, not re-counted here, per Contract discipline's "never both." + +**2. Conciseness.** Body is compact and domain-specific; no re-teaching of well-known concepts, evaluated at the ungraded Frontier-equivalent default established at step 4. One soft finding: the Notes-section restatement of "step 4 alone is VaultCo-specific and runs unconditionally every run" duplicates content already asserted inside step 4 itself. This is a defensible split (operational instruction vs. portability rationale), so I flag it as a **minor** duplication candidate rather than a hard Fail. + +**3. Degree of freedom.** Step 4 is pinned to an exact, single, unconditional command for a fragile, irreversible-adjacent operation — correct match per rubric.md's fragility test. Steps 1–3/5 are medium-freedom, driven by mandatory reference-file policy rather than open prose. **Clears.** + +**4. Clarity and structure.** Single linear pipeline (consistent with the given cohesion finding), consistent terminology (credential / rotation / dependent service throughout), each step names an observable completion result. No competing branch triggers to disambiguate, consistent with confirmed sequential cohesion. **Clears** on what's shown. + +**5. Progressive disclosure — does not clear.** This is the central finding, and the one the task's framing is testing directly. + +The excerpt correctly applies, and explicitly invokes, the given dimension-5 sequential-pipeline exemption for the three mandatory reference files' combined 690 lines. That exemption is scoped narrowly, in its own text, to exactly one question: reference-file line count. Both conditions are satisfied for the three reference files, and I am not re-deriving or contesting that — it was given. But this exemption is about whether mandatory reference-file line count may exceed the body cap; nothing in its text touches the separate, ordinary Mixed-portability rule, stated plainly in `SKILL.md`: *"Mixed: a portable core plus repo-specific detail should split the two into a clearly named reference file, not blend them."* + +Applying that rule to the actual artifact: step 4's full operational content — the exact `vaultco-cli rotate --id --auto-rollback` invocation, its webhook-propagation mechanism, and its auto-rollback semantics — is **not** in a reference file. It lives directly inside `SKILL.md`'s own body, under `## Procedure (VaultCo-specific, non-portable)`, sandwiched between `## Procedure (portable)` and `## Procedure (portable, continued)` — the same file, read on every ordinary run. The one reference file that names this touchpoint, `references/porting-boundary-map.md`, is explicitly excluded from the ordinary-run path by its own text: *"Read only when vendoring vault-secret-rotation outside a VaultCo deployment -- never on an ordinary run."* + +So on the path every real invocation actually takes, 100% of the non-portable operational content is read from `SKILL.md` itself; nothing defers to a reference file for that content. That is the textbook shape of *blend*, not *split* — the exact defect the Mixed rule exists to catch, and the dimension-5 Fail example names it directly. + +Note what that Fail bullet's own scope is: it fails a *misuse of the sequential-pipeline exemption itself*. It says nothing about, and grants no separate exemption for, leaving Mixed-declared non-portable content unsplit. As the task states, and as I confirm from the file actually read: no Mixed-portability substitute for a dimension-5-exempted target exists anywhere in this version of the rubric. Extending the confirmed exemption to also excuse the unrelated Mixed-split requirement would be inventing a rule the file does not contain — exactly what I am not doing here. + +This is graded once, under dimension 5, per Contract discipline's "never both." + +**Concrete remediation:** move step 4's operational detail into a new mandatory-every-run reference file (e.g. `references/vaultco-rotation-command.md`), leaving `SKILL.md` step 4 as a short portable pointer. + +**6. Durability.** Forward slashes throughout, no bare issue/PR citations, no time-sensitive content, no MCP tool citations to check. The Mixed-split issue is dimension 5's alone (see above); not re-raised here. **Clears** on the criteria dimension 6 itself owns. + +**7. Bundled scripts.** Not applicable — no `scripts/` shown in the excerpt. + +**8. Behavioural evidence.** Cannot check the target repository for an eval mechanism in this bare-mode, zero-further-tool-access run. Stated explicitly as **unmeasured**, not assumed either way. + +**9. Cross-model robustness.** Unmeasured for the same reason. Qualitative read only, labeled as a read, not evidence. + +## Step 6 — Verdict + +- **Well-formed:** cannot be confirmed — shape checker unavailable in this run (see Scope caveats). Not fabricated as PASS or FAIL. +- **Mature: does not clear.** Per rubric.md's Verdicts section, *"Mature -- well-formed, and every dimension 1-7 clears cleanly with no named gap."* Dimension 5 carries a named gap (Mixed-portability content blended into `SKILL.md` rather than split into a reference file, per the analysis above) that is independent of, and not covered by, the given dimension-5 sequential-pipeline exemption. That alone blocks Mature regardless of well-formedness. Dimensions 8–9 are separately unmeasured (permitted for those two specifically, per the same section). +- Step-level findings reported for triage, not blocking on their own: Invocation-mode-fit (unguarded side effects — propose `disable-model-invocation: true` or a stated reason), a candidate Skill-step-vs-bundled-script check on step 2's value-computation (unconfirmed, `rotation-policy.md` not supplied), and a minor dimension-2 duplication note. +- Warning-only axes (do not affect the verdict): Compatibility awareness — `NO_COMPATIBILITY_WARNING`; Confidentiality awareness — `PROPOSE_CONFIDENTIALITY_SAFEGUARD`. +- Blind spot: named above (webhook-ack vs. actual dependent-service consistency verification) — not folded into an existing dimension. + +```json +{ + "target": "vault-secret-rotation (excerpt)", + "well_formed": "unverifiable_in_this_run", + "mature": false, + "headline_finding": null, + "cohesion": "given_not_rederived: single-outcome sequential, confirmed", + "dimension5_body_cap_exemption": "given_not_rederived: applies to the three mandatory reference files only", + "dimensions": { + "1_discovery": "clear", + "2_conciseness": "clear_minor_note", + "3_degree_of_freedom": "clear", + "4_clarity_structure": "clear", + "5_progressive_disclosure": "fail", + "6_durability": "clear", + "7_bundled_scripts": "not_applicable", + "8_behavioural_evidence": "unmeasured", + "9_cross_model_robustness": "unmeasured" + }, + "dimension5_fail_reason": "Mixed-declared non-portable content (step 4) is read and executed directly from SKILL.md's own body on every ordinary run, not split into a reference file; references/porting-boundary-map.md is explicitly excluded from the ordinary-run path. The given dimension-5 sequential-pipeline exemption governs only the three mandatory reference files' combined line count and grants no license for this separate, ordinary Mixed-portability split requirement.", + "step_level_findings": [ + "invocation_mode_fit: unguarded_side_effects", + "skill_step_vs_bundled_script: candidate_unconfirmed (step 2 value computation)", + "conciseness: minor_duplication_candidate (Notes vs. step 4)" + ], + "compatibility_awareness": "NO_COMPATIBILITY_WARNING", + "confidentiality_awareness": "PROPOSE_CONFIDENTIALITY_SAFEGUARD", + "blind_spot": "no dimension checks whether dependent-service webhook acknowledgment implies actual downstream credential-store convergence", + "shape_check": "not_run_in_this_environment", + "note": "output-schema.json was not included among the concatenated .md reference files read for this run, so this block is a best-effort structured summary, not validated against that schema." +} +``` diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/transfer-check-sequential-pipeline-body-cap-exception-selection-after.md b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/transfer-check-sequential-pipeline-body-cap-exception-selection-after.md new file mode 100644 index 00000000..7c3bc45b --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/artifacts/transfer-check-sequential-pipeline-body-cap-exception-selection-after.md @@ -0,0 +1,40 @@ +# Dimension 5 (Progressive disclosure) Verdict — warehouse-inventory-reconciliation + +## Verdict: PASS — clears via the sequential-pipeline exemption + +**Reasoning, per `references/rubric.md`'s dimension 5 section:** + +The exemption bullet reads: + +> "A cohesion-confirmed, single-outcome sequential- or functional-cohesion orchestrator whose every-use content genuinely exceeds the body cap is a distinct case from an ordinary multi-file split, but only under a narrow, stated condition... The exemption applies only when both (1) the cohesion check has already confirmed, at its own Procedure step 2, that the target's steps are single-outcome sequential or functional cohesion — reused from that finding, never re-derived here... and (2) the target's combined every-use reference content, measured in lines, demonstrably exceeds `BODY_MAX_LINES` even after every dimension-2 padding cut... has actually been applied to that content directly." + +Applying both conditions to the given facts (taken as established, not re-derived, per the task's own instruction): + +- **Condition 1** — satisfied. The task states this target's own cohesion check already ran at its Procedure step 2 and "returned single-outcome sequential cohesion confirmed," with each step's output required as the next step's input and no caller-selectable narrower path. This is exactly the finding condition 1 requires, reused rather than re-derived here. +- **Condition 2** — satisfied. The three mandatory-every-run reference files total 640 lines against a 500-line `BODY_MAX_LINES` cap, and the task states every line is "genuinely load-bearing domain policy," with "none of it... restatement, throat-clearing, or padding a dimension-2 cut could remove, and none of it could be inlined into `SKILL.md` without blowing the cap outright." That is condition 2's "demonstrably exceeds `BODY_MAX_LINES` even after every dimension-2 padding cut... has actually been applied" bar, met as given. + +Both conditions holding, the rubric's own instruction is to grade the split's file-count minimization against the irreducible floor rather than fail outright: + +> "A skill meeting both conditions is graded on whether its split minimizes the common-case file count given that irreducible floor (co-location, content-named files, a `SKILL.md` pointer at each branch point... still apply in full) rather than being failed outright for a floor no rearrangement can lower." + +Checking that minimization against the excerpt: +- **Content-named files**: `count-reconciliation-rules.md`, `discrepancy-resolution-policy.md`, `audit-trail-requirements.md` are named for their domain content, not `doc1.md`/`doc2.md`. +- **Co-location**: each file is read at exactly the step that needs it (matching rules at step 2, resolution authority at step 4, retention rules at step 6) — not bundled together or read out of sequence. +- **`SKILL.md` pointer at each branch point**: each step states what the read is for and what it produces ("apply its matching rules to produce a discrepancy list," "resolve each classified discrepancy... per its authority-threshold rules," "record an audit entry for every resolution... and correction," closing the reconciliation) — not a bare "see reference." +- **File count**: exactly one file per step that needs one, no more — the minimal split given three genuinely distinct, non-mergeable policy domains (reconciliation math, resolution authority, retention rules), each independently irreducible per the given facts. + +No element of the dimension-5 **Fail** bullet applies: the exemption claim is backed by an independently-produced cohesion-check finding (condition 1), and the content does not fit under `BODY_MAX_LINES` even after dimension-2 cuts (condition 2) — the two ways this exemption specifically fails are both foreclosed by the givens. + +**Result: dimension 5 clears via the sequential-pipeline exemption.** + +## Mixed-portability substitute bullet — not invoked + +Grading this target required **no reference to** the new Mixed-portability substitute bullet. That bullet lives in the Portability level section's Mixed sub-bullet, and dimension 5's own parenthetical is explicit that it is conditional on declaration: + +> "(co-location, content-named files, a `SKILL.md` pointer at each branch point, and -- for a Mixed-declared target -- the Mixed-portability substitute in Step 4's own Mixed bullet above still apply in full)" + +Since this target is stipulated **Portable-declared, not Mixed**, only the first three (unconditional) elements of that parenthetical apply — co-location, content-named files, and a `SKILL.md` pointer at each branch point — exactly as graded above. The Mixed-portability substitute clause itself (the one added for Mixed-declared targets whose non-portable content is every-use, under the Portability level section) was never opened, cited, or needed to reach this verdict. + +## Confirmation + +This grading exercised only the pre-existing sequential-pipeline exemption (dimension 5's own long-standing bullet, unchanged in substance by the Mixed-portability addition) and confirms no regression: an ordinary Portable target with this fixture shape clears dimension 5 exactly as the pre-existing exemption's own precedent (issue #1662's fixture) would predict, with the newly-added Mixed-only substitute bullet correctly inert for it. diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-after.json b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-after.json new file mode 100644 index 00000000..961e8e80 --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-after.json @@ -0,0 +1,17 @@ +{ + "model_id": "claude-sonnet-5", + "n_fixtures": 2, + "mean_score": 1.0, + "scores": [ + { + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-selection", + "score": 1.0, + "split": "selection" + }, + { + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-false-positive-selection", + "score": 1.0, + "split": "selection" + } + ] +} diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-before-after-detail.json b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-before-after-detail.json new file mode 100644 index 00000000..304b4aa8 --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/claude-sonnet-5-before-after-detail.json @@ -0,0 +1,42 @@ +{ + "model_id": "claude-sonnet-5", + "n_runs": 4, + "scores": [ + { + "iteration": "mixed-portability-dimension5-substitute-selection", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-selection", + "split": "selection", + "condition": "before", + "commit": "b07fabc53f2bee4f4404fbe7f7620426e60363d6", + "score": 0.666667, + "artifact": "artifacts/mixed-portability-dimension5-substitute-selection-before.md" + }, + { + "iteration": "mixed-portability-dimension5-substitute-selection", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-selection", + "split": "selection", + "condition": "after", + "commit": "eced6acde2e8f251ab87e3a326cd2f320ba9dcec", + "score": 1.0, + "artifact": "artifacts/mixed-portability-dimension5-substitute-selection-after.md" + }, + { + "iteration": "mixed-portability-dimension5-substitute-false-positive-selection", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-false-positive-selection", + "split": "selection", + "condition": "before", + "commit": "b07fabc53f2bee4f4404fbe7f7620426e60363d6", + "score": 1.0, + "artifact": "artifacts/mixed-portability-dimension5-substitute-false-positive-selection-before.md" + }, + { + "iteration": "mixed-portability-dimension5-substitute-false-positive-selection", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-false-positive-selection", + "split": "selection", + "condition": "after", + "commit": "eced6acde2e8f251ab87e3a326cd2f320ba9dcec", + "score": 1.0, + "artifact": "artifacts/mixed-portability-dimension5-substitute-false-positive-selection-after.md" + } + ] +} diff --git a/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/manifest.json b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/manifest.json new file mode 100644 index 00000000..1da215ad --- /dev/null +++ b/evals/evaluating-skill-quality/results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/manifest.json @@ -0,0 +1,86 @@ +{ + "date": "2026-09-02", + "issue": "https://github.com/tvna/gitapex/issues/1676", + "commit": "b07fabc53f2bee4f4404fbe7f7620426e60363d6", + "record_contract": "gate-run", + "runner": { + "name": "evals/scripts/gitapex_run_eval_suite.py", + "version": "a045f8b967ae042d8c3e8f5829ccf5d2e81520d1" + }, + "fixture_set": "2 new selection fixtures under evals/evaluating-skill-quality/tasks/: mixed-portability-dimension5-substitute-selection.yaml (the primary, formally gated positive/qualifying case) and mixed-portability-dimension5-substitute-false-positive-selection.yaml (an anti-loophole discrimination fixture confirming the new substitute's own anti-self-assertion condition 2 cannot be defeated by a target's own Notes-section self-characterization). Both gate dimension 5's (Progressive disclosure) new Mixed-portability substitute for a Dimension-5-exempted target whose non-portable content is itself every-use.", + "trials_per_fixture": 1, + "models": { + "default": "claude-sonnet-5" + }, + "dispatch_mechanism": "NOT the runner named above's own live `claude` CLI subprocess -- Step 1 confirmed that runner's identity/version firsthand (same commit a045f8b9 as issue #1662's own precedent, no drift), but a live invocation attempt in this session's own environment (`claude --bare --tools \"\" -p \"Say OK\"`) failed immediately with an `Authentication error`, matching the already-tracked, repo-wide, root-cause-unconfirmed infrastructure defect in issue #1304 -- independently reconfirmed this session, the same signature issue #1662's own run record already disclosed. Substituted with the Agent tool's own isolated (no filesystem write access requested, Read/Glob/Grep-only subagent type) subagent dispatch: each trial read a single pre-combined text file (this repository's own SKILL.md + references/*.md for evaluating-skill-quality, concatenated via shell redirection into a scratch file, then read by the dispatch via its own Read tool call -- never pasted inline by the caller) and was instructed to reason only from that file plus the fixture's own task prompt, with no further tool access, simulating the runner's own bare-mode contract. This is a disclosed substitute measurement, not the confirmed runner's own live scoring path -- see known_gaps.", + "scorer": "skills/scorer-gated-skill-edits/scripts/gitapex_score_contract.py, scoring each fixture's own output_contains/output_not_contains assertions deterministically, run directly against the saved dispatch transcripts (not re-typed or paraphrased by the caller for scoring purposes). Primary fixture: `gitapex_score_contract.py --compare-to 0.666667`, score 1.000000 -> `1.000000 KEEP`.", + "iterations": [ + { + "label": "mixed-portability-dimension5-substitute-selection", + "rubric_change": "Adds a new Mixed-portability substitute bullet inside references/rubric.md's own Dimension 5 (Progressive disclosure) section, immediately after the existing cohesion-confirmed sequential-pipeline exemption bullet it depends on, for a target that already clears that exemption and whose non-portable content is itself every-use. Gated on two independently-verifiable conditions -- (1) this same dimension-5 walk has already granted the sequential-pipeline exemption, reused rather than re-derived; (2) the non-portable content is demonstrably reached and acted on, not merely read as inert text, on every ordinary origin-environment run, established by inspecting which procedure steps actually execute it, never accepted from the target's own 'every-use'/'interleaved' self-characterization -- satisfied via three positive requirements (distinct-heading isolation; a SKILL.md Notes declaration naming non-portable steps and their portable fallback; one dedicated, non-every-use reference file, independently confirmed to actually exist and to genuinely enumerate every touchpoint and its substitute, not accepted from the target's own claim) instead of ordinary file-level physical relocation. The Portability level section's own pre-existing Mixed bullet is left with only a short forward-pointer sentence; its own five original lines are otherwise byte-identical to commit b07fabc5. Relocated here (commit eced6acd) from an initial placement directly under the Portability level section's Mixed bullet (commit 98f6c314) after this branch's own mandatory Step 8 independent adversarial review found that placement structurally broken -- see known_gaps.", + "before_commit": "b07fabc53f2bee4f4404fbe7f7620426e60363d6", + "after_commit": "eced6acde2e8f251ab87e3a326cd2f320ba9dcec", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-selection", + "before_score": 0.666667, + "after_score": 1.0 + }, + { + "label": "mixed-portability-dimension5-substitute-false-positive-selection", + "rubric_change": "Same edit as above. This fixture is a negative/anti-loophole control, not itself a gated movement fixture: its own target self-characterizes non-portable content as \"every-use\"/\"interleaved\" in its Notes while its own Procedure text shows the content is actually conditional (skipped for a non-matching destination), directly testing the new substitute's own condition-2 anti-self-assertion requirement. Correctly fails both before and after the edit, and both before and after the Step 8 relocation -- the point of this fixture is that the new substitute must not be defeatable by a self-serving Notes claim regardless of which rubric section it is graded under.", + "before_commit": "b07fabc53f2bee4f4404fbe7f7620426e60363d6", + "after_commit": "eced6acde2e8f251ab87e3a326cd2f320ba9dcec", + "fixture_id": "evaluating-skill-quality-mixed-portability-dimension5-substitute-false-positive-selection", + "before_score": 1.0, + "after_score": 1.0 + } + ], + "score_files": [ + { + "model_id": "claude-sonnet-5", + "file": "claude-sonnet-5-after.json" + } + ], + "gate": { + "verdict": "KEEP", + "candidate_class": "ordinary", + "split": "selection", + "prior_mean": "0.666667", + "candidate_mean": "1.000000" + }, + "artifacts": [ + "claude-sonnet-5-after.json", + "claude-sonnet-5-before-after-detail.json", + "artifacts/mixed-portability-dimension5-substitute-selection-before.md", + "artifacts/mixed-portability-dimension5-substitute-selection-after.md", + "artifacts/mixed-portability-dimension5-substitute-false-positive-selection-before.md", + "artifacts/mixed-portability-dimension5-substitute-false-positive-selection-after.md", + "artifacts/transfer-check-sequential-pipeline-body-cap-exception-selection-after.md" + ], + "nonstandard_score_files": [ + { + "file": "claude-sonnet-5-before-after-detail.json", + "deviation": "Scored by the same scorer a standard record uses, but before/after per-iteration (before_commit/after_commit, before_score/after_score) rather than eval-scores.schema.json's flat one-score-per-fixture_id shape -- same declared deviation as issue #1662's own precedent record and the several before it. This is the file the gate[] verdict above was actually computed from (on the primary fixture only); claude-sonnet-5-after.json (score_files[]) is a same-run, standard-shape derivative summarizing only the final AFTER-edit score for both fixtures." + } + ], + "known_gaps": [ + "Step-ordering correction, found live during this branch's own mandatory Step 8 independent adversarial review (executing-a-branch-plan's own gate, distinct from this skill's own recommended Step 8 pass), after the fixtures below had already been gated once against an earlier placement: the substitute's first placement lived directly under the Portability level section's own Mixed bullet, graded at SKILL.md Procedure step 4 -- but its own condition 1 required a dimension-5 finding (step 5) that had not run yet at step 4, a genuine backward-reference defect confirmed against SKILL.md's own literal Procedure step-order annotations before acting on it, not assumed. Fixed by relocating the full substitute into dimension 5's own section (commit eced6acd), immediately after the sequential-pipeline exemption it depends on; the Portability level section keeps only a short forward-pointer sentence. Both fixtures were re-dispatched against the relocated rubric text and re-scored; the numbers in this record are the post-relocation numbers, not the original placement's.", + "Two further wording corrections landed in the same relocation pass, both found by the same Step 8 review: condition 2's original 'read' wording did not close its intended loophole (a conditionally-executed step could still claim to satisfy a mere 'read' test), fixed to require the content be demonstrably reached and acted on; and the third positive requirement (the dedicated reference file) was originally satisfiable by the target's own bare claim with no supplied content to check it against, fixed by adding the same anti-self-assertion discipline condition 2 already carries, and by supplying mixed-portability-dimension5-substitute-selection.yaml's own dedicated-file content directly in its fixture prompt so this can actually be verified rather than assumed.", + "Assertion-wording correction, found live while re-scoring the relocated rubric against real dispatch transcripts: the primary fixture's prior output_contains string ('Mixed-portability substitute', itself a replacement for an even earlier near-verbatim quote that also failed to survive paraphrasing) still under-discriminated once condition 2's own wording changed -- generic negative guards considered alongside it ('does not apply', 'is conditional') each collided with an innocuous, unrelated phrase in the correct PASS transcript. Replaced with 'Condition 2 holds', empirically confirmed via grep against all four saved transcripts (positive-before: 0 occurrences, positive-after: 1, false-positive-before: 0, false-positive-after: 0) to discriminate perfectly before this record was finalized.", + "Fixture-authoring correction, found live during the original (pre-relocation) gate run: the primary fixture's own first draft had its non-portable step invoke a rollback command conditionally ('for any dependent service that does not acknowledge... within timeout'), which an after-edit dispatch correctly identified as failing the substitute's own condition 2 -- a genuine defect in the fixture's own scenario design, not a rubric defect. Fixed by folding the rollback into a single, unconditional invocation, removing the conditional branch entirely. The discarded draft's own transcripts are not included in this record's artifacts/.", + "The confirmed runner (evals/scripts/gitapex_run_eval_suite.py, commit a045f8b9) could not actually be executed in this session's environment -- a live `claude --bare --tools \"\" -p` invocation failed with an `Authentication error`, matching issue #1304's already-tracked, repo-wide, root-cause-unconfirmed defect, independently reconfirmed this session (the same signature issue #1662's own run record already disclosed, now reconfirmed a second time rather than assumed still true). Scoring was performed instead via isolated Agent-tool subagent dispatch reasoning over a file the dispatch itself read (see dispatch_mechanism) -- a disclosed substitute, not the runner's own live scoring path. A future session with a working ANTHROPIC_API_KEY/apiKeyHelper should re-run both fixtures through the confirmed runner directly and reconcile any score drift against this record.", + "trials_per_fixture=1 for both fixtures, at both the original and relocated placements -- no repeat-run variance data; a single sample generalizing correctly is not proof a weaker tier or a different sample would.", + "Pre-existing selection fixtures (the other 94 in this corpus, including issue #1662's own sequential-pipeline-body-cap-exception-selection.yaml and issue #1346's own dispatch-self-guard-boundary-selection.yaml) were not re-scored on both sides. This edit's diff (a relocation plus wording fixes confined to dimension 5 and a one-sentence forward-pointer at the Portability level section, no other reword) was reasoned content-disjoint from the pre-existing corpus by direct inspection rather than confirmed by live re-dispatch across the whole corpus -- the same disclosed methodology this skill's own split.md log has used for a content-disjoint edit in multiple prior iterations. The sequential-pipeline-body-cap-exception-selection.yaml fixture WAS re-dispatched, against the final relocated rubric text, as this run's own Transfer check -- see below, not a full-corpus re-score.", + "Model identity for the Agent-tool dispatches is recorded as claude-sonnet-5 based on this calling session's own configured/default model; the Agent tool itself does not echo back a per-dispatch model identifier the way a live `claude --model` invocation's own output would, so this is the calling session's stated configuration, not a value independently re-derived from each dispatch's own transcript.", + "The Transfer check below substitutes an adjacent-fixture regression probe, run on the same model/harness (Agent-tool dispatch) as the primary gate, for SKILL.md's own literal 'adjacent model, harness, or nearby task' Transfer check text -- a genuinely different model or harness was not available in this session (see the runner-unavailability gap above); disclosed as narrower than the step's own full text describes.", + "Step 8's recommended prose/disclosure adversarial pass (scorer-gated-skill-edits SKILL.md Procedure step 8) is deferred to drafting-a-pr-to-merge's own mandatory Step 8 independent review, which runs before this branch's PR is left ready for human merge -- not run as a separate pass inside this gate iteration itself; disclosed here per that step's own known_gaps rule rather than silently treated as already covered.", + "drafting-a-pr-to-merge's own Step 8 inner-layer review (an isolated evaluating-skill-quality specialist dispatch against the full rubric.md diff, run after this branch was marked ready for review) found three genuine wording defects, all fixed: (1) the new substitute's own positive requirements b and c used two different terms ('portable fallback', 'portable substitute') for the same concept, and 'substitute' collided with the exemption's own proper name -- both unified to 'portable alternative'; (2) the substitute's own Fail-bullet clause illustrated only condition 2 and the third positive requirement failing, not condition 1 or the first/second positive requirements, unlike the sibling sequential-pipeline exemption's own Fail clause naming both of its conditions -- extended to name all five failure modes; (3) 'repository/platform-specific' introduced an undefined term colliding with this same combined skill file's own separate, established use of 'platform' for the Agentic-operation-mechanism-fit/isolation axis -- reverted to 'repository-specific' alone. None of these three changed the fixtures' own scored assertions (`output_contains`/`output_not_contains` text, unrelated to any of the three fixed phrases) or the underlying gating logic, so the already-recorded gate numbers in this file were not re-run; `gitapex_check_skill_shape.py` re-run 70/70 and the full local pytest suite (7883 tests) re-run clean after the fix, before this disclosure was written -- the same no-re-gate-needed disclosure pattern issue #1662's own precedent record (PR #1672) used for its own Step 8 wording-only fixes." + ], + "transfer_check": { + "adjacent_fixture": "evaluating-skill-quality-sequential-pipeline-body-cap-exception-selection (issue #1662's own cohesion-confirmed sequential-pipeline exemption fixture)", + "ran_against": "final, relocated after-edit rubric.md (this run's own candidate, commit eced6acde2e8f251ab87e3a326cd2f320ba9dcec)", + "score": 1.0, + "result": "No regression: re-dispatched the final relocated after-edit rubric.md against issue #1662's own pre-existing fixture text (the warehouse-inventory-reconciliation scenario, Portable-declared, no Mixed-portability content at all). The after-edit dispatch correctly re-derived both of the sequential-pipeline exemption's own conditions from the given facts (never re-deriving the cohesion finding itself, per instruction) and reached the identical Pass verdict issue #1662's own precedent fixture already established, explicitly confirming that grading this Portable target required no reference to the new Mixed-portability substitute bullet at all -- that bullet is correctly inert for a target the Portability-level Mixed sub-bullet does not apply to, and its relocation into dimension 5's own section changes nothing about that inertness.", + "artifact": "artifacts/transfer-check-sequential-pipeline-body-cap-exception-selection-after.md" + }, + "headline_pattern": "Issue #1676 closes the Mixed-portability closure gap issue #1632's own Round 12 review found: dimension 5's cohesion-confirmed sequential-pipeline exemption (issue #1662) gave a qualifying target no satisfiable way to close a separate Mixed-portability finding for every-use non-portable content, without either exceeding the exemption's own minimized file-count floor or corrupting a non-every-use reference file's own semantic contract. A new, narrow substitute closes this, gated on two independently-verifiable conditions (reused dimension-5 exemption; independently-inspected every-use non-portable content, never accepted from the target's own self-characterization) and satisfied via three positive requirements in place of ordinary file-level relocation. This branch's own mandatory Step 8 independent adversarial review found the substitute's first placement (directly under the Portability level section's own Mixed bullet) created a backward dependency on a dimension-5 finding that had not run yet at that grading step -- a genuine structural defect, confirmed against SKILL.md's own literal Procedure step-order annotations before acting. Relocating the full substitute into dimension 5's own section resolved it, along with two further condition/requirement wording fixes the same review found. The primary selection fixture, re-scored against the relocated rubric, moved 0.666667 -> 1.000000, KEEP; a live anti-loophole fixture confirmed the substitute's own condition-2 anti-self-assertion guard cannot be defeated by a target whose Notes section claims 'every-use' while its own Procedure text shows the content is actually conditional -- correctly failing before and after the edit, and before and after the relocation. A Transfer check against the adjacent, pre-existing sequential-pipeline-body-cap-exception fixture, re-run against the final relocated rubric, confirmed no regression to issue #1662's own exemption and that the new bullet is correctly inert for a Portable-declared target it does not apply to. The confirmed eval runner could not actually execute in this session's environment (issue #1304's own already-tracked, repo-wide infrastructure defect, reconfirmed this session); an isolated Agent-tool dispatch substitute was used instead and is disclosed as such throughout this record. Separately, this same edit's rubric change was applied to a re-grade of executing-a-branch-plan's own Mixed-portability status (verification only, no code change to that skill): both gating conditions are met, but its three positive requirements are not yet satisfied -- recorded in that skill's own metadata/gitapex.yaml decision log, out of this issue's own scope to fix, and not independently re-confirmed against this record's own final relocated wording (disclosed there as an open item)." +} diff --git a/evals/evaluating-skill-quality/split.json b/evals/evaluating-skill-quality/split.json index 412497b0..32ad5003 100644 --- a/evals/evaluating-skill-quality/split.json +++ b/evals/evaluating-skill-quality/split.json @@ -80,7 +80,9 @@ "dispatch-self-guard-boundary-selection.yaml", "structural-identifier-portability-selection.yaml", "durability-stale-reference-annotation-selection.yaml", - "sequential-pipeline-body-cap-exception-selection.yaml" + "sequential-pipeline-body-cap-exception-selection.yaml", + "mixed-portability-dimension5-substitute-selection.yaml", + "mixed-portability-dimension5-substitute-false-positive-selection.yaml" ], "test": [ "guardrail.yaml", @@ -103,7 +105,7 @@ "context-economy-test.yaml" ] }, - "partition": "35:42:18", + "partition": "35:44:18", "split_arithmetic_exclusions": [ "dispatch-required-negative-control.yaml" ] diff --git a/evals/evaluating-skill-quality/split.md b/evals/evaluating-skill-quality/split.md index 6aebe8f3..aeef341b 100644 --- a/evals/evaluating-skill-quality/split.md +++ b/evals/evaluating-skill-quality/split.md @@ -4713,3 +4713,169 @@ own `transfer_check`. ### Verdict KEEP. Refs #1662. + +## Iteration: issue #1676, Mixed-portability substitute for a Dimension-5-exempted every-use target + +Motivated by issue #1676, itself found during PR #1632's own continued +independent review of `executing-a-branch-plan`: issue #1662's just-merged +dimension-5 sequential-pipeline exemption grades a qualifying target on +minimizing an irreducible common-case file-count floor, but never mentions +the separate Mixed-portability split rule at all -- even though that rule +frames itself as its own dimension-5 requirement. A Dimension-5-exempted +target whose non-portable content is itself every-use (`executing-a-branch-plan` +is exactly this shape) has no satisfiable, textually-compliant way to close +a Mixed-portability finding: physically relocating the non-portable content +into a new every-use reference file pushes the exemption's own +already-minimized file count past its floor; folding it into a non-every-use +file instead destroys that file's own "never read on an ordinary clean run" +contract and reopens dimension 5 outright. + +Candidate edit, ordinary class (a new nested rubric.md bullet under the +existing Mixed bullet, no deletion of existing text): `references/rubric.md`'s +Portability level → Mixed bullet gains a nested substitute, gated on two +conditions neither self-assertable by the reviewed target -- (1) this same +review's own dimension-5 walk has already granted the cohesion-confirmed +sequential-pipeline exemption, reused rather than re-derived; (2) the +non-portable content is demonstrably read on every ordinary origin-environment +run, established by inspecting which procedure steps actually read it, never +accepted from the target's own "every-use"/"interleaved" self-characterization +-- satisfied via three positive requirements (distinct-heading isolation; +a `SKILL.md` Notes declaration naming non-portable steps and their portable +fallback; one dedicated, non-every-use reference file enumerating every +touchpoint and its substitute) instead of ordinary file-level physical +relocation. Plus a companion one-clause cross-reference inside the +Dimension-5 exemption's own "still apply in full" parenthetical. The Mixed +bullet's own pre-existing five lines, and the Dimension-5 exemption's own +text beyond that one clause, are unchanged. Refs #1676. + +### Placement correction, found live during executing-a-branch-plan's own Step 8 + +**Found during this branch's own mandatory Step 8 independent adversarial +review (executing-a-branch-plan's own gate, distinct from this skill's own +recommended Step 8 pass), after the fixtures below had already been gated +once against an earlier placement, and disclosed per this file's own +discrimination-not-just-match discipline:** the review found a real +structural defect in the substitute's first placement -- it lived under +the Portability level section (graded at SKILL.md Procedure step 4), but +its own condition 1 required a dimension-5 finding (step 5) that had not +run yet at step 4, a genuine backward reference confirmed against +SKILL.md's own literal Procedure step-order annotations before acting on +it, not assumed. Relocated the full substitute into dimension 5's own +section, immediately after the sequential-pipeline exemption it depends +on, resolving the backward reference; the Portability level section keeps +only a short forward-pointer sentence. The same review also found +condition 2's "read" wording did not close the loophole it was meant to (a +conditionally-executed step could still claim to satisfy a "read" test), +fixed to require the content be demonstrably reached and acted on; and +found the third positive requirement (the dedicated reference file) was +satisfiable by the target's own bare claim with no supplied content to +verify against, fixed by adding the same anti-self-assertion discipline +condition 2 already carries, and by supplying +`mixed-portability-dimension5-substitute-selection.yaml`'s own +dedicated-file content directly in its fixture prompt so this can actually +be checked. Both fixtures were then re-dispatched against the relocated +rubric text and re-scored; the gate result below is the final, +post-relocation result, not the original placement's. + +### Gate result + +Went through `scorer-gated-skill-edits`'s own held-out gate: 2 new +selection fixtures added to `split.json`'s split (44 total selection, +35:44:18). `mixed-portability-dimension5-substitute-selection.yaml`'s own +target skill (a Mixed-declared, Dimension-5-exempted "vault-secret-rotation" +scenario whose sole non-portable step is genuinely unconditional and +every-use, and which supplies all three of the new substitute's positive +requirements) is the primary gated fixture: re-scored against the final, +relocated rubric text, one isolated dispatch pair moved +**0.666667 -> 1.000000, KEEP**. The before-edit dispatch (dispatched +against the pre-issue-1676 commit, no substitute of any kind) correctly +reasoned to a dimension-5 Fail by quoting the ordinary Mixed rule verbatim +and confirming no substitute for a Dimension-5-exempted target exists in +that rubric text; the after-edit dispatch correctly independently +re-verified both of the relocated substitute's gating conditions from the +excerpt's own procedure structure -- never accepting the target's own +self-characterization -- and correctly graded all three positive +requirements Pass, explicitly naming "Condition 2 holds" for the +unconditional-execution finding. +`mixed-portability-dimension5-substitute-false-positive-selection.yaml`'s +own target (a "log-shipping-connector" scenario whose Notes section claims +its non-portable step is "every-use"/"interleaved" while its own Procedure +text shows the step is actually skipped for any non-matching destination) +is a live anti-loophole discrimination fixture, not itself gated for +movement: both before and after the edit, and both before and after the +relocation, the dispatch correctly quoted the target's own conditional +trigger text verbatim and correctly failed it -- confirming the new +substitute's own condition-2 anti-self-assertion guard cannot be defeated +by a self-serving Notes claim regardless of which rubric section it is +graded under. `gitapex_check_skill_shape.py`: 70/70. Full record, all four +transcripts plus the transfer check: `results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/manifest.json`. + +**Further corrections, found live while re-scoring the relocated rubric +against real dispatch transcripts and disclosed per this file's own +discrimination-not-just-match discipline:** the primary fixture's +`output_contains` assertion went through two more rounds after the +relocation. `"Mixed-portability substitute"` (itself already a +replacement for an even earlier near-verbatim quote that failed to survive +paraphrasing) still under-discriminated once condition 2's own wording +changed to "reached and acted on": generic negative guards considered +alongside it (`"does not apply"`, `"is conditional"`) each collided with an +innocuous, unrelated phrase inside the correct PASS transcript (an +unrelated "Indeterminate: does not apply" verdict-category line; a +"whether step 4's own execution is conditional" clause). Abandoned the +generic-phrase-avoidance strategy and instead empirically tested candidate +positive-only phrases directly against all four real saved transcripts +until finding `"Condition 2 holds"`, confirmed via `grep -c` to occur +exactly 0/1/0/0 times across before/after/false-positive-before/ +false-positive-after respectively -- perfect discrimination -- before this +record was finalized. Separately, and earlier, during the original +(pre-relocation) gate run: `mixed-portability-dimension5-substitute-selection.yaml`'s +own first draft had its non-portable step invoke a rollback command +conditionally ("for any dependent service that does not acknowledge... +within timeout"), which an after-edit dispatch correctly identified as +failing the substitute's own condition 2 -- a genuine scenario-design +flaw, not a rubric defect. Fixed by folding the rollback into a single, +unconditional invocation, removing the conditional branch entirely; that +discarded draft's own transcripts are not included in the run record's +artifacts/. + +**Eval-runner unavailability, disclosed rather than silently substituted:** +the confirmed runner (`evals/scripts/gitapex_run_eval_suite.py`, commit +`a045f8b9`, unchanged since issue #1662's own run) could not actually +execute in this session's environment -- a live `claude --bare --tools "" +-p` invocation failed immediately with an `Authentication error`, matching +issue #1304's already-tracked, repo-wide, root-cause-unconfirmed +infrastructure defect, independently reconfirmed this session (the same +signature issue #1662's own run record already disclosed, now reconfirmed +a second time rather than assumed still true). Scoring was performed +instead via isolated Agent-tool subagent dispatch reasoning over a +combined skill-content file the dispatch itself read via its own `Read` +tool call (never pasted inline by the calling session) -- a disclosed +substitute measurement, not the confirmed runner's own live scoring path. +Full accounting in the run record's own `known_gaps`. + +### Transfer check + +Run this iteration: the final, relocated after-edit rubric.md was +re-dispatched against the adjacent, pre-existing +`sequential-pipeline-body-cap-exception-selection.yaml` fixture (issue +#1662's own cohesion-confirmed sequential-pipeline exemption fixture, +Portable-declared). Score: **1.000000** -- no regression. The after-edit +dispatch correctly re-applied both of the pre-existing exemption's own +conditions to that fixture's warehouse-inventory-reconciliation scenario, +reached the identical Pass verdict issue #1662's own precedent already +established, and explicitly confirmed that grading a Portable-declared +target required no reference to the new Mixed-portability substitute +bullet at all -- that bullet is correctly inert for a target the +Portability-level Mixed sub-bullet does not apply to, and its relocation +into dimension 5's own section changes nothing about that inertness. This +is a same-model/same-harness adjacent-*fixture* probe, not a cross-model +or cross-harness re-run (no second model/harness was available in this +session -- see the eval-runner-unavailability gap above); disclosed as +narrower than SKILL.md's own "adjacent model, harness, or nearby task" +Transfer-check text describes. Full detail: +`results/2026-09-02-issue-1676-mixed-portability-dimension5-substitute/manifest.json`'s +own `transfer_check`. + +### Verdict + +KEEP. Refs #1676. diff --git a/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-false-positive-selection.yaml b/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-false-positive-selection.yaml new file mode 100644 index 00000000..fa99fa4b --- /dev/null +++ b/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-false-positive-selection.yaml @@ -0,0 +1,67 @@ +id: evaluating-skill-quality-mixed-portability-dimension5-substitute-false-positive-selection +name: Mixed Substitute Claim Must Be Independently Verified, Not Accepted On The Target's Own Self-Characterization +description: An adversarial anti-loophole probe for the new Mixed-portability substitute. The target is Mixed-declared, claims (given as fact) the cohesion-confirmed sequential-pipeline exemption, and its own Notes section explicitly asserts its platform-specific content is "every-use" -- and it even ships a porting-boundary-map.md-shaped reference file and distinct headings, structurally mimicking a genuinely qualifying target. But the Procedure text itself shows the platform-specific step only runs "only when the destination is ProviderX" -- a conditional branch, not every ordinary run -- directly contradicting the Notes' own "every-use" claim. The new substitute's own condition 2 requires this be established by inspecting which procedure steps actually read the content, never accepted from the target's own self-characterization; a review that pattern-matches on the Notes declaration and the porting-boundary-map file's mere presence, rather than actually reading the Procedure, would incorrectly pass this target. The review must fail it -- both before and after the rubric edit, since it fails condition 2 either way -- and must show its work by naming the conditional trigger it found, not merely assert a conclusion. +tags: + - quality + - portability + - progressive-disclosure + - adversarial +inputs: + prompt: | + Use evaluating-skill-quality. Review this draft SKILL.md excerpt for a + skill called "log-shipping-connector". + + Given, not to be re-derived: this target's cohesion check already ran + at its own Procedure step 2 and returned single-outcome sequential + cohesion confirmed, and this target's own dimension-5 walk (already + run, this same review) confirmed it clears the cohesion-confirmed + sequential-pipeline exemption -- its mandatory reference files total + over the 500-line `BODY_MAX_LINES` cap with no rearrangement able to + lower that floor. Do not re-run or second-guess either finding -- take + both as already established. + + --- + name: log-shipping-connector + description: Ship application logs from the local buffer to whichever destination this deployment is configured for. Use when the local log buffer needs draining to a durable destination. + --- + + # Log Shipping Connector + + ## Procedure (portable) + + 1. Read the local buffer's own configuration to determine the + configured destination and batch size. + 2. Read `references/batching-rules.md` (mandatory every run) and + assemble the next batch of buffered log lines per its own + size/time-window rules. + 3. Read `references/retry-policy.md` (mandatory every run) and + compute this batch's own retry/backoff schedule before shipping. + + ## Procedure (ProviderX-specific, non-portable) + + 4. Only when the destination is ProviderX (this deployment's own + default, but not its only configured destination), call + ProviderX's proprietary `providerx-ship` binary to deliver the + batch, using ProviderX's own authentication token format. For any + other configured destination, this step is skipped entirely and + the batch is delivered through the portable Procedure's own + generic HTTP-POST fallback instead (not shown here). + 5. Read `references/delivery-confirmation.md` (mandatory every run) + and record the delivery outcome from whichever of step 4's two + paths actually ran. + + ## Notes + + Portability: **Mixed**. Step 4's `providerx-ship` invocation is + ProviderX-specific and is read on every ordinary run of this skill, + interleaved with the portable steps around it -- there is no + caller-selectable narrower path that skips it. See + `references/porting-boundary-map.md` (read only when vendoring this + skill elsewhere) for its portable HTTP-POST substitute. +expected: + output_contains: + - "Only when the destination is" + output_not_contains: + - "LGTM" + - "no concerns" + - "satisfies the substitute" diff --git a/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-selection.yaml b/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-selection.yaml new file mode 100644 index 00000000..25b7a7b4 --- /dev/null +++ b/evals/evaluating-skill-quality/tasks/mixed-portability-dimension5-substitute-selection.yaml @@ -0,0 +1,109 @@ +id: evaluating-skill-quality-mixed-portability-dimension5-substitute-selection +name: Dimension-5-Exempted Mixed Skill Has A Satisfiable Portability Closure Via The Substitute +description: The target skill is Mixed-declared and its cohesion check has already confirmed (Procedure step 2, given as fact) single-outcome sequential cohesion with every-use reference content over BODY_MAX_LINES -- it already clears dimension 5's own cohesion-confirmed sequential-pipeline exemption. Its non-portable, platform-specific content (a proprietary secrets-vault API) is itself executed unconditionally on every ordinary run, not detail a caller only sometimes needs. The excerpt supplies all three of the new Mixed-portability substitute's positive requirements -- portable/non-portable content under distinct headings, a SKILL.md Notes declaration naming the non-portable step and its portable fallback, and a dedicated non-every-use porting-boundary-map.md reference file whose full content is also supplied so its own existence and completeness can be independently verified, not accepted on the target's own claim. Tests the substitute as now graded entirely within dimension 5's own walk (moved there after issue #1676's own Step 8 adversarial review found the original Portability-level placement created a backward dependency on a not-yet-run dimension-5 finding) -- before the rubric edit, the review must fail the target under the ordinary Mixed rule's literal file-level-split requirement; after the edit, the review must recognize the substitute is satisfied and pass it, while still independently re-deriving both gating conditions and all three positive requirements rather than accepting the target's own self-characterization. +tags: + - quality + - portability + - progressive-disclosure +inputs: + prompt: | + Use evaluating-skill-quality. Review this draft SKILL.md excerpt for a + skill called "vault-secret-rotation". + + Given, not to be re-derived: this target's cohesion check already ran + at its own Procedure step 2 and returned single-outcome sequential + cohesion confirmed -- the five steps below all converge on one outcome + ("every credential due for rotation is rotated in the vault and every + dependent service is updated to the new value, with no service left + holding a stale credential"), each step's output is the next step's + required input, and there is no caller-selectable narrower path + through this skill. This target's own dimension-5 walk (already run, + this same review) confirmed it clears the cohesion-confirmed + sequential-pipeline exemption: its three mandatory reference files + (`references/rotation-policy.md`, 260 lines, + `references/dependent-service-registry.md`, 230 lines, and + `references/rollback-procedure.md`, 200 lines -- 690 lines combined, + over the 500-line `BODY_MAX_LINES` cap) are each genuinely load-bearing + domain policy with no padding a dimension-2 cut could remove, and no + fourth-file or single-file rearrangement lowers that floor. Do not + re-run or second-guess either the cohesion finding or the dimension-5 + exemption -- take both as already established. + + --- + name: vault-secret-rotation + description: Rotate every credential due for rotation in the organization's secrets vault and propagate each new value to its dependent services. Use when a scheduled or emergency credential rotation is due. + --- + + # Vault Secret Rotation + + ## Procedure (portable) + + 1. Enumerate every credential due for rotation from the vault's own + rotation schedule. + 2. Read `references/rotation-policy.md` (mandatory every run) and + compute each credential's own new value per its rotation policy. + 3. Read `references/dependent-service-registry.md` (mandatory every + run) to identify every service that depends on each rotated + credential. + + ## Procedure (VaultCo-specific, non-portable) + + 4. Call VaultCo's proprietary `vaultco-cli rotate --id + --auto-rollback` command (a VaultCo-only binary, present only in + this organization's own deployment) to write each new credential + value into the vault and issue VaultCo's own propagation webhook + to every dependent service identified in step 3. This single + command always runs to completion on every rotation: the + binary's own `--auto-rollback` flag atomically reverts to the + prior value, internally, if any dependent service fails to + acknowledge within VaultCo's own webhook timeout, so no separate + conditional rollback step exists in this Procedure -- the command + itself, and its outcome (rotated or auto-rolled-back), is the + only VaultCo-specific content, and it is invoked exactly once, + unconditionally, every run. + + ## Procedure (portable, continued) + + 5. Read `references/rollback-procedure.md` (mandatory every run) and + record this rotation's own outcome -- whether step 4 completed as + a rotation or an auto-rollback -- per its own generic audit-trail + rules, closing the reconciliation. This step reads only the + outcome step 4 already produced and never itself calls + `vaultco-cli`. + + ## Notes + + Portability: **Mixed**. Steps 1-3, step 5, and all three mandatory + reference files are portable. Step 4's own `vaultco-cli` invocation + and webhook mechanism -- and only step 4's -- is VaultCo-specific, + is read and executed on every ordinary run with no + caller-selectable narrower path around it, and has no portable + substitute inline in this file: a copy of this skill vendored + outside a VaultCo deployment would need to replace step 4's own + single call with that deployment's own credential-write mechanism. + See `references/porting-boundary-map.md` (read only when vendoring + this skill elsewhere, never on an ordinary run) for that one + touchpoint and its portable substitute. + + Also given: `references/porting-boundary-map.md` genuinely exists + and reads in full as follows (do not accept this on the target's + own say-so alone -- its content is supplied here so you can verify + it actually enumerates the one non-portable touchpoint and its + substitute, rather than merely being claimed to): + + ``` + # Porting boundary map + + Read only when vendoring vault-secret-rotation outside a VaultCo + deployment -- never on an ordinary run. + + | Non-portable touchpoint | Portable substitute | + |---|---| + | Step 4's `vaultco-cli rotate --id --auto-rollback` | Replace with the target deployment's own credential-write-and-auto-rollback mechanism (e.g. a direct API call to that deployment's own secrets vault, mirroring the same atomic rotate-with-rollback contract). | + ``` +expected: + output_contains: + - "Condition 2 holds" + output_not_contains: + - "LGTM" + - "no concerns" diff --git a/skills/evaluating-skill-quality/metadata/gitapex.yaml b/skills/evaluating-skill-quality/metadata/gitapex.yaml index b6e62fca..1cd8f527 100644 --- a/skills/evaluating-skill-quality/metadata/gitapex.yaml +++ b/skills/evaluating-skill-quality/metadata/gitapex.yaml @@ -574,6 +574,13 @@ spec: verdict: fixed before: 0.666667 after: 1.0 + - kind: decision + anchor: "https://github.com/tvna/gitapex/issues/1676" + summary: "Dimension 5 gained a Mixed-portability substitute for a D5-exempted every-use target, gated on two conditions and satisfied via three positive requirements instead of file relocation. Relocated from an initial Portability-level placement into dimension 5 after Step 8 review found the original backward-referenced a not-yet-run dimension-5 finding. An anti-loophole fixture confirmed the guard resists a self-serving Notes claim, before and after relocation. A Transfer check confirmed no regression." + outcome: + verdict: fixed + before: 0.666667 + after: 1.0 skillDependencies: requires: [] relatedTo: diff --git a/skills/evaluating-skill-quality/references/rubric.md b/skills/evaluating-skill-quality/references/rubric.md index 2080925c..691dfce2 100644 --- a/skills/evaluating-skill-quality/references/rubric.md +++ b/skills/evaluating-skill-quality/references/rubric.md @@ -782,7 +782,13 @@ grading below. split, not just the intent to split: the repository-specific part belongs in a clearly named reference file (e.g. `references/this-repo-only.md`) a consumer can identify and drop, not - blended into the portable core. + blended into the portable core. This classification step only + establishes that the ordinary rule applies; a narrow substitute for a + target that also clears dimension 5's own cohesion-confirmed + sequential-pipeline exemption is graded entirely within dimension 5's + own walk below, not here -- see dimension 5's own Mixed-portability + substitute, since its first gating condition is a dimension-5 finding + this earlier classification step has no access to yet. ### Dependency file portability @@ -1708,6 +1714,58 @@ presence by shape. one condition, or neither, is graded exactly like any other split under the tests above. +- **A Mixed-declared target that clears the cohesion-confirmed + sequential-pipeline exemption above, and whose non-portable content is + itself every-use, has a narrow substitute for the Portability level + section's own ordinary Mixed file-level-split requirement -- but only + under two conditions, checked here, not there, because the first one + is this dimension's own finding.** The Portability level section's + Mixed bullet prices a *choice*: an author who could relocate the + repository-specific part into a dedicated reference file, and + simply has not done so yet. A target that has already cleared the + cohesion-confirmed sequential-pipeline exemption immediately above has + no such choice available for its every-use non-portable content + specifically: physically relocating that content into a new every-use + reference file would push the exemption's own already-minimized + common-case file count past its irreducible floor, and folding it into + a non-every-use file instead would contaminate that file's own "never + read on an ordinary clean run" semantic contract and reopen this + dimension outright. Treating such a target identically to an ordinary + Mixed skill that merely declined to split is not automatically + correct, but neither is exempting any Mixed-declared skill from the + file-level-split requirement on the bare claim that relocating its + non-portable content would be inconvenient: that would let any bundled + non-portable content dodge that rule merely by relabeling itself + "every-use." The substitute below applies only when both (1) this same + dimension-5 walk has already granted the target the cohesion-confirmed + sequential-pipeline exemption immediately above -- reused from that + finding, established earlier in this same walk, never re-derived here, + and never available to a target this walk has not itself cleared for + it -- and (2) the non-portable content is demonstrably reached and + acted on, not merely read as inert text, on every ordinary + origin-environment run, established by inspecting which procedure + steps actually execute it, never accepted from the target's own + "every-use"/"interleaved" self-characterization -- a step whose own + action is itself conditional (skipped on some runs, gated on a runtime + value) fails this condition even when its surrounding prose is read + every time. A target meeting both conditions satisfies the Portability + level section's own Mixed rule's stated purpose (a consumer can + identify and drop what it doesn't need) via three positive + requirements instead of the ordinary file-level physical relocation: + portable and non-portable content isolated under distinct headings, + never blended sentence-by-sentence; a `SKILL.md` Notes declaration + naming which steps execute non-portable content and each one's own + portable alternative; and one dedicated, non-every-use reference file -- + confirmed to actually exist and to genuinely enumerate every + non-portable touchpoint the target carries and its own portable + alternative, never accepted from the target's own claim that such a + file exists or is complete, the same anti-self-assertion discipline + condition 2 already applies -- read only at vendoring/porting time, + never on an ordinary run. A target meeting only one condition, or + neither, is graded exactly like any other Mixed-declared skill under + the Portability level section's own ordinary rule: the file-level + split it declines is not optional for it. + - **Fail:** a reference named `doc2.md` with no branch-point pointer in `SKILL.md`, or content the model reads on every single use pushed out to a reference that must be opened just to complete the ordinary path; a @@ -1719,7 +1777,19 @@ presence by shape. cohesion-check finding backing that exemption's own condition 1, or whose every-use content could in fact fit under `BODY_MAX_LINES` after actually applying dimension 2's own tests to it, failing that - exemption's own condition 2. + exemption's own condition 2; a Mixed-declared target claiming the + Mixed-portability substitute above without this same dimension-5 walk + having actually granted it the cohesion-confirmed sequential-pipeline + exemption (failing that substitute's own condition 1), whose + non-portable step is in fact conditional rather than executed every run + (failing that substitute's own condition 2), whose portable and + non-portable content is not actually isolated under distinct headings or + whose `SKILL.md` Notes declaration does not actually name the + non-portable steps and their portable alternative (failing that + substitute's own first or second positive requirement), or whose claimed + dedicated reference file does not actually exist or does not actually + enumerate every non-portable touchpoint (failing that substitute's own + third positive requirement). - **Pass:** `SKILL.md` links to each reference exactly where it becomes necessary, stating what context requires the read and what the reader will obtain; the common case resolves from `SKILL.md` alone; where a @@ -1730,7 +1800,11 @@ presence by shape. over-cap sequential pipeline qualifies for the narrow exemption above, the verdict names the irreducible floor and grades the split's own file-count minimization against it rather than failing outright for a - floor no rearrangement could lower. + floor no rearrangement could lower; where a Mixed-declared target also + qualifies for the Mixed-portability substitute above, the verdict + independently confirms both its conditions and all three of its + positive requirements against the target's own actual content before + crediting it, rather than accepting the target's own claim. ## 6. Durability diff --git a/skills/executing-a-branch-plan/metadata/gitapex.yaml b/skills/executing-a-branch-plan/metadata/gitapex.yaml index facc7dac..74754a17 100644 --- a/skills/executing-a-branch-plan/metadata/gitapex.yaml +++ b/skills/executing-a-branch-plan/metadata/gitapex.yaml @@ -42,3 +42,13 @@ spec: - kind: decision anchor: "https://github.com/tvna/gitapex/issues/1558" summary: "Step 6's per-task screening dispatches screening-a-low-trust-contribution's checks 2-8 to the new agents/review-persona.md subagent type (tools: Read, Grep, Glob) instead of reasoning about the task's own diff directly in the main thread, which at that point holds mcp__github write access, git push, and unrestricted Bash. Isolates tool privilege only -- the separate calling-context-contamination axis is disclosed in review-persona.md's own Limits section, not resolved here." + - kind: deferral + anchor: "https://github.com/tvna/gitapex/issues/1676" + summary: "Mixed-portability re-graded vs. rubric.md's new Mixed-portability substitute (verification-only). Both gating conditions met (D5 exemption granted; Claude-Code mechanics confirmed every-use). All three positive requirements unmet: no heading isolation, an incomplete Notes declaration, no porting-boundary-map file. Stays an open finding; closing it is separate follow-up, out of this issue's scope. Not re-verified against the rubric's later dimension-5 relocation; substance believed unchanged." + outcome: + verdict: open + condition_1_dimension5_exemption: met + condition_2_every_use_independently_verified: met + requirement_1_distinct_headings: not_met + requirement_2_notes_declaration: not_met + requirement_3_porting_boundary_map_file: not_met diff --git a/tests/test_gitapex_gate_split_fixture_coverage.py b/tests/test_gitapex_gate_split_fixture_coverage.py index f6b39af2..076f19ef 100644 --- a/tests/test_gitapex_gate_split_fixture_coverage.py +++ b/tests/test_gitapex_gate_split_fixture_coverage.py @@ -1772,7 +1772,7 @@ def test_real_split_json_partition_declarations_are_pinned_exactly(): declared[path.parent.name] = gate.parse_declared_partition(data) assert declared == { "battle-testing-a-skill": None, - "evaluating-skill-quality": (35, 42, 18), + "evaluating-skill-quality": (35, 44, 18), "explaining-the-work": (3, 2, 9), "merge-retrospective": (11, 7, 5), "scorer-gated-skill-edits": None, diff --git a/tests/test_gitapex_scan_eval_results_schema.py b/tests/test_gitapex_scan_eval_results_schema.py index b8095662..ebce35a4 100644 --- a/tests/test_gitapex_scan_eval_results_schema.py +++ b/tests/test_gitapex_scan_eval_results_schema.py @@ -934,6 +934,7 @@ def test_real_repository_nonstandard_score_files_are_all_declared() -> None: "evaluating-skill-quality/2026-08-26-issue-1347-structural-identifier-portability", "evaluating-skill-quality/2026-08-30-issue-1466-durability-stale-reference", "evaluating-skill-quality/2026-09-02-issue-1662-sequential-pipeline-body-cap-exception", + "evaluating-skill-quality/2026-09-02-issue-1676-mixed-portability-dimension5-substitute", } @@ -1076,6 +1077,7 @@ def test_real_repository_discovers_every_committed_run_directory() -> None: "evaluating-skill-quality/2026-08-26-issue-1347-structural-identifier-portability", "evaluating-skill-quality/2026-08-30-issue-1466-durability-stale-reference", "evaluating-skill-quality/2026-09-02-issue-1662-sequential-pipeline-body-cap-exception", + "evaluating-skill-quality/2026-09-02-issue-1676-mixed-portability-dimension5-substitute", "untrusted-input-triage/2026-08-01-issue-645-battle-test", "untrusted-input-triage/2026-08-01-issue-645-behavioral-eval", "untrusted-input-triage/2026-08-01-issue-646-behavioral-gate2",