Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/skill-eval-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ suites still assert on final text only).
| `establishing-ubiquitous-language` | 3 | 6 | none | no | [evals/establishing-ubiquitous-language/eval-status.md](../evals/establishing-ubiquitous-language/eval-status.md) |
| `evaluating-context-channel-maturity` | 3 | 13 | none | no | [evals/evaluating-context-channel-maturity/eval-status.md](../evals/evaluating-context-channel-maturity/eval-status.md) |
| `evaluating-deterministic-gate-quality` | 3 | 40 | none | no | [evals/evaluating-deterministic-gate-quality/eval-status.md](../evals/evaluating-deterministic-gate-quality/eval-status.md) |
| `evaluating-skill-quality` | 3 | 95 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) |
| `evaluating-skill-quality` | 3 | 96 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) |
| `executing-a-branch-plan` | 3 | 10 | none | no | [evals/executing-a-branch-plan/eval-status.md](../evals/executing-a-branch-plan/eval-status.md) |
| `explaining-the-work` | 3 | 14 | none | no | [evals/explaining-the-work/eval-status.md](../evals/explaining-the-work/eval-status.md) |
| `grounding-in-primary-sources` | 3 | 23 | none | no | [evals/grounding-in-primary-sources/eval-status.md](../evals/grounding-in-primary-sources/eval-status.md) |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
# Add a Dimension-5 exception for a cohesion-confirmed, over-cap sequential-pipeline skill

**Goal:** `evaluating-skill-quality/references/rubric.md`'s Dimension 5
(Progressive disclosure) has no passing configuration for a skill that is
(a) already confirmed sequential/functional cohesion at the cohesion
check's own Procedure step 2, and (b) whose every-use content genuinely
exceeds `BODY_MAX_LINES` no matter how it is split across reference
files. `skills/executing-a-branch-plan` is exactly this shape. Add one
narrow, loophole-resistant exception bullet to Dimension 5, gated on two
independently-verifiable conditions, without touching issue #1346's
existing self-guard exemption. Source: https://github.com/tvna/gitapex/issues/1662.

**Authorization record:** No approving comment exists on issue #1662
(checked via `github:issue_read` method `get_comments`, empty result --
opened minutes before this session started, by the repository owner).
Branch 2 of the Authorization gate applies: the active human operator's
own opening turn in this session explicitly instructed executing issue
#1662 through to just-before-merge ("こちらのPRを作りマージ直前まで進める").
This is a fresh, explicit, in-session confirmation for this specific
issue's execution.

**Structural precondition (issue #1306's own gate):** `planning-a-branch-from-an-issue`'s
Step 5 re-verification marker was written to issue #1662's own body at
2026-09-02T05:07:03Z and confirmed present via
`gitapex_check_branch_plan_reverified.py` (PASS) before this file was
authored.

**Threat-model triage (step 2):** Issue #1662 was read in full (11037
decoded characters) and grepped for injection-pattern indicators
(`ignore previous instructions`, `system-reminder`, `base64`,
`exfiltrate`, `credential`, `secret`, `urgent`, embedded script/tool-use
syntax) -- no matches. It is a well-formed, professionally-scoped ACM
issue authored by the repository owner (`author_association: OWNER`),
citing concrete line numbers and prior issues/PRs (#1346, #1632, #1648)
throughout. Every ACM row's Planned ops column describes a change to a
named file, not an instruction directed at the executing agent. Clean.

**Architecture:** One task, one wave (degenerate case per
`executing-a-branch-plan`'s own Related skills section -- every step
1/2/4-9 runs unchanged). File-ownership is entirely disjoint from any
other in-flight work:

- `skills/evaluating-skill-quality/references/rubric.md` -- insert one new
Dimension-5 exception bullet immediately after the existing self-guard
exemption bullet (after line 1663), plus matching Fail/Pass clauses.
No change to the existing self-guard bullet's own text.
- `evals/evaluating-skill-quality/tasks/*.yaml` (new) -- one selection
fixture exercising the new exception (a cohesion-confirmed sequential
pipeline whose every-use content exceeds `BODY_MAX_LINES`).
- `evals/evaluating-skill-quality/split.json`, `split.md` -- register the
new fixture, update the corpus-size arithmetic.
- `evals/evaluating-skill-quality/eval-status.md` -- prose summary of the
gate run.
- `evals/evaluating-skill-quality/results/<date>-issue-1662-*/manifest.json`
(new) -- the gate run record, schema-validated.
- `skills/evaluating-skill-quality/metadata/gitapex.yaml` -- decision log
entry.

**Interface dependencies:** none (single task). The rubric.md edit is a
prerequisite the eval-gate step reads before/after, entirely inside this
one task.

**Wave assignment:** wave 1 -- the single task above.

**Proof method:** `scorer-gated-skill-edits`'s own held-out gate
(selection-split mean strictly increases before -> after) plus
`gitapex_check_skill_shape.py` full run plus
`.github/scripts/gitapex_gate_split_fixture_coverage.py`.
10 changes: 10 additions & 0 deletions evals/evaluating-skill-quality/eval-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -1288,3 +1288,13 @@ The "Dimension 5/6 boundary clarifications" entry above's own final isolated sel
Issue #260 (merge retrospective for PR #258), Repair 1, proposed a durable rubric example for a real, repeated pattern: PR #258's first commit fixed a stale/dangling reference (a retired sibling skill name) by *annotating* it in place with a commit hash instead of removing or generalizing it, duplicating what `git log`/`git blame` already track permanently, before a second commit reworded it to drop the name outright instead. `references/rubric.md`'s dimension 6 (Durability) gained a bullet naming this exact pattern -- prefer removing or generalizing a stale in-repo reference over annotating it with commit provenance -- plus one clause each in the section's existing combined Fail:/Pass: block. `SKILL.md` needed no companion edit, matching the #185 entry's own precedent above.

Went through `scorer-gated-skill-edits`' own held-out gate: 1 new selection fixture added to `split.json`'s split (41 total selection, 35:41:18). One fresh isolated `claude -p` dispatch pair against the new selection fixture moved **0.833333 -> 1.000000, KEEP** -- the before-edit dispatch already reasoned to the correct remedy through general reasoning alone, the after-edit dispatch additionally grounding it in the new rubric text's own Fail/Pass wording, the same axis-did-not-exist-yet shape several prior iterations in this file already established. Full record, deterministic-check results, and both incidental pre-push-gate fixes found along the way: `evals/evaluating-skill-quality/split.md`'s Kept-edit log. Refs #1466, #260.

## Dimension-5 sequential-pipeline-body-cap exception (issue #1662)

Issue #1346's own residual-risk section had anticipated a follow-up: "If self-review after this fix still shows dimension 5 as Fail, a heavier restructuring... needs an explicit operator trade-off decision." Issue #1662 is that decision, for a structurally distinct case from #1346's own self-guard exemption -- a genuinely single-outcome sequential-pipeline skill (`skills/executing-a-branch-plan` is the motivating example: nine strictly sequential steps, ~1,400 mandatory reference lines over the 500-line `BODY_MAX_LINES` cap) that no file-layout arrangement could pass under the pre-edit dimension-5 text.

`references/rubric.md`'s dimension 5 gained a second, independently-gated exception bullet immediately after the existing #1346 self-guard bullet (unmodified), applying only when (1) the cohesion check has already confirmed sequential/functional cohesion at its own Procedure step 2 -- reused, never re-derived from the target's own prose -- and (2) the target's every-use reference content demonstrably exceeds `BODY_MAX_LINES` even after dimension-2 padding cuts. A qualifying skill is graded on whether its split minimizes the common-case file count against that irreducible floor, rather than failing outright.

Went through `scorer-gated-skill-edits`'s own held-out gate: 1 new selection fixture added to `split.json`'s split (42 total selection, 35:42:18). One isolated dispatch pair against the new selection fixture moved **0.666667 -> 1.000000, KEEP** -- the before-edit dispatch correctly quoted the pre-edit rubric's own Fail sentence verbatim and reasoned to Fail; the after-edit dispatch correctly reasoned to Pass under the new exception, independently re-deriving the cohesion precondition rather than accepting the review target's own embedded "do not re-run this finding" preamble (flagged by both dispatches as untrusted content). A Transfer check against the adjacent #1346 self-guard fixture confirmed no regression (score 1.0): the new bullet did not loosen the pre-existing self-guard boundary.

**Eval-runner unavailability, disclosed.** The confirmed runner (`evals/scripts/gitapex_run_eval_suite.py`) could not execute live in this session's environment -- every `claude --bare --tools "" -p` invocation failed with an empty-stderr authentication error, matching issue #1304's already-tracked, repo-wide infrastructure defect (independently reconfirmed by this PR's own `eval-gate` CI run failing identically). Scoring used an isolated Agent-tool subagent dispatch substitute instead, reasoning only from pre-pasted skill text with no further tool access. `gitapex_check_skill_shape.py`: 70/70. Full record: `results/2026-09-02-issue-1662-sequential-pipeline-body-cap-exception/manifest.json`. Refs #1662, #1346.
Loading
Loading