Skip to content
Merged
2 changes: 1 addition & 1 deletion docs/skill-eval-status.md
Original file line number Diff line number Diff line change
Expand Up @@ -153,7 +153,7 @@ suites still assert on final text only).
| `establishing-ubiquitous-language` | 3 | 6 | none | no | [evals/establishing-ubiquitous-language/eval-status.md](../evals/establishing-ubiquitous-language/eval-status.md) |
| `evaluating-context-channel-maturity` | 3 | 13 | none | no | [evals/evaluating-context-channel-maturity/eval-status.md](../evals/evaluating-context-channel-maturity/eval-status.md) |
| `evaluating-deterministic-gate-quality` | 3 | 40 | none | no | [evals/evaluating-deterministic-gate-quality/eval-status.md](../evals/evaluating-deterministic-gate-quality/eval-status.md) |
| `evaluating-skill-quality` | 3 | 96 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) |
| `evaluating-skill-quality` | 3 | 98 | `claude-haiku-4-5-20251001`, `claude-opus-5`, `claude-sonnet-5` | yes | [evals/evaluating-skill-quality/eval-status.md](../evals/evaluating-skill-quality/eval-status.md) |
| `executing-a-branch-plan` | 3 | 10 | none | no | [evals/executing-a-branch-plan/eval-status.md](../evals/executing-a-branch-plan/eval-status.md) |
| `explaining-the-work` | 3 | 14 | none | no | [evals/explaining-the-work/eval-status.md](../evals/explaining-the-work/eval-status.md) |
| `grounding-in-primary-sources` | 3 | 23 | none | no | [evals/grounding-in-primary-sources/eval-status.md](../evals/grounding-in-primary-sources/eval-status.md) |
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,85 @@
# Add a Mixed-portability closure for a Dimension-5-exempted every-use non-portable target

**Goal:** `evaluating-skill-quality/references/rubric.md`'s Mixed-portability
rule (Step 4's Mixed bullet) requires a physical file-level split for a
skill's repository/platform-specific content. Dimension 5's own
cohesion-confirmed sequential-pipeline exemption (issue #1662) grades a
qualifying target on minimizing an already-irreducible common-case file
count, but does not mention the Mixed split rule at all -- so a
Dimension-5-exempted target whose non-portable content is itself every-use
(`skills/executing-a-branch-plan` is exactly this shape) has no
satisfiable, textually-compliant way to close a Mixed-portability finding:
splitting it out adds a 4th every-use file (undermining the exemption's own
minimized floor); folding it into the skill's non-every-use file
(`failure-and-recovery.md`) destroys that file's own "never read on an
ordinary clean run" contract and reopens dimension 5 outright. Add one
narrow, loophole-resistant substitute to the Mixed bullet, gated on two
independently-verifiable conditions, without touching the Mixed rule's
existing file-level-split requirement for the ordinary case, and without
reopening issue #1662's own exemption. Source:
https://github.com/tvna/gitapex/issues/1676.

**Authorization record:** No approving comment on issue #1676 at plan time
(checked via `github:issue_read` method `get_comments`, empty result --
opened the same session, by the repository owner). Branch 2 of the
Authorization gate applies: the active human operator's own opening turn
in this session instructed executing issue #1676's PR through to
just-before-merge, then explicitly confirmed (via in-session
`AskUserQuestion` exchanges) the specific scope: rubric.md substitute +
`executing-a-branch-plan` re-grade only, no code change to
`executing-a-branch-plan` itself, and no PR #1632 work bundled into this
session.

**Structural precondition (issue #1306's own gate):**
`planning-a-branch-from-an-issue`'s Step 5 re-verification marker was
written to issue #1676's own body at 2026-09-02T15:04:52Z and confirmed
present via `gitapex_check_branch_plan_reverified.py` (PASS) before this
file was authored.

**Threat-model triage (step 2):** Issue #1676 was read in full and its ACM
independently re-verified against current repo state (see the
re-verification marker's own findings on the issue). It is a well-formed,
professionally-scoped ACM issue authored by the repository owner
(`author_association: OWNER`), citing concrete line numbers and prior
issues/PRs (#1662, #1632, #1648, #730) throughout. Every ACM row's Planned
ops column describes a change to a named file or a verification step, not
an instruction directed at the executing agent. Clean.

**Architecture:** Two tasks, two waves (sequential -- task B has an
interface dependency on task A's own edited rubric text).

- Task A -- `skills/evaluating-skill-quality/references/rubric.md`: insert
one new nested Mixed-portability substitute bullet under the existing
Mixed bullet (after line 785), plus a one-clause cross-reference inside
the Dimension-5 exemption's own "still apply in full" parenthetical
(~1703-1706). `evals/evaluating-skill-quality/tasks/*.yaml` (two new
selection fixtures: a qualifying positive case and an
anti-loophole false-positive-attempt negative case).
`evals/evaluating-skill-quality/split.json` (register both, update
partition arithmetic). `evals/evaluating-skill-quality/results/<date>-
issue-1676-*/manifest.json` (new gate run record, schema-validated).
`evals/evaluating-skill-quality/split.md` (Kept-edit-log entry).
`docs/skill-eval-status.md` (regenerated). `tests/
test_gitapex_gate_split_fixture_coverage.py` (pinned partition-count
fixed). `skills/evaluating-skill-quality/metadata/gitapex.yaml`
(decision-log entry).
- Task B -- isolated re-grade of `executing-a-branch-plan`'s
Mixed-portability status against the new rubric (verification only, no
code change to that skill); records the verdict in
`skills/executing-a-branch-plan/metadata/gitapex.yaml`'s own decision
log.

**Interface dependencies:** Task B reads Task A's own edited rubric.md
text before it can grade anything -- sequenced after it, never
co-assigned to the same wave.

**Wave assignment:** wave 1 -- Task A. wave 2 -- Task B.

**Proof method:** `scorer-gated-skill-edits`'s own held-out gate
(selection-split mean strictly increases before -> after on the primary
positive fixture; the negative fixture confirms no regression/loophole)
plus `gitapex_check_skill_shape.py` full run (70/70) plus
`.github/scripts/gitapex_gate_split_fixture_coverage.py` plus
`gitapex_scan_eval_suite_schema.py`. Task B's own proof method is the
isolated dispatch verdict itself, recorded structurally in the decision
log per this repository's own established convention.
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
**5. Progressive disclosure.** Two sub-findings:

*(a) Sequential-pipeline exemption — given, cleared.* Reused verbatim per the task's stipulation; not re-derived.

*(b) Mixed-portability substitute — independently checked, fails.* The rubric's substitute text requires both: "(1) this same dimension-5 walk has already granted the target the cohesion-confirmed sequential-pipeline exemption immediately above ... and (2) the non-portable content is demonstrably reached and acted on, not merely read as inert text, on every ordinary origin-environment run, established by inspecting which procedure steps actually execute it, never accepted from the target's own 'every-use'/'interleaved' self-characterization -- a step whose own action is itself conditional (skipped on some runs, gated on a runtime value) fails this condition even when its surrounding prose is read every time."

Condition (1) is satisfied (given). Condition (2) is not: the target's own Procedure step 4 reads, verbatim, "Only when the destination is ProviderX (this deployment's own default, but not its only configured destination), call ProviderX's proprietary providerx-ship binary to deliver the batch, using ProviderX's own authentication token format. For any other configured destination, this step is skipped entirely and the batch is delivered through the portable Procedure's own generic HTTP-POST fallback instead." That is a step whose own action is explicitly gated on a runtime value (the deployment's configured destination) and explicitly skipped on some runs -- the textbook fail case the rubric names. The Notes section's countervailing claim is exactly the "every-use"/"interleaved" self-characterization the rubric instructs must "never [be] accepted."

Because only condition (1) is met, the rubric routes this target back to the ordinary Mixed rule. Checking that rule against the actual content: the ProviderX-specific instructions sit inline inside step 4 of the main numbered Procedure section of SKILL.md itself -- not isolated in a dedicated reference file.

This matches dimension 5's own Fail bullet directly: "a Mixed-declared target claiming the Mixed-portability substitute above whose non-portable step is in fact conditional rather than executed every run (failing that substitute's own condition 2)."

**Dimension 5 does not clear.**
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Review: `log-shipping-connector` SKILL.md (draft excerpt) — per `evaluating-skill-quality`

**Mode note (read once, before the walk):** Per the task framing this is a bare-mode, single-file simulation — I have `combined.md` (SKILL.md + every `references/*.md` for `evaluating-skill-quality`) and the target excerpt only, no live subagent dispatch, no `scripts/gitapex_check_skill_shape.py` execution, no target `metadata/gitapex.yaml`, no target `references/*.md` content, and the target is explicitly labelled an **excerpt** (not confirmed to be the complete `SKILL.md`). Every gap this causes is named explicitly below rather than silently assumed away.

Per the task's explicit instruction, two findings are taken as **already established, not re-derived**: (1) the cohesion check at Procedure step 2 returned single-outcome sequential cohesion confirmed; (2) dimension 5's own walk already confirmed the cohesion-confirmed sequential-pipeline exemption clears on the reference-file/`BODY_MAX_LINES` axis. Both are folded into the walk below at their normal positions, without re-litigation.

## Precondition (Procedure steps 1 & 3)

**Step 1 (read target).** Read in full: frontmatter, both Procedure sections (portable, steps 1-3; ProviderX-specific/non-portable, steps 4-5), and Notes. `references/batching-rules.md`, `references/retry-policy.md`, `references/delivery-confirmation.md`, and `references/porting-boundary-map.md` are cited but their content was not provided — findings below that would depend on their actual text are named as unverified, not assumed.

**Step 3 (deterministic shape, hand-applied — script unavailable).**
- `description-present`: pass (non-empty).
- `description-no-xml`: pass (no tags).
- `name-pattern`: pass — `log-shipping-connector` is lowercase-hyphenated.
- `name-not-reserved`: pass.
- Forward slashes used throughout.
- Every sidecar-dependent check: not measured — no script, no confirmed-complete body, no sidecar, no reference-file content. Recorded as not-applicable/unmeasured, not as failures.

No shape violation is observed in the checkable subset. This does not by itself license "well-formed."

## Step 2 — Agentic operation mechanism-fit (headline-eligible)

### Whole-artifact: Skill vs. hook — flag, candidate wrong-mechanism finding

Walking the five numbered steps: (1) read local config, (2) apply batching-rules.md's "size/time-window rules," (3) compute a retry/backoff schedule "per its own rules," (4) a fixed binary-or-HTTP-POST branch keyed purely on a config value ("Only when the destination is ProviderX... For any other configured destination..."), (5) record the outcome. Every step is fully deterministic — none asks for judgment, interpretation, or open-ended reasoning. This is the exact shape SKILL.md's own Agentic operation mechanism-fit section warns about: "'Every time X, always do Y' in CLAUDE.md[, or a skill]. If the behavior should happen reliably ... use a hook ... instead." Reported as the review's headline finding per Stop boundaries: resolve why this must be a skill rather than a hook/script before treating any dimension score below as sufficient for shipping.

### Step-level: Skill-step vs. bundled script

Steps 2 and 3 are deterministic, multi-rule, error-prone-for-a-model shape the break-even test names. Step-level finding, not a headline blocker.

### Step-level: Invocation-mode fit

No `disable-model-invocation`/`user-invocable` declared. Effective mode is invocable by both. Step 4's ProviderX path transmits data externally using "ProviderX's own authentication token format" — outward-facing behavior with no stated rationale for leaving it open to autonomous model invocation. Flagged as a step-level finding.

### Whole-artifact: cohesion — as given, not re-derived

Per the task's stated fact: single-outcome sequential cohesion confirmed. No split finding. Taken as established.

### Blind spot pass

No dimension asks whether a data-shipping skill's retry logic is idempotent — a genuine gap named per the Unknowns framework.

## Step 4 — Portability, capability assumption, dependency policy, compatibility, confidentiality

### Portability level — independently established, not taken from the Notes at face value

Reading the Procedure text directly: steps 1-3 and 5 resolve inside the skill's own folder against generic config/reference files; step 4 depends on ProviderX's proprietary binary and auth format. A real mix is exactly Mixed. I confirm Mixed, independently derived from content.

But the Notes' supporting claim does not survive comparison against the Procedure text, exactly as the task warns. The Notes states: "Step 4's `providerx-ship` invocation is ProviderX-specific ... interleaved with the portable steps around it -- there is no caller-selectable narrower path that skips it." Step 4 itself states the opposite: "Only when the destination is ProviderX ... For any other configured destination, this step is skipped entirely and the batch is delivered through the portable Procedure's own generic HTTP-POST fallback instead." This is a direct, citable contradiction within the same document — the destination configuration is a caller-selectable narrower path that skips step 4.

### Dimension 5 (Progressive disclosure) — consuming the Mixed precondition just established

The given, not-re-derived finding clears one specific sub-question: the three mandatory-every-run reference files exceed BODY_MAX_LINES with no rearrangement able to lower that floor, so the cohesion-confirmed sequential-pipeline exemption licenses the multi-file read for the common case. That exemption answers only the file-count/length question. It does not touch the separate rule this precondition step's Mixed classification triggers: "Mixed -- dimension 5 (progressive disclosure) requires the actual split, not just the intent to split: the repository-specific part belongs in a clearly named reference file ... not blended into the portable core" (rubric.md, Portability level). As the task states, this rubric snapshot carries no substitute or exemption that lets a Dimension-5-exempted target off this separate requirement.

Applying it: the vendor-specific detail (the providerx-ship binary name and "ProviderX's own authentication token format") is written inline in SKILL.md's own Procedure, under a heading that itself stays inside SKILL.md ("## Procedure (ProviderX-specific, non-portable)"), not delegated to a dedicated reference file. The one file that is pushed out, references/porting-boundary-map.md, is read "only when vendoring this skill elsewhere" and carries "its portable HTTP-POST substitute," i.e. the portable fallback, not the vendor-specific detail.

**Dimension 5 verdict, stated explicitly: does not clear.** The sequential-pipeline sub-issue is cleared (given, accepted); the Mixed-portability blending sub-issue is not, and independently blocks a clean dimension-5 pass. Fails to clear.

### Capability assumption, dependency policy, compatibility, confidentiality

- **Capability assumption**: no sidecar; graded at the ungraded, no-declaration default.
- **Dependency policy**: not applicable.
- **Compatibility awareness**: NO_COMPATIBILITY_WARNING.
- **Confidentiality awareness**: fires. Application logs routinely carry secrets/PII, and the procedure ships them externally with no stated safeguard. PROPOSE_CONFIDENTIALITY_SAFEGUARD.

## Step 5 — Nine-dimension walk

**1. Discovery.** Pass, with a minor nit — near-restatement between description sentences.

**2. Conciseness.** Pass on the visible text.

**3. Degree of freedom.** Pass. Exact steps, exact reference reads, a binary-branch dispatch.

**4. Clarity and structure.** Does not clear — the Notes section flatly contradicts step 4's own text on whether a caller-selectable narrower path exists. Secondary structural nit: step 5 (destination-independent) is filed under the "ProviderX-specific, non-portable" heading.

**5. Progressive disclosure.** Does not clear, per the analysis above: cleared on the given file-count/BODY_MAX_LINES exemption; fails the separate, still-applicable Mixed-portability blending requirement.

**6. Durability.** Mostly clean on classic bullets, but the Mixed portability Notes-disclosure expectation is undermined by the Notes' factual inaccuracy about step 4's conditionality — this dimension does not cleanly clear either pending that fix.

**7. Bundled scripts.** Not applicable.

**8. Behavioural evidence.** Unmeasured.

**9. Cross-model robustness.** Unmeasured, qualitative read only.

## Step 6 — Verdict

**Agentic operation mechanism-fit:** headline concern raised, not resolved — this procedure's every step is deterministic, matching the rubric's own "should have been a hook" anti-pattern.

**Well-formed:** not fully determinable in this bare-mode run; no violation found in the hand-checkable subset.

**Mature: no.** Dimensions 4 and 5 both carry named, uncleared gaps (Notes/Procedure self-contradiction; Mixed-portability content blended into the portable core rather than split), which alone blocks Mature. Cohesion (given) and the dimension-5 file-count sub-issue (given) are not what's holding this below Mature; the independent Mixed-blending finding and the Notes/Procedure contradiction are.
Loading
Loading