chore(evaluating-skill-quality): (E) empirical comment-rubric validation (#1046)

[tvna]

Summary

Completes issue #1046's own Row 2 ((E) evaluation-driven empirical validation of the new dimension-7 comment rubric): a with-rule-vs-without-rule comparison on a real bundled script, on both an execute-only and a read-as-reference task. Row 1 (the scorer-gated KEEP gate on #1045's rubric.md draft) was attempted but is currently blocked on a CI-side ANTHROPIC_API_KEY failure -- see Facts.

Facts

• Row 3 (new selection-split fixtures) already landed via PR chore(evaluating-skill-quality): scorer-gated KEEP gate + (E) validation (#1046) #1333 -- not this PR's own work.
• Row 2 script pick: skills/scorer-gated-skill-edits/scripts/gitapex_score_contract.py, picked from PR refactor(scripts): trim changelog narration from bundled script comments #596's own 12 touched files. The originally planned candidate, gitapex_check_skill_shape.py, is now 6155 lines on main (confirmed via wc -l) -- too large for a tractable single-fixture comparison. gitapex_score_contract.py is the second-largest PR refactor(scripts): trim changelog narration from bundled script comments #596 file (510 lines), self-contained, and thematically apt: it is the very scorer this record's own trial outputs were graded with.
• Restyled variant: comments/docstrings only, applying dimension-7's rubric (durable rationale kept, changelog narration -- issue Add opt-in case-insensitive assertions to score_contract.py #628/fix(evals): reconcile output_contains case semantics between waza and gitapex_score_contract #858/Measure PR #612's real-world effect using in-repo script samples #618 citations -- trimmed or dropped). Verified zero non-comment/non-docstring/non-help-text code changed: ast.dump() of both files with docstrings stripped is identical outside string-literal content.
• Dispatch mechanism: 4 fresh Agent-tool subagents (2 tasks x 2 comment-style variants), each fully self-contained (only the target script's own text, no other context) and explicitly instructed not to use any tool and not to look at repository files. All 4 confirmed zero tool_uses in their own usage report.
• Results (scored via gitapex_score_contract.py itself against each task's own assertions): all 4 dispatches scored 1.000000 -- no correctness difference between current-style and restyled comments at this single-trial sample size. The restyled variant is a genuine, measured 3654 bytes / 14.8% smaller with zero logic change and zero correctness cost -- the same correctness-tied/cost-strictly-lower shape gitapex_score_contract.py's own pruning_compare() exists to reward.
• CI caught a real defect, fixed twice more after the initial commit (commits 9290a6dd, 479ae21e). (1) Independent review found 2 of the 4 committed "raw response" artifacts had silently drifted from the subagents' actual text (em-dashes normalized to commas, one "can't" -> "can not") -- restored both verbatim, re-scored, unchanged (1.000000 each). (2) test_real_repository_run_records_have_no_drift/test_main_returns_0_on_the_real_repository correctly rejected this record: evals/*/results/*/ is a governed corpus where record_contract must be exactly "gate-run" (a full scorer-gated-skill-edits selection-split KEEP/REJECT gate run) or "pre-contract" (reserved for eight frozen legacy records) -- this record is neither, since it has no selection-split verdict and used a different dispatch mechanism entirely. Relocated it out of the scanner's own scanned path instead of forcing an ill-fitting contract declaration: evals/evaluating-skill-quality/results/... -> evals/evaluating-skill-quality/e-validation/..., matching references/rubric.md dimension 8's own guidance for a baseline-recording artifact that is not a gate-run record. Confirmed no other evals/ scanner globs anything but eval.yaml/split.json/tasks/*.yaml/results/*/manifest.json, so the new sibling directory is inert to all of them. Full suite green after the fix.
• Row 1 attempt: .github/workflows/scorer-gated-keep-gate.yml dispatched from main at a045f8b9 (all default inputs) -- run https://github.com/tvna/gitapex/actions/runs/33246204028. Both arms failed in ~2 seconds with error: model CLI exited 1: (stderr intentionally redacted by redact_executor_failure_reason before reaching CI logs). Ruled out: the workflow's own preflight step confirmed the ANTHROPIC_API_KEY secret is non-empty and correctly wired; eval.yaml's executor: copilot-sdk field is a disclosed, unconsulted field per gitapex_run_eval_suite.py's own docstring, not a config mismatch (the real executor defaults to claude-cli, matching the CLI this workflow installs). The near-instant failure (no real model-call latency) matches the same failure class PR chore(evaluating-skill-quality): scorer-gated KEEP gate + (E) validation (#1046) #1333 already disclosed once, resolved that time by an API key rotation -- but this is this workflow's own first CI invocation of gitapex_run_eval_suite.py --executor claude-cli, so this may be newly exposed rather than a regression. This repo's own separate, pre-existing eval-gate check (waza-eval-gate.yml) failed identically on this same PR's every commit with the exact same error: model CLI exited 1: signature -- corroborating this is a repository-wide ANTHROPIC_API_KEY issue, not specific to the new workflow. eval-gate is confirmed not a required status check (same precedent PR chore(evaluating-skill-quality): scorer-gated KEEP gate + (E) validation (#1046) #1333 already established), so it does not block this PR. Full detail recorded on issue chore(evaluating-skill-quality): scorer-gated KEEP gate + evaluation-driven validation for bundled-script comment criteria (follow-up to #1045) #1046 itself (Re-verification note 3).

Assumptions

• Speculation: the ANTHROPIC_API_KEY GitHub Actions secret is invalid or expired again, mirroring the exact failure class PR chore(evaluating-skill-quality): scorer-gated KEEP gate + (E) validation (#1046) #1333 disclosed and fixed once already -- not independently confirmed, since this session has no access to the secret value and cannot test it directly.
• Speculation: a single-trial (n=1 per condition) correctness tie generalizes to a larger sample -- disclosed as a known gap in the committed manifest, not assumed true.

Risk / blast radius

Additive only -- one new evals/evaluating-skill-quality/e-validation/2026-08-29-issue-1046-comment-rubric-e-validation/ directory (a manifest.json plus artifacts). No existing file is modified; no runtime code path changes.

Rollback

git revert this PR's commits (443088d8, 9290a6dd, 479ae21e).

Verification

Acceptance Criteria Map (from issue #1046), restated:

Criterion	Proof method	Result
scorer-gated-skill-edits KEEP gate on the #1045 rubric.md draft	gitapex_run_eval_suite.py selection-split trials before/after via CI, gitapex_score_contract.py --compare-to, written run record	Blocked. Dispatch attempted (run 33246204028), both arms failed on an apparent ANTHROPIC_API_KEY issue outside this session's ability to fix or verify further. Needs the repository owner to check/rotate the secret, then re-dispatch.
(E) evaluation-driven empirical validation of the new criteria	A committed comparison record under evals/evaluating-skill-quality/ disclosing outcomes and known gaps	Done. evals/evaluating-skill-quality/e-validation/2026-08-29-issue-1046-comment-rubric-e-validation/manifest.json + artifacts/.
New selection-split fixtures for the new dimension-7 sub-rules	.github/scripts/gitapex_gate_split_fixture_coverage.py passes	Already done via PR #1333 -- not this PR's own scope.

Checklist

• Tests pass locally (uv run --frozen pytest -q tests/test_gitapex_scan_eval_results_schema.py -- 165 passed; plus eval-status/split-coverage/eval-suite-schema tests -- 222 passed; ruff/mypy/betterleaks pre-commit checks passed on every commit)
• Docs updated if behavior changed (n/a -- no behavior changed)
• Issue number cited in every commit
• If this PR adds/modifies a skills/*/SKILL.md, a docs/superpowers/specs/*.md design doc, a security-relevant skill, or a deterministic checker script (skills/*/scripts/*.py, evals/scripts/*.py, .github/scripts/*.py), a ## Skill audit evidence section discloses the required verdicts/waivers -- n/a, none of those paths are touched (only a new evals/evaluating-skill-quality/e-validation/* record directory)
• If this PR adds a new Kept-edit-log entry to any evals/*/split.md, that entry discloses a Transfer check line -- n/a, no split.md touched
• If this PR adds or increases a skills/*/SKILL.md's Stop-boundary bullets or named dispatch branches, evals/<skill>/tasks/*.yaml gained at least as many new fixtures -- n/a, no SKILL.md touched

Merge gate: independent review

This PR is also subject to the independent-review-pending required status check. There is nothing for you to do here now.

Related Issue

Refs #1046 (Row 1 still open/blocked -- see Facts)

[coderabbitai]

Important

• 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 929dfe04-35ee-4966-bd67-cc4726598a19

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.55%. Comparing base (a045f8b) to head (479ae21).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #1381   +/-   ##
=======================================
  Coverage   99.55%   99.55%           
=======================================
  Files         125      125           
  Lines       22167    22167           
  Branches     2720     2720           
=======================================
  Hits        22068    22068           
  Misses         99       99           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.