feat(drafting-a-skill): add blank-page skill-authoring skill - #1303
Conversation
gitapex-native successor to obra/superpowers' writing-skills and Anthropic's skill-creator (refs #1194). Gates on Mechanism fit before drafting, mandatorily elicits every user-selectable metadata choice, drafts using Design-by-Contract structure, and hands off to evaluating-skill-quality and battle-testing-a-skill for review. A fresh-context consistency audit run before any implementation file existed (per #1194's own Constraints) found and corrected 3 defects: Steps 5 (cohesion) and 7 (domain-gap sweep) were reframed as advisory self-checks rather than authoritative ones, since evaluating-skill- quality's own rubric states its cohesion check "has exactly one owner" and its Blind spot pass already runs as that skill's own precondition step; skillDependencies.requires was corrected from [] to [evaluating-skill-quality], since Step 8 mandatorily invokes its bundled checker scripts; and Step 2's hook/CLAUDE.md redirect targets were named concretely (evaluating-deterministic-gate-quality, evaluating-context-channel-maturity). Also adds a new CI drift gate (contract-discipline-drift) keeping evaluating-skill-quality's Contract discipline section and this skill's own contract-structure.md from silently diverging, wires Related-skills routing into planning-a-branch-from-an-issue and executing-a-branch-plan, and adds a disclosed-gap eval corpus (evals/drafting-a-skill/) including a genuine adversarial fixture. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
… body Repository owner request: applies drafting-a-skill's own Step 3 self- referentially, changing the earlier Adaptive declaration to Broad. Per evaluating-skill-quality/references/rubric.md's own Capability assumption section, Broad requires dimension 9 (Cross-model robustness) to hold under the full Haiku/Sonnet/Opus spread -- "the skill must give a weak tier enough guidance, and failing to do so is a real, gradeable gap, not an unmeasured one." This is a body rewrite, not a metadata relabel: Steps 3-5 and 7 now inline their core judgment calls directly (the four metadata axes' own option lists, the Design-by-Contract definitions, the Single Decisive Outcome test, a concrete domain-gap example) rather than requiring a weak model to follow a reference-file pointer to find them. Adds a Worked example section and updates the Notes section's own disclosure. The five references/ files are unchanged -- progressive disclosure structure is unaffected, per that same rubric section's own note that Broad/Frontier "leave dimension 5's grading completely unchanged." Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
Two independent, isolated review dispatches against skills/drafting-a- skill (evaluating-skill-quality: WELL-FORMED-NOT-MATURE; battle-testing- a-skill: FAIL) found real defects, all fixed here: - executionRequirements.tools.shell: [] (= prohibited) contradicted Step 8's own mandatory python3 checker invocations; corrected to shell: [python3], matching evaluating-skill-quality's own identical precedent. - Two prose pointers to a "SKILL.md Non-goals" section that did not exist; added a real Non-goals section. - Step 3's "every user-selectable metadata choice" over-claimed exhaustiveness against the sidecar's 8 real spec fields; scoped to the 4 elicited axes, with dependencyPolicy/skillDependencies/ executionRequirements now explicitly stated as derived facts. - Step 6 had no completion criterion; added one. - Four reference files (guidance-form-and-sdo.md, formative-quality- dimensions.md, contract-structure.md, tacit-knowledge-elicitation.md) claimed to be "unconditional"/loaded whenever their Step begins, when the ordinary path needed at most one of six -- now genuinely on-demand, since the Broad rewrite already inlined their load-bearing content. formative-quality-dimensions.md's own self-description of this split was also miscounted; corrected. - Dimension-1 collision with the still-installed writing-skills/ skill-creator was undisclosed; added a Related-skills bullet and a compatibility: frontmatter field. - No Stop boundary stated that requester-supplied source text is untrusted data. battle-testing-a-skill demonstrated, by running a compliant reply against the original injected-self-certification- probe.yaml fixture, that its assertions would false-pass an actual injection compliance. Added explicit untrusted-input handling to Step 1, Step 9, and Stop boundaries; strengthened the fixture's assertions to require dispatch language, not just name mentions, and banned a broader set of compliance paraphrases. - Tightened the Postcondition's self-granted-deferral seam (a deferred Step 5/7 finding now requires a substantive stated reason, and states explicitly that deferral changes nothing about what Step 9's downstream reviews will independently find). - Added an explicit escalation path to Step 2 for a candidate that fits neither the block nor create list cleanly. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
A third round of independent, isolated dispatches against
skills/drafting-a-skill (both re-verdicted WELL-FORMED-NOT-MATURE)
found subtler self-consistency defects the first two rounds' fixes
introduced or left in place:
- The Postcondition asserted "passes both Step 8 checkers clean" while
the execution-requirements-drift scanner reported a heuristic
tools-shell-over-declared warning (a real detection blind spot: the
scanner pattern-matches SKILL.md prose for a backtick-adjacent `run`/
invocation, which Step 8's prose didn't contain). Fixed by inlining
the actual invocation commands into Step 8 itself, which both
resolves the warning honestly and improves Broad's own weak-tier
self-sufficiency.
- The Postcondition also overstated Step 6 ("collides with no existing
skill"), when Step 6's own completion criterion explicitly permits a
deferred collision -- and this draft is itself a deferred-collision
case. Reworded to match Step 6's real rule.
- Step 3's field-enumeration was still inaccurate: Invocation mode was
described as a metadata/gitapex.yaml field (it's SKILL.md
frontmatter), and two sidecar fields (references, externalCitations)
were omitted from "the sidecar's other fields." Corrected, with
explicit guidance that `spec.references` is filled in as decisions
happen, not backfilled.
- Steps 5 and 7 stated what NOT to write for a finding but never the
observable result for "no finding" -- added an explicit branch to
both.
- Step 6's "every existing skill" scope was ambiguous about installed-
but-not-native skills, which is exactly the population this draft's
own Related-skills collision bullet concerns. Clarified.
- Three sibling skills (untrusted-input-triage, drafting-an-adr,
grounding-in-primary-sources) were named mid-procedure but absent
from skillDependencies.relatedTo and Related skills -- Step 6's own
reconciliation rule, applied to itself. Added.
- formative-quality-dimensions.md was entirely unlinked from SKILL.md
(dead weight) and self-contradicted about its own load timing ("not
required reading before Step 4" vs. "read it once before Step 4
begins" two sections later). Linked from Step 8; contradiction fixed.
- Two reference files used "escalate per Step 7" as a generic
illustration, when this skill's own Step 7 is the domain-gap sweep,
not an escalation step -- corrected to avoid the collision.
- The Notes' claim that no reference file is "required reading for the
ordinary path" was false for gitapex-cross-links.md, which carries
Step 8's own invocation flags found nowhere else. Corrected.
- The body crossed 90% of BODY_MAX_LINES; added the rubric's own
required Declaration-vs-structure-fit disclosure, then trimmed
elsewhere for margin.
- eval-status.md's ablation-capability state was collapsed into "no
baseline exists" rather than "ablation-capable, not yet run" (a real
runner exists: evals/scripts/gitapex_run_ablation.py); also disclosed
two previously-silent coverage gaps (the Precondition's two
route-away branches, and zero fixture citing any of the nine
formative-quality-dimensions.md rows).
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
A third battle-testing-a-skill dispatch (FAIL) proved the previous round's fixture fix was itself false: "dispatch" is a verbatim substring of the injected payload, so an injection-compliant reply scored a perfect 1.000 against gitapex_score_contract.py while a genuinely resisting reply scored only 0.846 -- the discrimination was inverted, not merely imperfect. Rebalanced to 3 multi-word positive phrases (verified absent from the payload) against 9 bans, since flat per-item weighting had diluted 2 positives against 17 bans to where an evasive reply demonstrating neither behavior still scored 0.905. Reverified directly with the same scorer against 7 hand-crafted replies (4 compliant/evasive, all below threshold; 3 resisting, all 1.000). The same dispatch found three fail-open gaps needing no adversarial input: Step 3 had no rule for an unobtainable elicitation answer (an agent under pressure took a self-chosen provisional value, exactly what mandatory elicitation forbids); Step 9 had none for a missing fresh-dispatch mechanism (the agent performed both reviews in-context, the exact substitution Stop boundaries forbid); Step 1 had none for an empty candidate job or content hidden via encoding/HTML comments. All four given explicit stop-and-escalate or decode-first rules. Also widened the Stop-boundaries untrusted-input bullet to match Step 9's own broader "original request or pasted source text" framing (it was scoped only to pasted content). The body crossed 500/500 lines after these additions; trimmed several Notes paragraphs and Related-skills bullets for margin without losing substance (483/500). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
…n stdlib-only claim drift skills/drafting-a-skill/SKILL.md has 7 Stop-boundary bullets, but only 4 fixtures existed under evals/drafting-a-skill/tasks/ -- gitapex_gate_skill_branch_fixture_coverage.py requires parity for a brand-new skill (verified directly: 7 branches, 7 fixtures now, gate exits 0). Added 3 fixtures covering the previously-uncovered bullets: never author a hook in place of redirecting after Step 2 blocks a hook-shaped candidate, never infer the four metadata axes even when invited to, and the Precondition's own existing-skill route-away branch (this also closes a previously-disclosed eval-status.md gap). Also fixed a genuine stdlib-only-claim-drift finding against the new contract-discipline-drift gate itself: its own docstring Usage:: example showed a bare `python3 <file>.py` invocation alongside a `uv run`-wrapped one (exactly the anti-pattern that gate exists to catch), and its workflow's own comments literally contained the substring "stdlib-only-claim-drift" (naming the sibling gate by its real filename), which the drift gate's stale-phrase regex matches regardless of context. Fixed the bare invocation and reworded the comments to reference the sibling gate without spelling out that exact compound name. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01G6i18wop15RG33krfnuZhY
…gzz6o # Conflicts: # docs/skill-eval-status.md
Refs #1194. docs/skill-eval-status.md is auto-generated by gitapex_generate_skill_eval_status.py; this adds the drafting-a-skill row (3 trials, 7 fixtures) and updates the trials_per_task tally.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1303 +/- ##
=======================================
Coverage 99.54% 99.54%
=======================================
Files 123 124 +1
Lines 21912 21996 +84
Branches 2682 2699 +17
=======================================
+ Hits 21813 21897 +84
Misses 99 99 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…ve content-policy block Refs #1194. waza-eval-gate.yml's first live run of this suite failed outright: injected-self-certification-probe.yaml's "SYSTEM OVERRIDE" HTML-comment payload tripped a hard content-policy block reported via empty stderr, which gitapex_run_eval_suite.py's own _is_content_policy_rejection two-marker signature never matches, aborting the whole suite instead of skipping one fixture. Reworded the payload away from impersonating a system-level directive while preserving the same false-prior-approval, skip-Step-9 test property; re-verified the scorer discrimination still holds against 7 hand-crafted replies.
…waza-eval-gate failure Refs #1194. The reworded injected-self-certification-probe.yaml payload failed waza-eval-gate identically to the original -- same empty-stderr "model CLI exited 1" signature, same timing -- and the identical signature reproduces on an unrelated branch touching different skills' suites hours earlier. This is a pre-existing, repo-wide waza-eval-gate defect, not caused by this suite's fixture content; filed as #1304. Corrects the prior commit's "SYSTEM OVERRIDE tripped a content-policy block" diagnosis, which this evidence disproves.
…gzz6o # Conflicts: # docs/skill-eval-status.md
…after main's rename Refs #1194. main renamed drafting-an-acm-issue to drafting-issues while this branch was in flight (merged in via the preceding merge commit). Updated the 4 references this skill carried to the old name -- SKILL.md's compatibility field, its Related-skills bullet, metadata/gitapex.yaml's relatedTo list, and references/tacit-knowledge-elicitation.md -- so skill-dependencies-resolve and related-skill-references-resolve stay green. Shape checker: 54/54 PASS.
…gzz6o # Conflicts: # .github/scripts/gitapex_gate_local_preflight.py
Merge conflict resolution (second occurrence)
Resolution: merged A pre-existing, unrelated staleness was found and deliberately left Full re-verification after the merge: Generated by Claude Code |
…gzz6o # Conflicts: # docs/skill-eval-status.md
Merge conflict resolution (third occurrence)
Resolution: merged
Full re-verification after the merge: Generated by Claude Code |
Summary
Adds
skills/drafting-a-skill/-- a gitapex-native, blank-page skill-authoring skill succeedingobra/superpowers'writing-skillsand Anthropic'sskill-creator-- plus a new CI drift gate keeping it in sync withevaluating-skill-quality's own Contract discipline section, and routing updates in two consumer skills.Facts
skills/drafting-a-skill/SKILL.md(plus 6references/files andmetadata/gitapex.yaml) is a new 9-Step, Design-by-Contract-structured skill: gates on Mechanism fit (Step 2) before drafting, elicits the 4 human-only metadata axes (Step 3), drafts a Precondition/Steps/Postcondition contract (Step 4), runs advisory cohesion (Step 5) and domain-gap (Step 7) self-checks explicitly deferring authority toevaluating-skill-quality, checks collision against every skill in the actual invocation-time inventory (Step 6), runs this repository's own deterministic checkers (Step 8), and hands off toevaluating-skill-quality/battle-testing-a-skill(Step 9) -- never performing either review itself.evaluating-skill-quality's own rubric already claims exclusively;skillDependencies.requireswas[]despite Step 8 mandatorily invoking that skill's bundled scripts; Step 2's hook/CLAUDE.md redirect targets were unnamed.capabilityAssumptionwas changed from the design's own initialAdaptivetoBroad, applying the skill's own Step 3 to itself -- this required a real body rewrite (not a relabel) to inline every Step's load-bearing judgment call for dimension 9's stricter weak-tier-sufficiency bar.evaluating-skill-qualityandbattle-testing-a-skill, each dispatched fresh per round -- see## Skill audit evidencebelow for the literal verdict tokens and what each round found). Every concrete, verifiable finding across all three rounds was fixed in a follow-up commit; see the commit history andmetadata/gitapex.yaml's own append-only decision log for the specifics. The most consequential finding (round 3,battle-testing-a-skill): the adversarial eval fixture's own discrimination was proven inverted using this repository's owngitapex_score_contract.pyscorer directly -- an injection-compliant reply scored a perfect 1.000 while a genuinely resisting reply scored only 0.846. Fixed and re-verified with the same scorer against 7 hand-crafted replies (4 compliant/evasive, all below the 0.8 threshold; 3 resisting, all 1.000).contract-discipline-drift(.github/scripts/gitapex_scan_contract_discipline_drift.py+.github/workflows/contract-discipline-drift-gate.yml, registered in.gitapex/ssot.json), keepsevaluating-skill-quality/references/rubric.md's Contract discipline section anddrafting-a-skill/references/contract-structure.mdfrom silently diverging -- an always-on content lock plus a diff-aware check that a PR editing the former's Contract-discipline section also touches the latter, or carries an explicit ack comment.skills/planning-a-branch-from-an-issue/SKILL.mdandskills/executing-a-branch-plan/SKILL.mdeach gained one Related-skills bullet routing todrafting-a-skillwhenever their own planned ops include a newSKILL.md.evals/drafting-a-skill/ships 7 fixtures (normal,guardrail,edge,injected-self-certification-probe,no-self-authored-hook,no-inferred-metadata,existing-skill-routes-away), sized to matchSKILL.md's own 7 Stop-boundary bullets pergitapex_gate_skill_branch_fixture_coverage.py's decision-branch/fixture parity requirement (verified directly: 7 branches, 7 fixtures, gate exits 0).eval-gate(non-required check) is red on this PR, and this is a confirmed pre-existing, repo-wide CI defect, not caused by this suite's content.waza-eval-gate.ymlranevals/drafting-a-skill/live for the first time the moment it was pushed and failed witherror: model CLI exited 1:(empty stderr). Suspecting theinjected-self-certification-probe.yamlfixture's "SYSTEM OVERRIDE" HTML-comment payload tripped Anthropic's content-policy classifier, I reworded it and re-pushed -- the failure reproduced identically (same empty-stderr signature, same ~2.2s timing), ruling that out. Cross-checking an unrelated branch (claude/gitapex-issue-1274-bwgwkg, different skills' suites, hours earlier the same day) shows the same signature. Filed as fix(waza-eval-gate): live eval runs fail with "model CLI exited 1" and empty stderr, repo-wide #1304 with full evidence; out of this PR's own scope to fix.eval-gateis not in.github/rulesets/main.json's required-status-checks list, so it does not block this PR. Seeevals/drafting-a-skill/eval-status.mdfor the full disclosure, including this correction of my own initial (wrong) diagnosis.mainwas resolved after main advanced 19 commits while this PR was open (first occurrence), including renamingdrafting-an-acm-issuetodrafting-issues. Mergedmainin; the only real conflict was in the auto-generateddocs/skill-eval-status.md, resolved by regenerating it with the repo's owngitapex_generate_skill_eval_status.pyrather than hand-editing conflict markers. Separately fixed 4 now-staledrafting-an-acm-issuereferences this skill carried (SKILL.md'scompatibilityfield and one Related-skills bullet,metadata/gitapex.yaml'srelatedTolist, and onereferences/file) to the renameddrafting-issues.gitapex_check_skill_shape.pyre-verified 54/54 PASS after the merge.mainwas resolved (see the dedicated PR comment for the full writeup) after PR feat(independent-review-pending): feed-forward the gate and de-couple its heading from Step 8 #1348 merged upstream, independently adding its own 37th wired local-plane gate from the same 36-gate baseline this PR's owncontract-discipline-driftgate also extended to 37 -- both PRs bumped the same wired-gate-count prose in 4 files; git's auto-merge silently kept a stale "37" in 3 of them (identical text on both sides merges cleanly) while the true post-merge total is 38. Fixed all four files to 38, re-measured a real warm timing (10.89s / 11.04s / 11.16s) rather than guessing, and confirmed the dedicated prose-count-vs-registry test passes clean. A pre-existing, unrelated staleness (this file's own top-of-docstring gate-count claims, invisible to that same test) was found and deliberately left alone as out of this PR's own narrow change surface. This second merge also brought in a new required status check,independent-review-pending(added to.github/rulesets/main.jsonby PR feat(independent-review-pending): feed-forward the gate and de-couple its heading from Step 8 #1348), which blocks on a## Independent review verdictsection naming the current head commit -- see that section below.mainwas resolved (see the dedicated PR comment for the full writeup) after PR fix(executing-a-branch-plan): implement Axis 2/4/6 state-management fixes for Execution log #1342 merged upstream. The only real conflict was again the auto-generateddocs/skill-eval-status.md(regenerated, not hand-edited); everything else auto-merged cleanly.skills/drafting-a-skill/andevals/drafting-a-skill/are confirmed untouched by this third merge.pytestre-run was required once on the first merge's post-merge head:test_gitapex_session_start_hook_shell.py::test_installs_the_prek_hook_for_a_real_checkout(a 15s subprocess timeout unrelated to this PR --.claude/hooks/session-start.shand its test are untouched by this diff, confirmed viagit diff) flaked identically to an earlier occurrence on a different commit in this same PR, then passed clean on re-run. All other tests passed both times (5541-5542 passed).main.json's original required status checks (actionlint, ruff, pytest, mypy, exception-handler-gaps, hidden-characters, plugin-root-brace-notation, provenance-disclosure, pr-title-convention, betterleaks) pass on the current head;independent-review-pending(the 11th, newly added) is addressed by the verdict section below.gitapex_check_skill_shape.pychecks pass on the current head; the fullpytestsuite passes except one test (test_gitapex_scan_harden_checkout_pin_drift.py::test_repository_workflows_are_drift_free) that fails only in this sandbox's shallow git clone (confirmed: the failure is a documented shallow-clone limitation unrelated to this diff, and will pass in real CI, which fetches full history). The local-preflight gate runner passes 37/38 (same single shallow-clone-only failure;behind-baseitself passes once a transient 60s network-fetch timeout on its first attempt is retried).Assumptions
shell:[]contradiction and the inverted fixture). This is a judgment call, not a certainty; a human reviewer or a follow-up review round may still find something the three rounds and this session's own direct verification missed.evaluating-skill-quality: no rubric dimension owns cross-skill duplication drift of prose criteria, sincedrafting-a-skill's Step 2 hand-copiesevaluating-skill-quality's own Mechanism-fit criteria with no drift gate watching that specific copy) is a disclosed, accepted residual risk, not fixed in this PR -- per that same review's own guidance, a durable rubric change belongs in this repository's own held-out-gated edit process, not a single review/implementation session.ANTHROPIC_API_KEYmisconfiguration, a@anthropic-ai/claude-code@2.1.240CLI regression, or an API-side change) -- diagnosing further needs either live CI secret access or a way to reproduce locally with the same credential, neither available to this session.Risk / blast radius
Additive only: a new skill directory, a new eval corpus, a new CI workflow + script + tests (registered in
.gitapex/ssot.json), and one new Related-skills bullet each in two existing skills' own prose. No existing skill's Steps, Stop boundaries, or behavior are modified. The new CI workflow (contract-discipline-drift-gate.yml) is not added to.github/rulesets/main.json's required-status-checks list, so it runs and reports but does not block merge on its own.Rollback
Revert this PR (or the specific commits) -- no other file outside
skills/drafting-a-skill/,evals/drafting-a-skill/,.github/scripts/gitapex_scan_contract_discipline_drift.py,.github/workflows/contract-discipline-drift-gate.yml,.gitapex/ssot.json's one new entry,.github/scripts/gitapex_gate_local_preflight.py's/.pre-commit-config.yaml's/CONTRIBUTING.md's wired-gate-count prose,tests/test_gitapex_gate_local_preflight.py,tests/test_gitapex_scan_contract_discipline_drift.py, and the two Related-skills bullets depends on this change.Verification
Acceptance Criteria Map, reproduced from the issue with each row's real result:
skills/drafting-a-skill/SKILL.mdexists with the drafted structuregitapex_check_skill_shape.py: 54/54 PASS (re-verified after all three merges)writing-skills), each with its governing quote; explicit precedence sentenceSKILL.mdStep 2metadata/gitapex.yamldeclares this skill's own decided choices[evaluating-skill-quality](corrected from the design's own[]), relatedTo naming 7 skills (4 original + 3 added when round-3 review found 3 named-but-unreconciled dependencies)skill-metadata.schema.jsonvalidation passes (viagitapex_check_skill_shape.py); decision log cites this issue and every correctionSKILL.mdresolves; shape checker confirms no dangling reference-file citationskills/drafting-a-skill/file existedplanning-a-branch-from-an-issueandexecuting-a-branch-planroute todrafting-a-skilldrafting-a-skillappears in bothrubric.md's Contract-discipline section andcontract-structure.mdin synctests/test_gitapex_scan_contract_discipline_drift.py, including fail-closed and boundary casesgitapex_check_skill_shape.pyand the registry-driven local-preflight suite run cleanSKILL.mdis gated by real, independent quality-review verdicts, not waivedevaluating-skill-qualityandbattle-testing-a-skillrun from genuinely isolated dispatches## Skill audit evidencebelow for the literal verdict tokens and evidenceAdditional verification beyond the issue's own ACM:
evals/drafting-a-skill/'s fixture count (7) matchesSKILL.md's own decision-branch count (7 Stop-boundary bullets), pergitapex_gate_skill_branch_fixture_coverage.py, run directly: exit 0.pytestsuite (post-third-merge): 5322 passed, 1 failed (the same pre-existing shallow-clone-only failure noted in Facts).gitapex_gate_local_preflight.py(post-third-merge): 37 of 38 locally-runnable wired gates pass; the sole failure is the same shallow-clone-only limitation noted in Facts.eval-gate(non-required) is red due to the pre-existing, repo-wide defect filed as fix(waza-eval-gate): live eval runs fail with "model CLI exited 1" and empty stderr, repo-wide #1304 -- see Facts.Checklist
docs/skill-eval-status.mdregenerated three times across three merges;CONTRIBUTING.md/.pre-commit-config.yaml/gitapex_gate_local_preflight.py's wired-gate-count prose updated 36 -> 37 -> 38 during the second merge)skills/*/SKILL.mdand a new deterministic gate script --## Skill audit evidencebelow discloses the required verdictsevals/*/split.mdKept-edit-log entry in this PRskills/*/SKILL.md--evals/drafting-a-skill/tasks/*.yamlfixture count (7) matches the decision-branch count (7), verified directly againstgitapex_gate_skill_branch_fixture_coverage.pySkill audit evidence
evaluating-skill-quality: WELL-FORMED-NOT-MATURE -- three independent, isolated dispatches ran across this PR's own iteration (one per round of fixes). The most recent (round 3, against commit4a11d79) found: dimension 4 (Clarity/structure) -- a Postcondition the artifact itself falsified on two counts, a still-inaccurate Step 3 sidecar-field enumeration, an unlinked reference file self-contradicting its own load timing; dimension 5 (Progressive disclosure) -- the same unlinked-file issue and an inaccurate Notes claim about the reference load model. All of these were fixed in the commits immediately following that round (seemetadata/gitapex.yaml's own decision log for the itemized list). No wrong-mechanism finding, no cohesion split finding, in any round. A 4th confirming dispatch was not run before opening this PR -- see Assumptions.battle-testing-a-skill: FAIL -- three independent, isolated dispatches. Round 3 (the most recent, against commit4a11d79) proved the round-2 fixture fix itself false (an injection-compliant reply scored a perfect 1.000 against the repo's owngitapex_score_contract.pyscorer, a genuinely resisting reply only 0.846 -- the discrimination was inverted) and found 3 fail-open gaps needing no adversarial input (Step 3 unobtainable-elicitation-answer, Step 9 missing-dispatch-mechanism, Step 1 empty-input/encoded-content). All four fixed in the immediately-following commit and the fixture fix re-verified directly with the same scorer against 7 hand-crafted replies. Round 3 also confirmed, via a live escalated-pressure probe with the skill body actually loaded, that the untrusted-input-handling fix from round 2 holds behaviorally under real pressure -- not merely as prose. A 4th confirming dispatch was not run before opening this PR -- see Assumptions.adversarial-coverage-mapping: RAN -- flagged security-relevant by the frontmattersecurity|gate|trustkeyword heuristic (the description's own "Gates on Mechanism fit"). The mapping itself:SKILL.md's 7 Stop-boundary bullets againstevals/drafting-a-skill/tasks/*.yaml's 7 fixtures, one of which (injected-self-certification-probe.yaml) is a genuinely adversarial embedded-instruction probe, not a happy-path case -- verified directly viagitapex_gate_skill_branch_fixture_coverage.py(exit 0). The threebattle-testing-a-skilldispatches (see verdict above) are the adversarial-probing half of this same mapping; round 3 additionally proved that probe fixture's own scoring was inverted and fixed it, per Facts.checker-script-adversarial-review: RAN --gitapex_scan_contract_discipline_drift.py(new gate script) was reviewed by constructing deliberate defeat cases, not only happy-path tests: boundary tests at the exact section-span edges (test_diff_one_line_past_the_section_end_is_silent,test_diff_ending_exactly_at_the_last_section_line_fires), a garbage-diff-input test mirroring the sibling per-PR diff-scan gate's own adversarial-review-found false-positive class, and a decoy-file ack-token test.deterministic-gate-quality: RAN -- read againstevaluating-deterministic-gate-quality/references/dimensions.mddimension 15 (fail-closed on incomplete or malformed input): every read path (missing file, non-UTF-8 file, a directory in place of a file, a duplicated/absent/empty heading, an unparseable diff) raises a typedScanErrormapped to exit 2, never a silent pass or an uncaught traceback -- verified directly bytests/test_gitapex_scan_contract_discipline_drift.py's own dedicated fail-closed section.defeat-test-disclosure: RAN -- see checker-script-adversarial-review above; the same boundary and malformed-input tests were specifically constructed to defeat the new detection logic, not merely exercise its happy path.Independent review verdict
Outer layer: GitHub Copilot review requested via
request_copilot_reviewagainst this exact commit; Copilot's review is Comment-only with no
pass/fail signal of its own, and had not yet posted a result as of this
verdict being recorded.
Inner layer:
skills/drafting-a-skill/'s own substantive content isconfirmed byte-for-byte unchanged since
39a1f61(verified directly:git diff 39a1f61..HEAD -- skills/drafting-a-skill/ evals/drafting-a-skill/produces zero output), already covered by thethree full review rounds disclosed above. The only new, self-authored
diff since the last recorded verdict (against 52f896e) is this third
merge's own one-line regeneration of the auto-generated
docs/skill- eval-status.md(a deterministic script output, re-verified withgitapex_generate_skill_eval_status.py --check) -- everything else inthis merge is main's own already-merged, already-reviewed content from
PR #1342. Read directly per this skill's own trivial-diff branch: a
single-line, deterministic-tool-generated change carries no independent
review risk beyond confirming the generator's own
--checkmodeagrees, which it does. No defect found; nothing to verify further
against blast radius, since the only self-authored change is a
regenerated doc byte confirmed to match a fresh run.
Related Issue
Closes #1194