Skip to content

feat(gates): add unguarded-shell-pipe-in-docs gate - #1713

Merged
tvna merged 7 commits into
mainfrom
claude/issue-1567-gate-proposal-q6r5nq
Sep 3, 2026
Merged

feat(gates): add unguarded-shell-pipe-in-docs gate#1713
tvna merged 7 commits into
mainfrom
claude/issue-1567-gate-proposal-q6r5nq

Conversation

@tvna

@tvna tvna commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

Adds a documentation-lint gate (issue #1531, refs #1567) that flags an
unguarded cmd1 | cmd2-shaped shell pipe example, with no nearby pipefail
disclosure, in a skills/*/SKILL.md, a skills/*/references/*.md file, or a
checker/gate script's own module docstring -- and fixes the 8 pre-existing
instances of that exact shape the gate's own authoring found in this
repository, so it ships already green.

Live re-verification (against current origin/main, post-#1675 + post-#1701)
found #1488 and #1626 already fully resolved with no residual gap; see the
closing comment on #1567 for the reproduction. This PR is scoped to #1531
only.

Facts

  • Issue gate-proposal: retro #1530 repair 1: Undocumented fail-open via unguarded shell pipe in a merge-blocking gate's own invocation #1531's own table cites the motivating defect: the documented
    git log ... | python3 gitapex_check_task_commit_provenance.py
    invocation piped two commands directly together, so an upstream git log
    failure (a stale/unresolvable BASE ref, a rebase, a shallow worktree)
    silently reported a clean PASS: no commits in range instead of blocking
    a merge.
  • gitapex_check_task_commit_provenance.py's own module docstring already
    documents the fix (a two-step, never-piped invocation) and warns against
    the pipe form -- but nothing in this repository checked that no other
    documented recipe reintroduces the same shape.
  • The new gate's own authoring, run against the real repository, found 8
    further live instances of the same undisclosed-pipe shape: 7 checker/gate
    script Usage:: recipes (.github/scripts/gitapex_detect_changed_gate_ scripts.py, gitapex_gate_acm_issue_disclosure.py,
    gitapex_gate_independent_review_pending.py, gitapex_gate_pr_title_ convention.py, gitapex_gate_provenance_disclosure.py,
    skills/executing-a-branch-plan/scripts/gitapex_check_canonical_ governance_paths.py, gitapex_check_file_ownership_conflicts.py) and 1
    worked-example timing transcript
    (skills/evaluating-deterministic-gate-quality/references/gitapex-worked- examples.md). Each is fixed in this same change (a one-line pipefail
    caveat for the 7 scripts; an explicit exception-marker comment for the
    transcript, since altering its own copy-pasteable content would corrupt
    the recorded measurement).
  • #1488/#1626 re-verification (documented on gate-proposal-umbrella: local-hook fail-open remediation #1567 directly): reproduced
    both original defects live against hooks/check-pr-skill-audit- disclosure.sh on current origin/main (commit 9db591d, confirmed via
    git log to include both feat(gate-preconditions): establish environment/repo-state preconditions before gates run #1675 and fix(hooks): resolve PATH-dependent bare python3 false-deny (#1697, #1581) #1701 before testing). A synthetic
    create_pull_request payload modifying a .github/scripts/gitapex_gate_ *.py file with no ## Skill audit evidence section was denied
    (exit 2); a payload disclosing only 1 of 3 applicable lines was still
    denied
    , correctly naming the 2 remaining missing lines; a payload
    disclosing all 3 passed (exit 0). No residual gap for either issue.
  • Independent review (5 adversarial passes: correctness, blast-radius,
    reuse, convention, security) found a real, high-confidence gap: the
    regex could not see a pipe split across a shell line-continuation, and
    13 further real, previously-undetected instances of this same defect
    class already existed in the repository. All 13 are fixed in this same
    change; see the Independent review verdict section below for the full
    disposition of every finding.

Assumptions

  • Issue gate-proposal: retro #1530 repair 1: Undocumented fail-open via unguarded shell pipe in a merge-blocking gate's own invocation #1531's own text states no-raw-gh-cli-in-docs-gate.yml "already
    scans skills/*/SKILL.md and script docstrings for a different unsafe-
    shell-command shape" -- checked directly against that gate's own source
    (.github/scripts/gitapex_gate_no_raw_gh_cli_in_docs.py) and this is
    not accurate: that gate's own scope is docs/**/*.md only, via
    git ls-files -- docs, with no SKILL.md/references/script-docstring
    coverage at all. This does not change what the new gate should do
    (issue gate-proposal: retro #1530 repair 1: Undocumented fail-open via unguarded shell pipe in a merge-blocking gate's own invocation #1531's own proposed scope stands on its own), so it is noted here
    as a correction to the issue's own premise rather than acted on further.
  • "Nearby disclosure" is interpreted as: pipefail mentioned inside the
    same fenced block (Markdown) or the same module docstring (Python), or an
    explicit gitapex-allow-unguarded-shell-pipe HTML-comment exception
    marker (with a reason) directly above the fence/flagged line -- a
    deterministic, precedented interpretation (mirrors the sibling gate's own
    exception-marker convention) rather than a looser "anywhere in the
    surrounding prose" reading, which would need markdown-heading-based
    section detection with no precedent in this repository. Disclosed as a
    scope choice in the new gate's own module docstring (which spells out the
    exact marker syntax).

Risk / blast radius

New CI workflow (non-required by default; promoting it to a required status
check is a separate GitHub admin action) plus a new local-preflight-wired
gate. Docstring-only edits to 20 existing checker/gate scripts (no logic
changed) and one exception-marker line in a worked-example doc. No runtime
behavior changes anywhere in this diff.

Rollback

Revert this PR's merge commit. The new workflow file, gate script, and
.gitapex/ssot.json registry entry are removed together; the docstring
caveats and the exception-marker line revert with it (git revert, no
manual follow-up needed).

Verification

Acceptance Criteria Map:

Criterion Interpretation Planned ops Proof method Residual risk
[from #1488] skill-audit-disclosure PreToolUse hook did not block the PR create call Re-verify against #1675+#1701 before assuming residual work, per #1567's own resolution order Live reproduction against hooks/check-pr-skill-audit-disclosure.sh on current origin/main: synthetic create_pull_request payload, no-disclosure body Denied (exit 2) -- already resolved by #1675+#1701, no code change needed none identified
[from #1626] skill-audit-disclosure PR-body gap (2 of 6 lines checked) Re-verify against #1675+#1701 before assuming residual work Live reproduction, partial-disclosure body (1 of 3 applicable lines); confirm the remaining lines are still enforced Denied (exit 2), correctly named the 2 still-missing lines -- already resolved by #1675+#1701, no code change needed none identified
[from #1531] Undocumented fail-open via unguarded shell pipe in a merge-blocking gate's own invocation New documentation-lint gate, sibling to no-raw-gh-cli-in-docs .github/scripts/gitapex_gate_unguarded_shell_pipe_in_docs.py + CI workflow + .gitapex/ssot.json registration + 65 tests (example + Hypothesis property), plus 21 total pre-existing instances fixed in the same change (8 found at authoring time, 13 more found by independent review) Test-first: reintroduce the exact original defect shape, confirm FAIL; confirm the current fixed file passes; live mutation tests (dropping the regex's trailing \b boundary; dropping the escaped-backslash guard) confirm property tests catch each regression A documentation-only lint cannot verify the actual runtime invocation an agent performs matches the documented one (issue #1531's own stated residual risk); the whole-block/whole-docstring exemption scope and the line-wide (not span-aware) backtick exclusion are disclosed, pinned-by-test known gaps

Commands run and their output:

  • uv run --frozen python3 .github/scripts/gitapex_gate_unguarded_shell_pipe_in_docs.py -> OK: 92 Markdown file(s) and 167 Python file(s) carry no unguarded shell pipe examples.
  • uv run --frozen pytest tests/test_gitapex_gate_unguarded_shell_pipe_in_docs.py tests/test_gitapex_gate_unguarded_shell_pipe_in_docs_properties.py --cov=... --cov-report=term-missing -q -> 65 passed, 100% coverage
  • uv run --frozen pytest -q (full suite) -> 8071 passed
  • uv run --frozen python3 .github/scripts/gitapex_gate_local_preflight.py -> all 45 wired gate(s) passed
  • git push (pre-push hook re-runs the same local preflight) -> passed, twice more after the review-driven fix rounds

Checklist

  • Tests pass locally
  • Docs updated if behavior changed (CONTRIBUTING.md, .pre-commit-config.yaml, gate docstring/tests: wired-gate count 44 -> 45)
  • Issue number cited in every commit
  • Skill audit evidence disclosed below (no SKILL.md touched; checker/gate scripts touched)
  • N/A: no evals/*/split.md Kept-edit-log entry added
  • N/A: no skills/*/SKILL.md Stop-boundary/dispatch-branch change

Skill audit evidence

This diff touches no skills/*/SKILL.md, so the base battle-testing-a-skill
/ evaluating-skill-quality audits do not apply
(gitapex_compute_skill_audit_flags.py --check-diff confirms
skill-md-changed: false). It does touch multiple deterministic
checker/gate scripts (the new gate itself, .gitapex/ssot.json, and many
existing gate scripts with docstring-only edits):

  • checker-script-adversarial-review: RAN
  • deterministic-gate-quality: RAN
  • defeat-test-disclosure: RAN

Evidence for the new gate (gitapex_gate_unguarded_shell_pipe_in_docs.py):
false-positive calibration against the real corpus (Python type hints,
Markdown tables, backtick-quoted illustrative prose, a curl | bash
security discussion) before finalizing the consumer-token vocabulary and
the backtick-exclusion rule; fail-closed defeat cases constructed and
confirmed (empty repo, non-existent root, non-UTF-8 file, invalid Python
syntax -- all exit 2, never a silent pass, per dimension 15); live mutation
tests confirmed property tests catch each regression before it shipped; a
full independent review round (5 adversarial personas) found and this PR
fixed a real detection gap (a pipe split across a shell line-continuation,
with 13 real live instances) plus several smaller correctness/convention
issues -- full disposition in the Independent review verdict section below.
The modified existing gate scripts received docstring-only edits (no logic
changed), so their own prior fail-closed properties are unaffected.

Independent review verdict

  • Verdict: CLEAN
  • Verified commit: b416aad

Outer layer: GitHub Copilot review requested via
request_copilot_review at 2026-09-03T04:58Z. No response posted as of
this recording; per the repository owner's explicit direction this turn,
the wait for it is skipped rather than held for the full 30-minute window
this skill's own Step 8 otherwise specifies -- disclosed here as
owner-skipped, not as "unreachable."

Inner layer: reviewing-an-artifact run at low effort via 5 parallel
adversarial dispatches (correctness, blast-radius, reuse-and-simplification,
convention, security) against this PR's full diff. Findings and their
disposition, each independently verified (FABRICATED check + live
confirmation) before acting:

  • Fixed (code + regression test): the gate's regex could not see a
    pipe split across a shell line-continuation -- live-confirmed against 13
    real, previously-undetected instances of issue gate-proposal: retro #1530 repair 1: Undocumented fail-open via unguarded shell pipe in a merge-blocking gate's own invocation #1531's own defect class
    already in this repository; all 13 fixed with the same pipefail-caveat
    convention as the original 8, plus a new _effective_line join, 5 new
    example tests, 2 new property tests, and a live mutation-test confirming
    detection.
  • Fixed (code + regression test): _MARKDOWN_PATHSPECS lacked the
    :(glob) pathspec magic _PYTHON_PATHSPECS already carried, so a bare
    * could cross a / and match a doubly-nested SKILL.md/references file
    outside the intended scope -- live-confirmed via git ls-files, fixed,
    pinned with 2 regression tests.
  • Fixed (docs): a stale :func: cross-reference and a hardcoded,
    already-drifting test-count claim in the properties test's own
    docstring; a typo; an ssot.json trigger field omitting the
    _properties.py file; an imprecise attribution sentence.
  • Fixed (docs + disclosure + test): an overclaiming "closes both
    false-positive classes" sentence (a Markdown table cell literally equal
    to a consumer token, e.g. | Parser | jq |, still matches) -- softened,
    disclosed as a known gap, pinned by a regression test.
  • Disclosed as a known gap (not fixed), each pinned by a regression
    test
    : the whole-fenced-block/whole-module-docstring exemption scope
    (a pipefail mention or allow marker clears every match in the same
    unit, not only the one it was written to justify) and the backtick
    exclusion's line-wide, not span-aware, check (an unrelated backtick term
    earlier on a line can exempt a real, unquoted recipe later on that same
    line). Closing either needs a position-aware span check this gate does
    not implement; judged disproportionate to add for a documentation-lint
    advisory check whose own stated residual risk already accepts this class
    of imprecision, and a human reviewer reading a marker's stated reason
    against the block's actual content remains the backstop, the same trust
    this gate's own sibling already places in its analogous marker.
  • Considered and reverted: tightening find_violations's fail-closed
    check from "raise when neither corpus has any files" to "raise when
    either corpus alone is empty" -- live-confirmed this breaks 30 of this
    file's own tests (each legitimately populates only one category to
    isolate what it tests), so reverted; test_repository_scan_reaches_a_ real_tracked_set's own real-corpus floor assertion is judged the
    proportionate backstop for that concern instead.
  • Not in scope for this PR (pre-existing, not introduced by this
    diff)
    : a gitapex_gate_local_preflight.py's own "49 registered
    gates"/"24 excluded" claim not reconciling against the current registry,
    and skills/executing-a-branch-plan/scripts/test_gitapex_check_task_ full_verification.py's own stale "42 real wired gates" claim (already
    stale before this diff bumped the true count to 45) -- both predate this
    diff and are unrelated to its own change surface.
  • Checked, found clean, no finding: command-injection/subprocess
    safety, path-traversal, the workflow file's own permission scoping and
    action pinning, the fence-pairing/marker-indexing/off-by-one arithmetic,
    the docstring-only-ness of every existing-script edit, and the
    bookkeeping consistency of every "44 -> 45 wired gates" claim this diff
    did touch.

Zero outstanding confirmed findings remain after the fixes above; all
unconfirmed-concern/disclosed-gap items are recorded above rather than
silently dropped. Full audit trail (every candidate raised, and why each
was fixed, disclosed, reverted, or ruled out of scope) is in this PR's own
commit history (the review-driven fix commits following the initial
implementation).

Related Issue

Closes #1567
Refs #1531, #1488, #1626

Issue #1531 (refs #1567): a documented "git log ... | python3
gitapex_check_task_commit_provenance.py" invocation piped two commands
directly together, so an upstream git log failure silently reported a
clean result instead of blocking a merge. Add a documentation-lint
gate (sibling to no-raw-gh-cli-in-docs) that flags a cmd1 | cmd2
shaped shell pipe example in a SKILL.md, a skills/*/references/*.md
file, or a checker/gate script's own module docstring, when no
pipefail disclosure is nearby.

The new gate's own authoring found 8 pre-existing instances of this
exact defect class already in the repository (7 checker-script Usage
recipes, 1 worked-example transcript); each is fixed with a brief
pipefail-risk caveat or an explicit exception marker in this same
change, so the gate ships already green rather than red from day one.

Registers the gate in .gitapex/ssot.json (ci and local planes), wires
a CI workflow, and updates the wired-gate-count prose (44 to 45)
across CONTRIBUTING.md, .pre-commit-config.yaml, and the
local-preflight runner's own docstring and tests. Also records a
metadata sidecar correction for the one-line count drift the
pipefail-caveat edit caused in gitapex-worked-examples.md.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Aou8TJ15YpJfjaoQC6biK
Issue #1531. Add direct-call tests for describe(), markdown_violations_
in_text, violations_in_markdown_file, violations_in_python_file,
_tracked_files, _read_text, and GateUnguardedShellPipeInDocsArgs's
_root_must_exist accepting path -- each was already exercised
indirectly through find_violations()/main(), but function-body-test-
coverage-gate wants each mentioned by name in a test the same diff
adds.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Aou8TJ15YpJfjaoQC6biK
@coderabbitai

coderabbitai Bot commented Sep 3, 2026

Copy link
Copy Markdown

Important

  • 🔍 Trigger review

This repository does not receive automatic reviews because it has fewer than 10 stars.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Team

Run ID: a9f123e6-6875-4e51-83c2-3649184bb11f


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@tvna
tvna deployed to ruleset-verify September 3, 2026 04:47 — with GitHub Actions Active
@codecov

codecov Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 99.67%. Comparing base (9db591d) to head (b416aad).

Additional details and impacted files
@@           Coverage Diff            @@
##             main    #1713    +/-   ##
========================================
  Coverage   99.66%   99.67%            
========================================
  Files         152      153     +1     
  Lines       24792    24963   +171     
  Branches     2981     3010    +29     
========================================
+ Hits        24710    24881   +171     
  Misses         82       82            

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tvna
tvna deployed to ruleset-verify September 3, 2026 04:58 — with GitHub Actions Active
Issue #1531. Independent review (5 adversarial passes: correctness,
blast-radius, reuse, convention, security) found and this fixes:

- The regex could not see a pipe split across a shell line-continuation
  (a "cmd \\" line followed by "| consumer" on the next line) -- this
  repository's own common Usage:: wrapping convention for a long
  producer command. Live-confirmed 13 real, previously-undetected
  instances of the exact #1531 defect class already in the repository
  (git diff/git -c ... diff piped into a checker with no pipefail
  disclosure); each is now fixed with the same pipefail caveat as the
  first 8. Detection now joins a backslash-continued line with its
  successor before matching.
- _MARKDOWN_PATHSPECS lacked the :(glob) pathspec magic
  _PYTHON_PATHSPECS already carried, so a bare `*` could cross a `/` and
  match a doubly-nested SKILL.md/references file outside the intended
  one-level scope -- live-confirmed via git ls-files, fixed, and pinned
  with two regression tests.
- A stale :func: cross-reference and test-count claim in the properties
  test's own docstring.
- Softened an overclaiming "closes both false-positive classes" line
  (a Markdown table cell literally equal to a consumer token still
  matches) and disclosed the whole-block/whole-docstring exemption
  scope and the backtick exclusion's line-wide (not span-aware) shape
  as known gaps, each pinned by a regression test.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_018Aou8TJ15YpJfjaoQC6biK
@tvna
tvna deployed to ruleset-verify September 3, 2026 05:13 — with GitHub Actions Active
@tvna
tvna marked this pull request as draft September 3, 2026 05:28
@tvna
tvna marked this pull request as ready for review September 3, 2026 05:35
@tvna
tvna merged commit 45e265e into main Sep 3, 2026
43 of 44 checks passed
@tvna
tvna deleted the claude/issue-1567-gate-proposal-q6r5nq branch September 3, 2026 05:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

gate-proposal-umbrella: local-hook fail-open remediation

2 participants