Skip to content

feat(policy): dogfood the policy-tests gate on the maintainer repo - #10

Merged
maybebyte merged 5 commits into
mainfrom
chore/dogfood-policy-tests
Jul 30, 2026
Merged

feat(policy): dogfood the policy-tests gate on the maintainer repo#10
maybebyte merged 5 commits into
mainfrom
chore/dogfood-policy-tests

Conversation

@maybebyte

Copy link
Copy Markdown
Owner

Summary

Dogfoods the template's policy-tests layer onto the maintainer repo: a stdlib-only suite pins the gate config literals — basedpyright recommended + failOnWarnings, ruff select=["ALL"], fenced AGENTS.md just examples resolving to real recipes, and 40-hex SHA pins on every third-party Action — so none can be silently weakened. Closes gap #4 (policy-tests) and the offline half of gap #8 (Action SHA-pin policy) from docs/superpowers/plans/2026-07-01-dogfood-gap-audit.md.

Changes

  • New tests/policy/ suite (5 stdlib-only tests: re/tomllib/pathlib), with the docs-can't-lie check scoped to fenced AGENTS.md blocks — a documented divergence from the template's whole-doc scan (rationale in the committed design spec).
  • New policy justfile recipe, chained into just ci (now … test policy audit), mirroring the template's conditional ci.
  • AGENTS.md: new "Policy gate" section plus a sweep of every ci gate-list mention; design spec + implementation plan committed under docs/superpowers/.

Testing

  • TMPDIR=… mise exec -- just cici: all gates passed: generation matrix 75 passed in 3:44 (the 5 policy tests auto-collected), explicit policy leg 5 passed, pip-audit "No known vulnerabilities found".
  • mise exec -- just precommit — all commit-stage and pre-push hooks passed (basedpyright included).
  • uv run pytest tests/policy -q — 5 passed; uv run ruff check tests/policy — All checks passed!; uv run basedpyright — 0 errors, 0 warnings, 0 notes.

Notes for reviewers

  • zizmor N/A — no workflow file touched. CI enforces the new suite inside the existing test job with no CI change: bare uv run pytest auto-collects tests/policy (--collect-only shows the 5 tests).
  • AGENTS.md invariants are now load-bearing: the file must keep an even number of triple-backtick runs (18 today) and never name a template-only recipe (fuzz/mutate) with a just prefix — the new section documents both.

Related

The `audit` recipe annotates its forced `--no-dev` divergence inline, at
the site a maintainer reads when syncing against the template. The
`policy` recipe claimed a bare mirror while silently dropping
`--no-cov`, leaving AGENTS.md as the only record of the divergence.
@maybebyte

maybebyte commented Jul 30, 2026

Copy link
Copy Markdown
Owner Author

Code review

No issues found. Checked for bugs, template parity, and AGENTS.md compliance.

Eight independent review lenses (AGENTS.md adherence, shallow bug scan, git history, prior-PR comments, code-comment compliance, template parity, docs accuracy, adversarial test-efficacy) produced 18 raw findings, deduped to 5. Each was re-checked by two adversarial verifiers; none cleared the confidence bar. The main refutations:

  • The unanchored just ([a-z][a-z-]*) recipe regex has no current trigger, is inherited verbatim from the template it dogfoods, and fails loudly rather than passing silently.
  • The *.yml glob in test_actions_are_sha_pinned matches the repo's only workflow, and the mutable-tag risk is independently covered by the blocking zizmor job — consistent with the section's own "net-new value is annotation hygiene" framing.
  • "the first just <recipe> per fenced invocation" is accurate: the regex captures only the word following just , so just fmt-check lint typecheck yields fmt-check alone.

One sub-threshold item was addressed rather than dismissed: the policy recipe claimed a bare mirror of the template while dropping --no-cov, unlike audit, which annotates its forced --no-dev divergence inline. Fixed in 00c100f.

# Policy gate: pins the gate config literals (type mode, failOnWarnings, ruff select, Action
# SHAs) so they cannot be silently weakened — not the ignore lists or the recipe bodies.
# Stdlib-only; also auto-collected by `just test`. Mirrors template/justfile.jinja's `policy`,
# except `--no-cov` is dropped: there is no pytest-cov here, so the template's flag would be
# an unrecognized argument. Never restore it from a mechanical template sync.
policy:

@maybebyte
maybebyte merged commit c64bfa4 into main Jul 30, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant