Skip to content

Add a rules registry that makes every enforced and unenforced rule visible - #3884

Draft
anth-volk wants to merge 7 commits into
phase-0-enforcementfrom
rules-registry
Draft

anth-volk wants to merge 7 commits into
phase-0-enforcementfrom
rules-registry

Conversation

@anth-volk

@anth-volk anth-volk commented Oct 12, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes #3885

Summary

Stacked PR. This is the middle of a native GitHub stack: Phase 0 (#3887) → this PR → steps 1.3, 1.5 and 1.6 (#3890). It's based on phase-0-enforcement and was rebased onto it, so the diff shows only this layer.

  • Adds rules/registry.toml: every rule that governs this repository, with its tier (mechanical, metric or judgment), status (enforced, ratcheted, advisory or doc-only), enforcer, and the CI job that runs it. There are 48 rules: 35 enforced, 2 advisory, and 11 doc-only gaps. The doc-only rules are taken from v2-code-organization.md, api-routes.md, alembic-migrations.md, repository-maintenance.md and observability.md. Phase 0 made the CRUD boundary tests discover modules by filename, so api.v2.crud-checks-cover-every-resource is now enforced by the new classification test. Code-owned rule machinery and the agent no-bypass rule from Phase 0 are registered as doc-only until branch rules enforce them.

  • Adds scripts/rules/, a standard-library tool with validate, render and guard commands, registered as the rules-registry quality guard. The guard fails when:

    • an enforcer doesn't exist;
    • a CI job doesn't run or collect the enforcer it claims to (it checks the make test paths);
    • a boundary test, quality guard, ruff ignore, mypy path or PR job isn't registered;
    • a baseline grows relative to the base branch;
    • a proof test isn't collected by any CI job;
    • an exception or review_by date expires;
    • the rendered views are stale.

    The tool is byte-identical with the same directory in policyengine-sim-api (Add a rules registry that makes every enforced and unenforced rule visible policyengine-sim-api#752); see scripts/rules/SYNC.md.

  • Renders docs/generated/rules.md (a summary, a Gaps section, and a per-category table with Mermaid graphs), and a generated Rules block in AGENTS.md.

  • In pr.yml, the quality-guards job now runs the tool's unittest suite and checks out with fetch-depth: 0 so baselines can be compared with the base branch. push.yml also gets a quality-guards job, which does not gate deployment.

  • Adds a proof fixture and test for api.v2.validators-transformations-no-db. The existing check is factored into _has_database_query_dependency so the proof exercises the same code.

  • Documents how to add or change a rule in docs/engineering/skills/rules-registry.md.

Ruff rules are never ratcheted: adopting a ruff family is one standalone PR that fixes every violation.

Testing

  • python -m unittest discover -s scripts/rules/tests -t .: 54 tests pass. The first version, with 50, was run on Python 3.11 to 3.14; the 4 added since cover proof tests that must run in CI.
  • python scripts/run_quality_guards.py: migration-contracts: passed and rules-registry: passed.
  • python -m scripts.rules guard --base-ref origin/master: passes. The base has no registry yet, so the baseline comparison is skipped.
  • uv run --frozen --extra dev pytest tests/unit/v2/test_data_crud_boundaries.py: 41 tests pass after the rebase. That's Phase 0's 39 discovered modules and classification test, plus this PR's proof test.
  • ruff format --check . passes with ruff 0.15.12 (the locked version) and with the latest ruff.
  • The same negative checks were run on the sim copy of the tool and failed as expected: an unregistered test or guard; a stale AGENTS block; a bad invocation; an expired exception; a ruff baseline; and a project missing from the test matrix. An unregistered CI job is covered by the unit tests.

🤖 Generated with Claude Code

@codecov

codecov Bot commented Oct 12, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 87.56%. Comparing base (8cedb09) to head (880260a).

Additional details and impacted files
@@                 Coverage Diff                  @@
##           phase-0-enforcement    #3884   +/-   ##
====================================================
  Coverage                87.56%   87.56%           
====================================================
  Files                      198      198           
  Lines                    12211    12211           
  Branches                  2159     2159           
====================================================
  Hits                     10692    10692           
  Misses                     922      922           
  Partials                   597      597           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

anth-volk and others added 7 commits October 13, 2026 18:59
Standard-library tool (scripts/rules) that loads rules/registry.toml, checks
each rule's enforcer and CI wiring, finds unregistered checks, compares
ratchet baselines with the base branch, and renders the human and agent
views. Byte-identical with policyengine-sim-api; see scripts/rules/SYNC.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Registers rules-registry in the quality guards, runs the tool's unittest
suite in the quality-guards job, fetches full history so baselines can be
compared with the base branch, and adds quality guards to the push workflow.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Registers the existing boundary tests, quality guards, ruff/mypy checks and
CI jobs, plus rules stated only in skills docs as doc-only. Adds a proof
fixture and test for api.v2.validators-transformations-no-db.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guard checked that a proof's fixtures and test exist, but not that any CI
job runs the test, so a proof could silently stop being exercised. A
pytest-enforced rule's proof test must now be collected by the rule's own CI
job; any other rule names the job with proof.workflow, proof.job and
proof.invocation.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Enforce api.v2.crud-checks-cover-every-resource with the new connector
classification test (it was a doc-only gap), record the explicit ruff rule
selection, and register the code-owned rule machinery and the agent
no-bypass rule as doc-only until branch rules enforce them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
api.process.stacked-prs-full-ci-at-bottom: only the lowest unmerged PR
in a stack runs the full workflow, and the CI complete gate needs every
pr.yml job. Enforced by test_pr_workflow_is_gated, with bad and good
workflow fixtures proving the check, and it references the ci-complete
job so the hidden-rule check knows what that job enforces.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch was successfully deployed

1 active (outdated) deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a rules registry that makes every enforced and unenforced rule visible

1 participant