Skip to content

fix(audit): evaluate only the audited framework's controls (#442) - #560

Merged
mlieberman85 merged 1 commit into
darnitdevorg:mainfrom
mlieberman85:fix-442-audit-control-scope
Oct 6, 2026
Merged

mlieberman85 merged 1 commit into
darnitdevorg:mainfrom
mlieberman85:fix-442-audit-control-scope

Conversation

@mlieberman85

Copy link
Copy Markdown
Contributor

Summary

Controls live in a process-wide registry keyed by control id. Each audit registered its framework's controls and then evaluated everything the registry held, so a long-lived process (the MCP server) mixed frameworks. Reproduced on main:

Audit, in order Controls before Controls after
reproducibility (fresh process) 5 5
openssf-baseline 71 (5 of them RE-) 66 (65 OSPS + the STAGE1 reference control, #466)
reproducibility again 71 (65 of them OSPS) 5

With this change, run_sieve_audit loads an audit's controls from its own framework definition (including composed controls) plus operator custom controls. It falls back to the registry only when no framework definition resolves. Registration into the registry is unchanged for its other consumers.

docs/architecture/framework-design.md section 6.3.1 states the rule and adds a scenario.

Closes #442

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Testing

  • Tests pass locally (uv run pytest tests/ -v): 5112 passed, 26 skipped
  • Added tests for new functionality (if applicable): tests/darnit/tools/test_audit_control_scope.py. It runs reproducibility, then baseline, then reproducibility in one process, offline through RecordedGhApi, and failed before the fix.
  • Linting passes (uv run ruff check .)

Also passing: the integration tests with GitHub Actions environment variables set (334 passed) and the false-PASS corpus gates.

AI assistance

  • No AI assistance was used
  • AI assistance was used

Claude (Claude Code, claude-opus-5-5) reproduced the issue and made the fix and test. This description was also drafted with Claude. The commit carries an Assisted-by: Claude:claude-opus-5-5 trailer.

Additional Notes

🤖 Generated with Claude Code

…org#442)

Controls live in a process-wide registry keyed by id, so a long-lived
process audited every control an earlier audit had registered: a
reproducibility audit after an openssf-baseline audit returned 71 controls
instead of 5. An audit now loads its controls from its own framework
definition (plus operator custom controls), and uses the registry only
when no framework definition resolves.

Assisted-by: Claude:claude-opus-5-5
Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
@mlieberman85
mlieberman85 merged commit 599041c into darnitdevorg:main Oct 6, 2026
8 checks passed
@mlieberman85
mlieberman85 deleted the fix-442-audit-control-scope branch October 6, 2026 17:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Controls leak across frameworks within a process: reproducibility audit returns 71 controls instead of 5 after a baseline audit

1 participant