Repository navigation
fix(audit): evaluate only the audited framework's controls (#442) - #560
Merged
mlieberman85 merged 1 commit intoOct 6, 2026
Merged
Conversation
…org#442) Controls live in a process-wide registry keyed by id, so a long-lived process audited every control an earlier audit had registered: a reproducibility audit after an openssf-baseline audit returned 71 controls instead of 5. An audit now loads its controls from its own framework definition (plus operator custom controls), and uses the registry only when no framework definition resolves. Assisted-by: Claude:claude-opus-5-5 Signed-off-by: Michael Lieberman <mlieberman85@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Controls live in a process-wide registry keyed by control id. Each audit registered its framework's controls and then evaluated everything the registry held, so a long-lived process (the MCP server) mixed frameworks. Reproduced on
main:reproducibility(fresh process)openssf-baselinereproducibilityagainWith this change,
run_sieve_auditloads an audit's controls from its own framework definition (including composed controls) plus operator custom controls. It falls back to the registry only when no framework definition resolves. Registration into the registry is unchanged for its other consumers.docs/architecture/framework-design.mdsection 6.3.1 states the rule and adds a scenario.Closes #442
Type of Change
Framework Changes Checklist
docs/architecture/framework-design.md) if behavior changeduv run python scripts/validate_sync.py --verboseand it passesTesting
uv run pytest tests/ -v): 5112 passed, 26 skippedtests/darnit/tools/test_audit_control_scope.py. It runs reproducibility, then baseline, then reproducibility in one process, offline throughRecordedGhApi, and failed before the fix.uv run ruff check .)Also passing: the integration tests with GitHub Actions environment variables set (334 passed) and the false-PASS corpus gates.
AI assistance
Claude (Claude Code, claude-opus-5-5) reproduced the issue and made the fix and test. This description was also drafted with Claude. The commit carries an
Assisted-by: Claude:claude-opus-5-5trailer.Additional Notes
tests/darnit/sieve/baseline_capture.pystill runs each pair in a subprocess, the Controls leak across frameworks within a process: reproducibility audit returns 71 controls instead of 5 after a baseline audit #442 workaround from feature 037. It can switch to in-process capture in a follow-up.tests/conftest.py) can stay as defense in depth.🤖 Generated with Claude Code