Skip to content

fix(core): compute repo compliance via calculate_compliance in audit_org - #537

Merged
mlieberman85 merged 2 commits into
darnitdevorg:mainfrom
i-am-paradox:fix/515-audit-org-calculate-compliance
Oct 7, 2026
Merged

mlieberman85 merged 2 commits into
darnitdevorg:mainfrom
i-am-paradox:fix/515-audit-org-calculate-compliance

Conversation

@i-am-paradox

Copy link
Copy Markdown
Contributor

Summary

In packages/darnit/src/darnit/tools/audit_org.py, repository compliance was determined by a local status-count rule (fail_count == 0 and warn_count == 0 and (pass_count + na_count == total)). As noted in #515, this rule drifts from the shared calculate_compliance function used by single-repo audits, harnesses, and attestations.

This updates aggregate_org_results to call calculate_compliance(check_results, level) when check results are present, determining whether each repository is compliant at the audited level, and recording per-level compliance in the repository summary. A fallback remains in place for summary-only data to preserve backwards compatibility with existing test mocks.

Type of Change

  • Bug fix (non-breaking change fixing an issue)
  • New feature (non-breaking change adding functionality)
  • Breaking change (fix or feature causing existing functionality to change)
  • Documentation update
  • Refactoring (no functional changes)

Framework Changes Checklist

If this PR modifies the darnit framework (packages/darnit/):

  • Updated framework spec (docs/architecture/framework-design.md) if behavior changed
  • Ran uv run python scripts/validate_sync.py --verbose and it passes

Control/TOML Changes Checklist

If this PR modifies controls or TOML configuration:

  • Control metadata defined in TOML (not Python code)
  • SARIF fields (description, severity, help_url) included where appropriate
  • Ran validation to confirm TOML schema compliance

Testing

  • Tests pass locally (uv run pytest tests/ -v)
  • Added tests for new functionality (if applicable)
  • Linting passes (uv run ruff check .)

Added unit tests in tests/darnit/test_audit_org.py:

  • test_results_with_warn_error_and_pending_is_non_compliant: verifies that WARN, ERROR, and PENDING controls result in non-compliant status.
  • test_results_with_only_pass_and_na_is_compliant: verifies that repositories with only PASS and N/A controls result in compliant status.
  • test_multi_level_compliance_calculation: verifies level-specific compliance evaluation.

All 23 tests in tests/darnit/test_audit_org.py pass cleanly. uv run python scripts/validate_sync.py --verbose passes.

AI assistance

  • No AI assistance was used
  • AI assistance was used

Assisted by Antigravity (Gemini 2.5 Pro) for test authoring and compliance logic alignment. All changes reviewed, verified, and tested against local pytest suite and sync validator.

Additional Notes

Fixes #515

In aggregate_org_results, repository compliance was computed using a local
status-count heuristic (fail_count == 0 and warn_count == 0 and pass + na == total),
which drifted from the shared calculate_compliance standard used across single-repo
audits, harnesses, and attestations.

This updates aggregate_org_results to use calculate_compliance from darnit.tools.audit
when check results are available, determining compliance at the audited level, and
retaining fallback for summary-only data.

Fixes darnitdevorg#515

Assisted-by: Antigravity:eni-gemini-2.5-pro
Signed-off-by: Paradox <159635503+i-am-paradox@users.noreply.github.com>

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for taking this on. There's a problem: line 283 only checks compliance[level], but calculate_compliance scores each level on its own. So L1 FAIL with L2 PASS, audited at level 2, now reports COMPLIANT, where main correctly reports non-compliant. Please:

  1. Require every level from 1 up to the audited level to pass, the same way the attestation's level_achieved works, and add a test for this case.
  2. Drop the fallback (lines 289-299) that keeps the old local rule. Real results always include results, so update the test mocks instead.
  3. Use PENDING instead of PENDING_LLM in the tests; it was renamed in 041.

Drafted with Claude Code; reviewed and posted by me.

…test mocks

Address review feedback on darnitdevorg#537:
- Require all levels from 1 up to audited level to pass for repository compliance, matching attestation level_achieved behavior.
- Drop legacy summary-only fallback in aggregate_org_results and update test mocks with check results.
- Use PENDING instead of PENDING_LLM in test cases.
- Add test verifying that level 2 audit requires level 1 to pass.

Signed-off-by: Paradox <159635503+i-am-paradox@users.noreply.github.com>
@i-am-paradox

Copy link
Copy Markdown
Contributor Author

Updated in 7a73c29:

  • aggregate_org_results now requires all levels from 1 up to the audited level to pass (all(compliance.get(lvl, False) for lvl in range(1, level + 1))), matching the attestation level_achieved progression. Added test_level_2_audit_requires_level_1_to_pass covering L1 FAIL with L2 PASS.
  • Dropped the legacy summary fallback in aggregate_org_results and updated the test mocks in TestAggregateOrgResults to include check results.
  • Switched the pending test fixture status to PENDING.

@mlieberman85 mlieberman85 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the quick turnaround, this looks good.

@mlieberman85
mlieberman85 merged commit 8d84b21 into darnitdevorg:main Oct 7, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

audit_org computes repository compliance itself instead of calling calculate_compliance

2 participants