Skip to content

fix(contract): surface run warnings in the SARIF projection - #285

Open
rohanpoudel2 wants to merge 7 commits into
openai:mainfrom
rohanpoudel2:fix/sarif-run-warnings
Open

fix(contract): surface run warnings in the SARIF projection#285
rohanpoudel2 wants to merge 7 commits into
openai:mainfrom
rohanpoudel2:fix/sarif-run-warnings

Conversation

@rohanpoudel2

@rohanpoudel2 rohanpoudel2 commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

Summary

Fixes #251.

A scan whose target drifted mid-run records a warning, but that warning never reached the SARIF
projection, so a SARIF consumer could not tell the results described a stale tree. Drift leaves
coverage.completeness at complete — the scan really did review everything it set out to review,
the tree simply moved underneath it — and notifications were only ever built from coverage.deferred
and only when completeness was not complete. Run-level warnings therefore had no route into SARIF
regardless of completeness.

This projects durable run warnings into runs[].invocations[].toolExecutionNotifications, including
target-drift warnings on otherwise complete scans.

Changes

  • Accept run warnings when building and writing SARIF projections, and report them whatever the
    coverage completeness. Deferred coverage already contributes its reason verbatim, so the same text
    is not notified twice.
  • Pass persisted warnings through scan completion and regenerated exports.
  • Close two paths that rewrote exports/results.sarif through finalize_scan without the warnings
    the scan had recorded, so a regenerated projection silently dropped the notifications the first
    completion wrote: re-finalizing an already-complete scan, and backfilling legacy finding details.
    Both now read the persisted warnings back, matching what regenerated exports already did.
  • Validate the SARIF invocation and notification structure.
  • Document that execution success comes from coverage while notifications can come from either
    coverage or run warnings.

Reconciliation with #601

#601 landed in this area after this branch was opened and rewrote the same block. Both contracts are
preserved:

  • Execution success still comes from coverage completeness, and the invocation is still always
    emitted, including the synthesized reason for incomplete coverage with no deferred entries.
  • Run warnings join the notification list independently of completeness.
  • toolExecutionNotifications is attached only when there is at least one notification, so a
    complete, warning-free scan keeps the bare successful invocation fix: Set SARIF success from coverage #601 produces.
  • Invocation validation accepts an unsuccessful execution, which is legitimate for incomplete
    coverage, and an absent notification list, since a complete scan with no warnings notifies nothing.

No public CLI surface changes: no new commands, flags, accepted values, environment variables, or
defaults.

Testing

From sdk/typescript/, with main merged in at f222faf:

  • bun test --timeout 30000 tests-ts/scan-recovery.test.ts and tests-ts/cli-export.test.ts — 68
    passed, 0 failed, 547 assertions across the two files.
  • pnpm run test — 1,577 passed, 29 skipped, 0 failed, 21,641 assertions across 1,606 tests in
    87 files.
  • pnpm run types — passed.
  • pnpm run format — passed, all matched files use Prettier style.
  • pnpm pack — passed.
  • git diff --check — passed.

Coverage added for combined incomplete coverage and run warnings without duplicate notifications, for
warnings surviving a repeated finalization of an already-complete scan, and for warnings surviving
legacy finding-detail backfill. The last two delete the generated SARIF before re-triggering the path,
so they fail without the corresponding fix. The existing complete-with-drift and regenerated-export
cases are retained, and the clean-complete assertion now requires the notification key to be absent.

Tests use synthetic fixtures and temporary directories. No live scan, network, or credentials were
required. Post-push GitHub Actions checks remain pending.

Two local check failures are unrelated to this branch and reproduce identically on unmodified main
in the same environment, which was verified against a clean checkout: audit:prod reports one
pre-existing high advisory, and check:package fails in its installed-package smoke because the
packed tarball resolves @openai/codex at a prerelease version the public registry does not serve.
package.json and pnpm-lock.yaml are byte-identical to main on this branch, so no dependency
changed here.

Risk and rollout

SARIF gains warning notifications only when durable run warnings exist, so warning-free projections
stay byte-compatible. Findings, seals, fingerprints, and coverage completeness are unchanged, and the
execution-success semantics introduced by #601 are unchanged. Consumers that already read
toolExecutionNotifications for deferred coverage will now also see run warnings there, which is the
intent. The two regeneration paths become consistent with initial completion rather than emitting a
projection that silently omits recorded warnings. No migration or configuration change is required.

Public disclosure review

  • No customer, partner, prospect, or user identities, data, or identifying details are included.
  • No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
  • I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

A scan whose target drifted mid-run records a warning, but the SARIF
projection only ever built toolExecutionNotifications from deferred
coverage rows, and only when completeness was not complete. A drifted
target leaves completeness at complete, so no invocations block was
emitted and the warning had no route into SARIF at all.

Run warnings now reach the projection independently of completeness,
deduplicated against the deferred reasons that already notify verbatim.
@github-actions github-actions Bot added the bug Something isn't working label Aug 5, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

1 similar comment
@chatgpt-codex-connector

Copy link
Copy Markdown

# Conflicts:
#	sdk/typescript/_bundled_plugin/scripts/finalize_scan_contract.py
#	sdk/typescript/_bundled_plugin/scripts/workbench_db.py
# Conflicts:
#	sdk/typescript/_bundled_plugin/scripts/finalize_scan_contract.py
Re-finalizing an already-complete scan and backfilling legacy finding
details both rewrite exports/results.sarif through finalize_scan without
the warnings the scan recorded, so a regenerated projection dropped the
notifications the first completion had written. Both paths now read the
persisted warnings back, matching what regenerated exports already do.

Reconcile the projection with the coverage-derived execution status added
in openai#601. Success still comes from coverage completeness and the
invocation is still always emitted, while run warnings join the
notification list whatever the completeness. Invocation validation
accepts an unsuccessful execution and an absent notification list, since
a complete scan with no warnings notifies nothing.
@rohanpoudel2
rohanpoudel2 force-pushed the fix/sarif-run-warnings branch 2 times, most recently from 999cffa to 3778da4 Compare August 22, 2026 00:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Run warnings never reach the SARIF projection, so drift is invisible to SARIF consumers

1 participant