feat: match repeated findings across scans

[mldangelo-oai]

Summary

Repeated scans can describe the same security bug in different ways. The old matcher sent every complete finding to Codex at once, which could hit the limit of 1,048,576 characters per message as the scan history grew.

This change gives Codex short summaries of known issues and lets it ask for more evidence. We keep the limit, reuse confirmed matches, and leave similar findings separate when they need different fixes. Existing Codex sign-in works; there is no new service or API key to set up. The existing scans compare and scans match commands remain the way to use it.

How matching works

The matcher uses ordinary code to remember confirmed matches and asks Codex when it needs a new judgment. A findingId is a stable finding identity; an occurrenceId identifies its report in one scan.

Consider these made-up IDs:

Earlier scan 1: findingId=f1, occurrenceId=o1
Earlier scan 2: findingId=f2, occurrenceId=o2
Saved fact:     f1 and f2 describe the same bug
Later scan:     findingId=f3, occurrenceId=o3

The old matcher sent the full reports o1, o2, and o3 together in one Codex request. Stable-ID shortcuts and saved comparisons already existed, but a new judgment still had to fit into that one message.

The new matcher handles it in four steps:

1. groupFindings() joins f1 and f2 using a disjoint-set, also called union-find. This is a small in-memory data structure for keeping track of which IDs are already known to belong together.

2. findingCatalogue() chooses the later report, o2, to represent that group. Its Map retains both a compact record and the original reports. The entry has this shape:

   new Map([
     [
       "o2",
       {
         card: compactFinding(report2),
         occurrences: [report1, report2],
       },
     ],
   ]);

   compactFinding() selects fields such as cause, fix, location, and attack path. The catalogue also keeps differences from older descriptions. These records are built by TypeScript, not by another LLM call.

3. Codex receives the compact records for o2 and o3. It can return a structured request for their full evidence, or say that they describe the same bug. The host already has the full records in memory. It selects them by ID and sends them in pages that fit the existing message limit.

4. If Codex confirms that o3 matches o2, the matcher checks the result and expands o2 back to [o1, o2]. The CLI saves the confirmed links o1 -> o3 and o2 -> o3 in the existing SQLite workbench. Later comparisons can reuse those links. Direct SDK callers receive the expanded result and decide where to save it.

If every match is already known, the matcher skips Codex. Possible matches stay uncertain, and related findings do not join a confirmed group. The original reports remain unchanged.

The building blocks are a disjoint-set, an in-memory Map, a structured JSON conversation, and the existing SQLite tables. There is no embedding search or LLM sorting step. This changes how much text the model sees up front. Full records still occupy host memory, and the model may still need to consider many pairs.

Changes

• Build summaries from existing finding IDs and confirmed matches. Keep the affected code, cause, suggested fix, and attack path, plus any differences in older descriptions. Skip the model call when all matches are already known.
• Let Codex request more summaries or full evidence for selected findings. Prepare each selection once, split long text into pages, and avoid sending the same record twice. A proposed duplicate cannot be saved while its required evidence is missing or incomplete. Other unfinished evidence does not block the result.
• Parse each model response once, then check IDs and conflicting decisions as the matcher expands and joins confirmed groups. Check which confirmed group a finding belongs to instead of building every possible pair just to validate it.
• Keep duplicates, possible duplicates, and related findings separate. Related findings are not merged or closed. Older comparisons reflect later confirmed matches without losing their original decisions or uncertainty in other scan pairs. Only mark a missing issue resolved if the later scan covered all its earlier locations.
• Match scans from oldest to newest, reusing links known at that point even when the original scan files are missing. Prepare and save each scan-pair result one at a time. Recomputing a pair ignores its old decision. scans match --all --force rebuilds the history without old model decisions. Handle reports that split one issue into several findings or combine several into one.
• Expose one SDK matching function, matchScanFindings, with input and result types, progress, and cancellation. The internal codex and allowHistoricalUncertainty options are omitted from the published types. Keep per-call environment support so callers can select credentials or a state directory without changing the whole process environment. Automatic matching after a scan uses the shared matcher's result directly. Errors in optional progress callbacks do not stop matching. Ctrl-C keeps saved comparisons.
• Send large results through standard input when the plugin supports it. Older custom plugins still save confirmed and uncertain matches in their original format. Add two database indexes, read saved links in batches that fit SQLite's parameter limit, load related findings only when shown, and follow existing finding IDs instead of looking up repository paths again. There is no new dependency or search database.
• Run the MCP contract tests with Node, as the shipped plugin does. Both tests use one async helper that reports launch errors. This is a test-only follow-up to a pre-existing Windows CI failure; it does not change the CLI.

Why this approach

A model can say A matches B and B matches C, then disagree about A and C. An O(n log n) sort can limit comparisons, but it cannot guarantee that every duplicate ends up next to its match. We group confirmed matches in code and let Codex consider the remaining candidates together. --all starts at most one conversation per later scan that needs a new decision. Follow-up requests must advance through the evidence or ask for a finding not requested before. They cannot keep requesting different combinations of the same records.

Research on comparing candidates together and the cost of model calls informed this choice. We also tried word-based filters and comparing nearby items after sorting. Those experiments used model-generated answers that people had not checked, so they do not prove accuracy. Embeddings, which let us search for similar text, remain an option if tests with human-checked answers show they are needed.

Testing

How to test in QA

Start with the repeatable tests. From the repository root, run:

cd sdk/typescript
pnpm install --frozen-lockfile
bun test --timeout 30000 tests-ts/finding-catalogue.test.ts tests-ts/scan-comparison.test.ts tests-ts/cli-workbench.test.ts tests-ts/scan-matching-e2e.test.ts tests-ts/workbench-scan-history.test.ts
bun test --timeout 30000 tests-ts/runtime.test.ts --test-name-pattern 'saves comparisons with a'

The tests use made-up findings, including four temporary scans and six saved comparisons. They need no credentials or network. They check repeated issues, related-but-separate bugs, saved matches, --force, cancellation, missing scan files, corrected older comparisons, and older custom plugins. They also check that missing evidence is read before a duplicate is confirmed and that progress errors cannot stop matching. Original scan files must stay unchanged, and invalid responses must not replace a saved comparison.

To check the installed SDK, run these commands from the same SDK directory:

pnpm pack --pack-destination ../../dist
pnpm run check:package ../../dist/openai-codex-security-0.1.14.tgz

This compiles a TypeScript consumer of the installed package. Normal matching options must work, while codex and allowHistoricalUncertainty must be rejected. Installing the package may download dependencies, but this check does not scan a repository or call a live model.

For a live check, start from the same SDK directory. Use a disposable repository you are authorized to scan and replace the example paths and IDs below. This uses the branch's CLI and a separate QA history. Unlike the repeatable tests, it makes real model calls.

pnpm run build
export CODEX_SECURITY_QA_CLI="$PWD/bin/codex-security.mjs"
export CODEX_SECURITY_STATE_DIR=/path/outside-the-test-repo/qa-state
cd /path/to/authorized-test-repo

node "$CODEX_SECURITY_QA_CLI" scan . --auth chatgpt
# Make a small test change, then scan again.
node "$CODEX_SECURITY_QA_CLI" scan . --auth chatgpt
node "$CODEX_SECURITY_QA_CLI" scans list
node "$CODEX_SECURITY_QA_CLI" scans match BEFORE_SCAN_ID AFTER_SCAN_ID
node "$CODEX_SECURITY_QA_CLI" scans compare BEFORE_SCAN_ID AFTER_SCAN_ID

If both scans report the same bug, expect one persisting issue. A bug that needs a different fix should stay separate, even if marked related. Run matching again to check that it reuses saved results. Try scans match --all --force, Ctrl-C during a longer run, and --format json. Progress should not corrupt the JSON output.

Checks already run

• The QA commands above: 102 matching tests passed with 3 platform-specific skips, and both older-plugin tests passed.
• At 4d9c1105, the full suite passed with 1,492 tests, 23 platform-specific skips, and no failures using both seed 12345 and random seed 2864254137. Earlier QA against the actual plugin from the PR's base commit saved and reused confirmed and uncertain matches.
• Type checks, formatting, Ruff, and git diff --check passed at 4d9c1105. pnpm pack, the installed-package check, and the compiled-SDK smoke test also passed. The compare and match help, command schemas, and all 31 generated TypeScript declaration files are byte-for-byte identical to 26a30474.
• The two MCP contract tests passed locally. The full runtime test file also passed with both earlier Windows CI seeds, 3057713696 and 2697452318: 125 passed and 9 Windows-only tests skipped on macOS in each run. The normal Windows CI jobs test this under Node 22 and Node 24.
• For the SDK cleanup in c822db2b, the new public-API test fails against the earlier PR build (de40b69c) and passes against the rebuilt package. The compare and match help, command schemas, and compiled matcher JavaScript are byte-for-byte identical to de40b69c.
• At de40b69c, live ChatGPT-auth tests passed a direct comparison and one that requested more evidence. Both kept an independently fixable bug separate. Live model tests remain outside the repeatable CI suite.
• In an in-memory test with 1,000 scans, loading saved links took one database query instead of 1,000 and returned the same 999 links in the same order. A regression test also checks empty selections, excluded scans, and a low SQLite parameter limit. A made-up history with 10,000 related pairs needs three database queries to display one finding. A database-upgrade test with 19,900 comparisons checks both lookup directions, unchanged rows, and consistency. Coverage includes Python 3.10, SQLite's parameter limit, and both the old command-line JSON input and new UTF-8 standard input.
• Tests also cover the model's response format, unknown IDs, repeated or overlapping requests, resuming partially sent evidence, premature confirmations, long Unicode text, and JSON output. A strict Node test checks that a rejected progress callback cannot terminate the process. Cost-limit tests cover one successful call, an oversized request that makes no call, and a request for more evidence that preserves the completed scan without saving partial matches.
• A synthetic group with 1,000 findings on each side now creates zero temporary pair keys during validation, down from one million. In a Python validation-only probe, peak traced memory fell from about 98 MB to 0.8 MB. A 2.2-million-character evidence selection still takes three pages, but is prepared once instead of three times. A separate 19 MiB evidence test reduced the extra memory kept by JavaScript from about 162 MB to 22 MB. These measure local bookkeeping, not model speed or accuracy.
• In a size-only test, 10 scans with 100 recurring issues became 100 summaries. The input fell from 2,778,401 to 33,931 characters, about 82 times smaller. That measures this test's input size, not accuracy or model cost.
• Independent local review of de40b69c: three fresh Codex reviews and a separate verification found no actionable issues. The SDK cleanup in c822db2b passed two fresh Codex reviews and a separate verification. The test-only follow-up at 26a30474 passed three fresh reviews and a separate verification. The cleanup at 4d9c1105 also passed three fresh reviews and a separate verification. GitHub CI and review remain separate checks on the pushed commit.

Risk and rollout

Existing comparisons and the old command-line JSON input remain readable. Older custom plugins can still save confirmed and uncertain matches, but need an upgrade to save related-finding links or results too large for command-line arguments. The database upgrade adds two indexes without rewriting saved comparisons. On a large history, building them may hold the normal database write lock for a while. Original scan files and triage decisions stay unchanged. This does not change which credentials Codex chooses or require a SQLite extension.

Grouping different bugs by mistake can affect later matches. The matcher asks for the same broken check and fix, keeps uncertain answers, and treats related findings as suggestions rather than a complete map. Review model judgments before making consequential triage decisions. Use scans match --all --force to reconsider old matches.

Splitting messages does not give the model unlimited room or guarantee less work than comparing every pair. Large histories and long evidence can still need more calls. An unfinished evidence request keeps one compact copy of its text in memory; that copy is released when the request finishes. Measure wrong matches, missed matches, tokens, calls, reuse of earlier results, and time before adding a more complicated search system. Saved scan-pair comparisons still grow roughly fourfold when the number of scans doubles; the indexes keep ordinary reads from searching every pair.

For --max-cost and the SDK's maxCostUsd, the scan's cost total still excludes one permitted automatic matching call. If more calls are needed, the completed scan stays saved, no partial comparison is stored, and a warning points to scans match --all. Running that command is separate work; the allowance does not increase.

Public disclosure review

• No customer, partner, prospect, or user identities, data, or identifying details are included.
• No credentials, personal data, private source, scan findings, or nonpublic links or tickets are included.
• I reviewed the branch name, title, description, commits, changes, comments, logs, screenshots, attachments, and links for public disclosure.

[mldangelo-oai]

@codex review

Please review the current head: 33b758489f2734e7917630c9548e320a14de2ab0.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: 33b758489f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

The title and description now use plain English and include QA steps. The code is unchanged at 33b758489f2734e7917630c9548e320a14de2ab0.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Chef's kiss.

Reviewed commit: 33b758489f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

Please review the current head: de40b69c16fc082cb05dd591ca8974dbc5c2e9c3.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Hooray!

Reviewed commit: de40b69c16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

The description now includes a concrete walkthrough of finding IDs, grouping, evidence requests, and saved links. Please check it against the implementation at de40b69c16fc082cb05dd591ca8974dbc5c2e9c3. No code changed.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Another round soon, please!

Reviewed commit: de40b69c16

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

Please review the current head: c822db2bba7b3157ff95eee52b7f4015529739fb.

This follow-up narrows the published SDK options and simplifies the user documentation without changing CLI behavior. The PR description includes the installed-package type check and updated QA results.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Already looking forward to the next diff.

Reviewed commit: c822db2bba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

Please review the current head: 26a304747184f58f855e651ac5fd63461216a539.

This test-only follow-up runs the MCP contract checks with Node, as the shipped plugin does, and preserves subprocess errors. The PR description records the two failed CI seeds and the fresh local QA results. No CLI commands or runtime behavior changed.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. Can't wait for the next one!

Reviewed commit: 26a3047471

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

@codex review

Please review the current head: 4d9c1105336cba115247604f62f035387ac107a1.

This cleanup removes repeated response parsing and temporary comparison copies, shares the catalogue-page path, and uses the existing SQLite batching helper for saved links. CLI help, command schemas, and published TypeScript declarations are unchanged. The PR description includes the new regression tests, query-count experiment, and full QA results.

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. 🎉

Reviewed commit: 4d9c110533

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[mldangelo-oai]

Split this into smaller PRs:

• test: run MCP contract checks with Node #573 runs the MCP contract tests with Node. It can merge independently.
• feat: preserve cross-scan finding relationships #574 adds the scan-history storage and compatibility changes.
• feat: match repeated findings across scan history #575 adds the matcher and keeps the existing CLI commands. It is based on feat: preserve cross-scan finding relationships #574.

The recommended merge order is #573, #574, then #575. After #574 is squash-merged, update #575 from main, change its base to main, and rerun checks and review.

The replacement PRs retain the feature and its correctness fixes. The storage tests also share one Python-launch helper. Each PR has passing CI and a clean Codex review on its current head. Closing this PR as superseded; its branch and review history are preserved.