Repository navigation
Library pipeline hardening: rollback restores generation details, extraction provenance, real-reader hostile tests #3750
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
7012710
Record OCR engine and chunker versions per document and add a read-on…
claude 6e32a5f
Run the hostile-document matrix through the real PDF reader and queue…
claude 21acf50
Restore generation metadata and index quality on index rollback (#3732)
claude 8fb48e8
Report target_snapshot_missing in the reindex rollback dry run
claude a5a4c42
test(sql): seed generation snapshots in the rollback behaviour test (…
claude d465bbd
fix(provenance): address review on #3750
claude File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
12 changes: 12 additions & 0 deletions
12
docs/outstanding-issues-inbox/3a784e6b-ff5e-42be-897f-2492a3754aea.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "version": 2, | ||
| "id": "3a784e6b-ff5e-42be-897f-2492a3754aea", | ||
| "createdOn": "2026-10-21", | ||
| "action": "update", | ||
| "payload": { | ||
| "id": "#7D8WZ2", | ||
| "detail": "UPDATE 2026-10-21: tests/hostile-document-real-extractor.test.ts now runs generated hostile PDFs (password-protected, truncated, not-a-PDF, blank page, prompt-injected with forged evidence fences and hidden white text) through upload admission, the REAL Python reader, the real chunker, the evidence ranker and the fenced answer source block, offline (runs in the CI coverage job, which installs PyMuPDF). Still open: the live leg (Storage upload, worker ingestion with embeddings, vector retrieval, generated answer) is not run anywhere. Next action: run it on staging once the staging rebuild (#3113) is done and the owner lifts the re-index freeze, with the owner's spend approval for embeddings and answers. PRIOR DETAIL: Seven-class helper coverage exists in tests/hostile-document-ingestion-matrix.test.ts, but it uses an extraction mock and fabricated search results, so no uploaded hostile document is checked against real ingestion, retrieval and answer paths. Close only when each class runs end to end.", | ||
| "baseRowFingerprint": "349c4edbdc550bbad485d7460ac918adc67543c76f5977feaf68cc2cd38c49a7", | ||
| "targetUlid": "01M51JV9GC7D8WZ29KQKRSF5HP" | ||
| } | ||
| } |
12 changes: 12 additions & 0 deletions
12
docs/outstanding-issues-inbox/b8553cec-2ce1-42f8-a5dc-694e031e996e.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "version": 2, | ||
| "id": "b8553cec-2ce1-42f8-a5dc-694e031e996e", | ||
| "createdOn": "2026-10-21", | ||
| "action": "done", | ||
| "payload": { | ||
| "id": "#EG02B7", | ||
| "outcome": "Already fixed 2026-10-19 by PR #3677: migration 20261019155200_prune_superseded_index_generations_on_commit.sql adds a trigger on the generation pointer that prunes superseded generations at every commit and queues their orphaned image objects in storage_cleanup_jobs for the janitor. Verified on main 81ba5aefa by the Library pipeline hardening thread; no further migration needed.", | ||
| "baseRowFingerprint": "89a7d8c1ffc00de7c23fa1a527e63e4b943088799914ec02ce10d43dfeb2a88a", | ||
| "targetUlid": "01M51JV9J6EG02B7D44998GPSQ" | ||
| } | ||
| } |
12 changes: 12 additions & 0 deletions
12
docs/outstanding-issues-inbox/f7ff9913-697d-47b8-b3df-ef6fac9ae528.json
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,12 @@ | ||
| { | ||
| "version": 2, | ||
| "id": "f7ff9913-697d-47b8-b3df-ef6fac9ae528", | ||
| "createdOn": "2026-10-21", | ||
| "action": "update", | ||
| "payload": { | ||
| "id": "#2F8K1V", | ||
| "detail": "UPDATE 2026-10-21: documents.metadata.extraction_provenance now also records the chunker versions of the committed chunks and the OCR engine (Tesseract) versions and count of OCR'd pages (worker/extraction-provenance.ts), on top of the reader version, PyMuPDF version and table strategy recorded since PR #3632. npm run extraction:provenance (read-only; needs owner confirmation before a live run) lists documents by reader, chunker or OCR version, and npm run reindex:rollback -- --chunker-version already scopes a rollback. No migration needed. Remaining: documents indexed before these keys existed report 'not recorded' until their next index, and re-extraction by version is still an operator step. Close when the owner accepts that. PRIOR DETAIL: Verified offline: only a coarse indexing_version ('rag-deep-memory-v1') is stored; no extractor/OCR/table-strategy/chunker provenance exists in schema or worker.", | ||
| "baseRowFingerprint": "e95fa094042f40eebb3ec86a2355667d537b124f4e88aee1d638f6ad06e2b3ea", | ||
| "targetUlid": "01M3CHC29Y2F8K1VF3T2SYJQCG" | ||
| } | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| /** | ||
| * Read-only report: which reader, OCR engine and chunker produced each document's committed | ||
| * index generation (audit F04, #2F8K1V). Use it to list the documents a parser defect touched, | ||
| * then pass those ids to a reviewed re-extraction or `npm run reindex:rollback`. | ||
| * | ||
| * npm run extraction:provenance | ||
| * npm run extraction:provenance -- --extractor-version 2026-10-21.merged-cell-provenance-v11 --list | ||
| * npm run extraction:provenance -- --chunker-version 1.3.1-page --list | ||
| * npm run extraction:provenance -- --ocr-engine-version 5.3.4 --list | ||
| * | ||
| * Selects only; it never writes, queues a job, chunks or re-indexes. It still reads the live | ||
| * database, so it needs the owner's confirmation before it is run against production. | ||
| * Documents indexed before the record existed report "not recorded"; list them with | ||
| * `--extractor-version "not recorded" --list`. | ||
| */ | ||
| import { loadEnvConfig } from "@next/env"; | ||
| import { | ||
| buildProvenanceReport, | ||
| parseProvenanceArgs, | ||
| type ProvenanceDocumentRow, | ||
| } from "@/lib/extraction-provenance-report"; | ||
| import { assertSupabaseHealthy, probeSupabaseHealth } from "@/lib/supabase/health"; | ||
|
|
||
| loadEnvConfig(process.cwd()); | ||
|
|
||
| const PAGE = 1000; | ||
|
|
||
| async function main() { | ||
| const args = parseProvenanceArgs(process.argv.slice(2)); | ||
| const [{ env, requireServerEnv }, { createAdminClient }] = await Promise.all([ | ||
| import("@/lib/env"), | ||
| import("@/lib/supabase/admin"), | ||
| ]); | ||
| requireServerEnv(); | ||
| const supabase = createAdminClient(); | ||
|
|
||
| console.log("=== Extraction provenance report (read-only) ==="); | ||
| console.log(`Supabase project: ${env.SUPABASE_PROJECT_NAME ?? "unknown"} (${env.SUPABASE_PROJECT_REF ?? "unknown"})`); | ||
| assertSupabaseHealthy(await probeSupabaseHealth(supabase), "Extraction provenance report"); | ||
|
|
||
| const rows: ProvenanceDocumentRow[] = []; | ||
| // Keyset paging, so documents added or removed during the scan cannot shift a page. | ||
| for (let lastId: string | null = null; ;) { | ||
| let query = supabase.from("documents").select("id,title,metadata").order("id").limit(PAGE); | ||
| if (lastId) query = query.gt("id", lastId); | ||
| const { data, error } = await query; | ||
| if (error) throw new Error(error.message); | ||
| const page = (data ?? []) as ProvenanceDocumentRow[]; | ||
| rows.push(...page); | ||
| if (page.length < PAGE) break; | ||
| lastId = page[page.length - 1].id; | ||
| } | ||
|
|
||
| const report = buildProvenanceReport(rows, args); | ||
| console.log(`Documents scanned: ${report.scanned}; matching filter: ${report.matched.length}`); | ||
| for (const [label, counts] of [ | ||
| ["Extractor version", report.byExtractorVersion], | ||
| ["Chunker version", report.byChunkerVersion], | ||
| ["OCR engine version", report.byOcrEngineVersion], | ||
| ] as const) { | ||
| console.log(`\n${label}:`); | ||
| const entries = Object.entries(counts).sort(([a], [b]) => a.localeCompare(b)); | ||
| if (entries.length === 0) console.log(" (none)"); | ||
| for (const [version, count] of entries) console.log(` ${version}: ${count}`); | ||
| } | ||
| if (args.listDocuments) { | ||
| console.log("\nDocuments:"); | ||
| for (const document of report.matched) console.log(` ${document.id} ${document.title ?? "(untitled)"}`); | ||
| } | ||
| } | ||
|
|
||
| main().catch((error) => { | ||
| console.error(error instanceof Error ? error.message : String(error)); | ||
| process.exitCode = 1; | ||
| }); |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.