Repository navigation
Reconcile admin ingestion, native PDF extraction and grounded Answer - #3546
Conversation
Capture the scoped ingestion and Answer implementation with portable test candidates and a sanitised plan, progress and verification record. Retain failed and unrun evidence; no new readiness checks or bulk ingestion.
|
Warning Review limit reachedYour organization has reached its usage spending cap. Adjust your spending cap in the billing tab. Next included review available in 49 minutes. View limit details
📝 Walkthrough
Merge Risk: 🟡 Moderate · up to
|
|
Updates to Preview Branch (codex/ingestion-rag-pilot-handover-20261014) ↗︎
Tasks are run on every commit but only new migration files are pushed.
View logs for this Workflow Run ↗︎. |
🏁 CodeAnt Quality Gate ResultsCommit: ✅ Overall Status: PASSEDQuality Gate Details
|
…ilot-handover-20261014
Owner approved in the PR 3546 thread on 16 Oct 2026: "Yes, fix the deep memory leftover facts in the PR 3546 migration." Legacy index-health repair stamps producer-less cards, index units and sections with the committed index generation. The fenced commit's legacy branch only matched a null index generation, so those rows survived a deep-memory refresh next to the new ones. The branch now also matches the committed generation, keeping the local-worker and deep-memory version markers that exclude db_repair rows. Updates the unmerged migration, schema.sql, its seal hash, and the drift manifest (schema hash, and the function fingerprint computed with a local Postgres using the same normalisation CI uses). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
|
Code review by qodo was updated up to the latest commit 6668b99 |
|
Code review by qodo was updated up to the latest commit ad07275 |
The guard that refuses to null another producer's card-to-section link now uses the same legacy predicate as the supersede deletes, including rows health repair stamped with the committed generation. Reseals the migration and refreshes the drift manifest. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
|
Code review by qodo was updated up to the latest commit 54f8559 |
The PR added the delivered-prose check that names required parts an admitted-passage answer does not cover, without tests. Coverage for the RAG answer files fell to 88.9% of functions against a 90% floor. These tests pin the observed behaviour: a missing required part, a missing facet detail, a negated safety constraint, a mismatched population, and the no-gap case. No source change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
|
Code review by qodo was updated up to the latest commit 6afd8bc |
Addresses the Qodo review note that the facet test only checked one missing detail. The fixture now delivers route and frequency, so the gap must name exactly the missing dosing maximum and the absent monitoring. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
Code Review ✅ Approved 8 closed / 8 findings🔴 High risk · Adding fenced ingestion migrations and admin ingestion/Answer delivery could commit incorrect document generations or serve unsupported clinical guidance if provenance checks fail. Reconciles admin ingestion, native PDF extraction and grounded Answer delivery, with fenced persistence and new migrations for the worker. Six earlier findings are resolved, including the missing Answer route module, the migration auto-apply concern, pilot ticket 503s, and the foreign-section guard now covering the committed-generation branch in ✅ 8 closed✅ Bug: Answer routes import a module that does not exist
✅ Quality: Migrations say 'CANDIDATE ONLY / no production application' but auto-apply
✅ Bug: Expired pilot ticket makes every Answer request return 503
✅ Bug: Expired pilot ticket again returns 503 for default All-documents Answers
✅ Bug: Image-free PDF path completes without publication or fidelity gate
...and 3 more closed from earlier reviews VerificationRules No rules evaluated Functional validation Not enabledAuto-approval Approved For more verification details, comment OptionsAuto-apply is off → Gitar will not commit updates to this branch. Comment with these commands to change the behavior for this request:
Important Your trial ends in 5 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more. Was this helpful? React with 👍 / 👎 | Gitar |
|
Code review by qodo was updated up to the latest commit ebbee0c |
Review round, 16 Oct 2026 (after PR #3566)
20261015181500seal entry is added next to this PR's two entries, and no existing entry is edited (owner's named yes). Both.gitleaksignorenotes are kept. The drift manifest points at the mergedschema.sql, and CI's freshness job regenerates the snapshot.fail_or_retry, and the claim query never reclaims an exhausted job, so it stayedprocessingfor good. After: an error carrying a Postgres SQLSTATE rolled back and left the lease held, so it reachesfail_or_retry. Connection (08), operator-intervention (57P), PostgREST gateway and code-less errors still cancel as uncertain.20261015185000now use the same guard as the section delete, so the database repair job's stopgap rows survive a deep-memory commit. This is an unmerged migration of this PR, edited with the owner's named yes. Its seal entry andschema.sqlmatch.rag-deep-memory-v2.6aec8cab):RAG impact: no ranking, selection or ordering change. The deep-memory version stamp aligns managed output with the current
rag-deep-memory-v2thatdeep-memory.tsalready writes. The supersede-delete guard only narrows which legacy rows are removed.Local evidence on 67d98d7:
verify:cheappassed lint and typecheck. The unit suite ran 31,489 tests, with 2 skipped. The 2 failures were a stale source-shape assertion and a shifted doc anchor, both fixed in 67d98d7, and those two files now pass 283/283. The full suite was not re-run after that fix. Migration immutability and migration-role guards pass.Main merged in again (16 Oct 2026, after PR #3571)
PR #3571 changed two files this PR also changes. Both sides are kept.
src/lib/openai.ts: this PR's ingestion egress wrapper stays. Main's per-call embeddingtimeoutMsandmaxRetriesnow pass through it, andIngestionOpenAIOptionscarries both fields.worker/python/extract_pdf_assets.py: main rewrote edge-strip row recovery around text line segments. This PR's rotated-page fix is carried into the new_extract_page_line_segments, which maps text positions into the rotated page space that table rects use. On a 90-degree page the segment matches this PR's earlier rotated block exactly. Table payloads keep bothrepeated_populated_geometryand main'scolumn_ranges.tests/pdf-table-extraction.test.tsimportspymupdfinstead of the deprecatedfitzalias, whose stdout warning broke the test's JSON parse.src/lib/rag/rag.ts29 lines over its no-growth budget. The published-original layout loader this PR added moves unchanged intosrc/lib/rag/rag-published-original-loader.ts, and the caller signal is still read when the loader runs.rag.tsis now 4,360 lines.RAG impact: no retrieval behaviour change. The extractor merge keeps both sides' behaviour, the loader move is a pure code move, and no retrieval, selection or ordering code changed.
Local evidence on the merged head:
verify:cheappasses lint, typecheck and 31,472 unit tests (the 2 earlier failures were thefitzwarning, now fixed). The Python extractor tests pass 8/8 with PyMuPDF 1.28.2, and the table-extraction tests pass 3/3.Import loops fixed (16 Oct 2026)
Before: the full unit suite failed its architecture check because of two runtime import loops this PR introduced. One was between the XLSX preflight and provenance modules, and the other came from
worker/candidate-chunks.tsre-exporting the scoped candidate projection.After: the XLSX schemas and classifiers move unchanged into
src/lib/extractors/xlsx-source-model.ts. Provenance re-exports them, so every existing import still resolves. The candidate-chunks re-export is removed, and its one test imports directly. The clinical doc line anchors that this PR's earlier growth ofrag-source-block.tsandrag.tshad shifted are remapped. The owner gave a named yes for the protected extractor edit (docs/decisions/2026-10-16-import-loops-fixed-in-3546.md), and the ledger item that queued the loops separately is removed.RAG impact: no retrieval behaviour change. The extractor edit is a pure code move with identical exports, and no ranking, selection or ordering code changed.
Codex reference-branch fixes (16 Oct 2026)
Before:
After:
fail_or_retry_ingestion_job, but only when its claim identity parses into a current, unexpired fence owned by this worker. Payload fields are never used, and a failed transition never strands valid siblings.docs/decisions/2026-10-16-duplicate-sighting-receipt-merge.md).These are ported from
codex/pr3546-f04-candidate(f04c841). The owner chose to put the native PDF provenance change and the pre-claim refusal into a follow-up PR, and both are queued in the ledger inbox. No new migration.Owner decisions applied (16 Oct 2026)
Before:
After:
src/lib/deep-memory-preparation.tsis added toragRankingPatterns.docs/decisions/2026-10-16-*entry.Pilot ticket: the blanket refusal of All-documents answers on a stale ticket (2d05bf0) is reverted. A stale ticket now judges those requests exactly as a valid ticket does. It still refuses the pilot owner, pilot document and pilot query, and an unreadable ticket still refuses everything.
RAG impact: no ranking change. Original-section delivery is limited to the query class that can use it, and it falls back to baseline packing on mixed selections. When one row is observed twice, the receipt merge now keeps the fuller receipt. Selection and ordering are untouched. No canary run.
Review fixes (15 Oct 2026, Claude review thread)
Before: the worker called five database functions that did not exist, so a real upload or reindex would stall in "processing" and deep memory would fail silently. An expired pilot ticket would return 503 to every Answer request, and 74 offline unit tests failed.
After: the missing fenced functions exist in new migrations, and ordinary Answers keep working when a pilot ticket is stale.
Database: this PR adds two migrations.
20261015185000_fenced_ingestion_reset_and_deep_memory.sqladds the fenced reset, image upload and deep-memory functions.20261015200000_fenced_ingestion_followups.sqladds a null-safe staging ownership check and allowsextraction_provenancein the document patch.Every function is SECURITY DEFINER with an empty search_path and is callable by service_role only. Each checks the lease before any write and never touches the served generation. Both migrations are sealed. Merging this PR applies both migrations to the live database within seconds. The owner gave a named yes to merge on 16 Oct 2026.
Bot commits reviewed (Qodo, 18:36):
requireEveryPage: false, because the Python extractor emits no page provenance.store: falseunder the pilot.Other fixes:
/admin/pilotis linked from the admin ingestion panel.allowImportingTsExtensionsreplaces six suppression comments.Summary
Connect administrator ingestion and fenced generation persistence to source-grounded Answer delivery. Changes from main are preserved, including its citation-tap, keyword-timeout, owner-permission and SQL-behaviour CI changes. PR #3544 is already merged and is preserved through ancestry. The native PDF adapter retains page provenance, intact text and table/image relationships. The complete qualified history is recorded in
docs/audit/ingestion-rag-2026-10-15/complete-record.md.Areas touched: source-intake, answer-engine, data-platform, app-experience, delivery, knowledge.
Verification (earlier, by the original author)
at mostsafe-refusal limitation still fails; no assertion was weakened.Risk and rollout
check:driftandcheck:migration-history.Clinical Governance Preflight
🤖 Generated with Claude Code
https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95