Skip to content

Reconcile admin ingestion, native PDF extraction and grounded Answer - #3546

Merged
gitar-bot[bot] merged 78 commits into
mainfrom
codex/ingestion-rag-pilot-handover-20261014
Oct 16, 2026
Merged

gitar-bot[bot] merged 78 commits into
mainfrom
codex/ingestion-rag-pilot-handover-20261014

Conversation

@BigSimmo

@BigSimmo BigSimmo commented Oct 14, 2026 •

Copy link
Copy Markdown
Owner

Review round, 16 Oct 2026 (after PR #3566)

  • Qodo bot commits undone. d712a15 and 7d5ea74 are reverted with the owner's approval. Gitar confirmed both findings closed.
  • Main merged in. Main's 20261015181500 seal entry is added next to this PR's two entries, and no existing entry is edited (owner's named yes). Both .gitleaksignore notes are kept. The drift manifest points at the merged schema.sql, and CI's freshness job regenerates the snapshot.
  • Stuck job on its last try (Codex P1, 10c7b1d). Before: a database rejection on a job's final attempt cancelled the attempt and skipped fail_or_retry, and the claim query never reclaims an exhausted job, so it stayed processing for good. After: an error carrying a Postgres SQLSTATE rolled back and left the lease held, so it reaches fail_or_retry. Connection (08), operator-intervention (57P), PostgREST gateway and code-less errors still cancel as uncertain.
  • Deep-memory supersede delete (CodeRabbit, major, 1263ccd). The card and index-unit deletes in 20261015185000 now use the same guard as the section delete, so the database repair job's stopgap rows survive a deep-memory commit. This is an unmerged migration of this PR, edited with the owner's named yes. Its seal entry and schema.sql match.
  • CodeRabbit quick fixes (88835ed).
    • A recovered ambiguous upload writes its audit row. The replay-conflict winner path is unchanged, because the winning request already wrote its own row.
    • Managed deep-memory preparation stamps rag-deep-memory-v2.
    • The Storage duplicate response (HTTP 400, statusCode 409) is recognised.
    • A failed egress authority load no longer records a terminal refusal for the whole attempt.
  • Deferred to a follow-up (ledger inbox 6aec8cab):
    • Per-sheet XLSX shared-string projection. The pilot accepts only TXT and PDF.
    • The pilot-client browser proof rerun.
    • The live canary pair for the extractor, chunking and table-fact input changes and the original-section eligibility. This is a paid run that needs the owner's separate yes, planned right after merge.

RAG impact: no ranking, selection or ordering change. The deep-memory version stamp aligns managed output with the current rag-deep-memory-v2 that deep-memory.ts already writes. The supersede-delete guard only narrows which legacy rows are removed.

Local evidence on 67d98d7: verify:cheap passed lint and typecheck. The unit suite ran 31,489 tests, with 2 skipped. The 2 failures were a stale source-shape assertion and a shifted doc anchor, both fixed in 67d98d7, and those two files now pass 283/283. The full suite was not re-run after that fix. Migration immutability and migration-role guards pass.

Main merged in again (16 Oct 2026, after PR #3571)

PR #3571 changed two files this PR also changes. Both sides are kept.

  • src/lib/openai.ts: this PR's ingestion egress wrapper stays. Main's per-call embedding timeoutMs and maxRetries now pass through it, and IngestionOpenAIOptions carries both fields.
  • worker/python/extract_pdf_assets.py: main rewrote edge-strip row recovery around text line segments. This PR's rotated-page fix is carried into the new _extract_page_line_segments, which maps text positions into the rotated page space that table rects use. On a 90-degree page the segment matches this PR's earlier rotated block exactly. Table payloads keep both repeated_populated_geometry and main's column_ranges.
  • tests/pdf-table-extraction.test.ts imports pymupdf instead of the deprecated fitz alias, whose stdout warning broke the test's JSON parse.
  • The design-system route count is now 226, because both sides added one route.
  • Together the two sides put src/lib/rag/rag.ts 29 lines over its no-growth budget. The published-original layout loader this PR added moves unchanged into src/lib/rag/rag-published-original-loader.ts, and the caller signal is still read when the loader runs. rag.ts is now 4,360 lines.

RAG impact: no retrieval behaviour change. The extractor merge keeps both sides' behaviour, the loader move is a pure code move, and no retrieval, selection or ordering code changed.

Local evidence on the merged head: verify:cheap passes lint, typecheck and 31,472 unit tests (the 2 earlier failures were the fitz warning, now fixed). The Python extractor tests pass 8/8 with PyMuPDF 1.28.2, and the table-extraction tests pass 3/3.

Import loops fixed (16 Oct 2026)

Before: the full unit suite failed its architecture check because of two runtime import loops this PR introduced. One was between the XLSX preflight and provenance modules, and the other came from worker/candidate-chunks.ts re-exporting the scoped candidate projection.

After: the XLSX schemas and classifiers move unchanged into src/lib/extractors/xlsx-source-model.ts. Provenance re-exports them, so every existing import still resolves. The candidate-chunks re-export is removed, and its one test imports directly. The clinical doc line anchors that this PR's earlier growth of rag-source-block.ts and rag.ts had shifted are remapped. The owner gave a named yes for the protected extractor edit (docs/decisions/2026-10-16-import-loops-fixed-in-3546.md), and the ledger item that queued the loops separately is removed.

RAG impact: no retrieval behaviour change. The extractor edit is a pure code move with identical exports, and no ranking, selection or ordering code changed.

Codex reference-branch fixes (16 Oct 2026)

Before:

  • A plain-text admin source with no saved section layout was refused.
  • A claimed job row that failed the shape contract was left until its lease expired.
  • If two search methods found the same passage and only one carried its section path, a single-document summary failed with an error.

After:

  • A plain-text source may omit its layout. A layout that is present must still match the source hash.
  • A malformed claimed row is failed terminally through the existing fenced fail_or_retry_ingestion_job, but only when its claim identity parses into a current, unexpired fence owned by this worker. Payload fields are never used, and a failed transition never strands valid siblings.
  • A repeat sighting with less detail keeps the fuller receipt. A genuine conflict still fails closed (owner's named yes; docs/decisions/2026-10-16-duplicate-sighting-receipt-merge.md).

These are ported from codex/pr3546-f04-candidate (f04c841). The owner chose to put the native PDF provenance change and the pre-claim refusal into a follow-up PR, and both are queued in the ledger inbox. No new migration.

Owner decisions applied (16 Oct 2026)

Before:

  • Two upload tests expected an admitted document to be deleted when the upload was cancelled late.
  • A broad question that mixed sources returned an error.
  • Every signed-in answer skipped the saved-answer and search caches.
  • The privacy readiness table still listed the OpenAI and Railway agreements as pending.

After:

  • An admitted upload survives a late cancel. The two tests are rewritten, and cleanup before admission is still covered.
  • A mixed original-section selection falls back to a normal cited answer. An integrity failure inside one published document still fails closed.
  • Only an unscoped broad summary bypasses the caches.
  • The readiness table now matches the register.
  • src/lib/deep-memory-preparation.ts is added to ragRankingPatterns.
  • The import cycles are now fixed in this PR (see above).
  • The indexing repair job keeps provider calls off until its scanner is connected.
  • Each decision has a docs/decisions/2026-10-16-* entry.

Pilot ticket: the blanket refusal of All-documents answers on a stale ticket (2d05bf0) is reverted. A stale ticket now judges those requests exactly as a valid ticket does. It still refuses the pilot owner, pilot document and pilot query, and an unreadable ticket still refuses everything.

RAG impact: no ranking change. Original-section delivery is limited to the query class that can use it, and it falls back to baseline packing on mixed selections. When one row is observed twice, the receipt merge now keeps the fuller receipt. Selection and ordering are untouched. No canary run.

Review fixes (15 Oct 2026, Claude review thread)

Before: the worker called five database functions that did not exist, so a real upload or reindex would stall in "processing" and deep memory would fail silently. An expired pilot ticket would return 503 to every Answer request, and 74 offline unit tests failed.

After: the missing fenced functions exist in new migrations, and ordinary Answers keep working when a pilot ticket is stale.

Database: this PR adds two migrations.

  • 20261015185000_fenced_ingestion_reset_and_deep_memory.sql adds the fenced reset, image upload and deep-memory functions.
  • 20261015200000_fenced_ingestion_followups.sql adds a null-safe staging ownership check and allows extraction_provenance in the document patch.

Every function is SECURITY DEFINER with an empty search_path and is callable by service_role only. Each checks the lease before any write and never touches the served generation. Both migrations are sealed. Merging this PR applies both migrations to the live database within seconds. The owner gave a named yes to merge on 16 Oct 2026.

Bot commits reviewed (Qodo, 18:36):

  • Kept the upload log context.
  • Kept requireEveryPage: false, because the Python extractor emits no page provenance.
  • Kept store: false under the pilot.
  • Kept the alcohol-summary gate.
  • Undid four changes: the inverted admin source check, the wrong reset RPC, the sparse-table admission, and the whole-chunk negation filter.

Other fixes:

  • The upload user client now comes from the cookie session only.
  • A 409 replay conflict uses the standard error envelope.
  • /admin/pilot is linked from the admin ingestion panel.
  • allowImportingTsExtensions replaces six suppression comments.

Summary

Connect administrator ingestion and fenced generation persistence to source-grounded Answer delivery. Changes from main are preserved, including its citation-tap, keyword-timeout, owner-permission and SQL-behaviour CI changes. PR #3544 is already merged and is preserved through ancestry. The native PDF adapter retains page provenance, intact text and table/image relationships. The complete qualified history is recorded in docs/audit/ingestion-rag-2026-10-15/complete-record.md.

Areas touched: source-intake, answer-engine, data-platform, app-experience, delivery, knowledge.

Verification (earlier, by the original author)

  • Hosted pilot: exactly one unchanged approved TXT question and one Copy. An independent clinical and privacy review accepted the output. Low trust and five warnings remain visible.
  • The corrected recorded-response proof passed 13 required checks and 10 unsafe cases. The historical at most safe-refusal limitation still fails; no assertion was weakened.
  • Bounded OCR repair: OCR now requests 450 DPI within the unchanged limits. This is local evidence only, not hosted acceptance.

Risk and rollout

  • This PR contains migrations, so merging applies them to the live database. The owner gave a named yes to merge on 16 Oct 2026. After merge, the post-merge live-drift workflow must pass both check:drift and check:migration-history.
  • Worker queue dispatch remains disabled. Bulk ingestion and re-indexing have not started.
  • General scanned-table admission remains held.
  • Generic legal/clinical release GO and all-user corpus-wide serving are not inferred.

Clinical Governance Preflight

  • Source, numeric, period, population, citation, fencing and server-only access checks retained.
  • Actual bounded clinical delivery/Copy and independent specialist review accepted.
  • Existing restricted reference/education owner decisions and contractual facts retained.
  • General product legal/clinical classification and unrestricted release readiness established.

🤖 Generated with Claude Code

https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95

Capture the scoped ingestion and Answer implementation with portable test
candidates and a sanitised plan, progress and verification record. Retain
failed and unrun evidence; no new readiness checks or bulk ingestion.
@coderabbitai

coderabbitai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown
Contributor

Review in Change Stack →

Warning

Review limit reached

Your organization has reached its usage spending cap. Adjust your spending cap in the billing tab.

Next included review available in 49 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available. Your 99 included PR review attempts over the past 7 days set your current allowance at 1 review per hour.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: BigSimmo/PsychSift/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Team
  • Run ID: 894ef622-2dde-4cd8-8874-2fce64768fbe
📥 Commits

Reviewing files that changed from the base of the PR and between cc21aa0 and ebbee0c.

⛔ Files ignored due to path filters: 2
  • deno.lock is excluded by !**/*.lock
  • package-lock.json is excluded by !**/package-lock.json
📒 Files selected for processing: 206
  • .gitleaksignore
  • AGENTS.md
  • data/repo-awareness-snapshot.json
  • docs/README.md
  • docs/audit/ingestion-rag-2026-10-15/complete-record.md
  • docs/audit/ingestion-rag-2026-10-15/ingestion-evidence-index.json
  • docs/audit/ingestion-rag-2026-10-15/rag-evidence-inventory.json
  • docs/audit/ingestion-rag-2026-10-15/source-inventory.json
  • docs/branch-review-index.md
  • docs/branch-review-records/263f1f0680e9d5cda1bed9bb2a62853d3a34b78b5ad60054971520fdc884cf1c.record.md
  • docs/branch-review-records/6d2fd2e81761e17d32bb56bd41261efe8f9239ef266cf85a512c1021f889dd4b.record.md
  • docs/branch-review-records/7c72c71e0806655f9c5223a2a42aa255d84ff820805f3f6a50edea27cc174ac2.record.md
  • docs/branch-review-records/b32b5aa5dbcc4482fe841337047d0a7d116861f8446e56b47d8cf9b6f1334055.record.md
  • docs/clinical-hazard-analysis.md
  • docs/clinical-hazard-controls.json
  • docs/decisions/2026-10-16-admitted-upload-survives-client-cancel.md
  • docs/decisions/2026-10-16-deep-memory-preparation-is-protected.md
  • docs/decisions/2026-10-16-duplicate-sighting-receipt-merge.md
  • docs/decisions/2026-10-16-import-loops-fixed-in-3546.md
  • docs/decisions/2026-10-16-indexing-repair-provider-calls-stay-off.md
  • docs/decisions/2026-10-16-original-section-delivery-scope.md
  • docs/decisions/2026-10-16-provider-agreements-verified-for-first-release.md
  • docs/design-system/ADOPTION.md
  • docs/design-system/adoption-contract.json
  • docs/design-system/adoption-manifest.json
  • docs/design-system/evidence/admin-source-pilot.json
  • docs/governance/openai-standard-agreement-evidence-2026-10-05.md
  • docs/governance/privacy-readiness.v1.json
  • docs/governance/railway-dpa-completion-evidence-2026-10-05.md
  • docs/organisation/systems/answer-engine.json
  • docs/organisation/systems/delivery.json
  • docs/outstanding-issues-inbox/5d7ae72b-9125-4b0c-9a0e-8d85023a1524.json
  • docs/outstanding-issues-inbox/6aec8cab-618e-4533-bf70-1d871fe03323.json
  • docs/production-readiness.md
  • docs/rag-injection-threat-model.md
  • docs/site-map.md
  • package.json
  • scripts/build-worker.mjs
  • scripts/pr-policy.mjs
  • src/app/admin/pilot/page.tsx
  • src/app/admin/pilot/pilot-client.tsx
  • src/app/api/answer/route.ts
  • src/app/api/answer/stream/route.ts
  • src/app/api/upload/route.ts
  • src/components/developer-area/hub/ingestion-panel.tsx
  • src/lib/admin-reference-preparation-receipt.ts
  • src/lib/admin-text-source-section-layout.ts
  • src/lib/answer-pilot-budget.js
  • src/lib/answer-pilot-diagnostic.js
  • src/lib/answer-pilot-ticket.js
  • src/lib/answer-verification.ts
  • src/lib/answer-wire-budget-guard.js
  • src/lib/chunking.ts
  • src/lib/deep-memory-preparation.ts
  • src/lib/deep-memory.ts
  • src/lib/document-enrichment.ts
  • src/lib/document-index-units.ts
  • src/lib/document-ingress-safety.ts
  • src/lib/extractors/document.ts
  • src/lib/extractors/docx-source-provenance.ts
  • src/lib/extractors/pdf-source-provenance.ts
  • src/lib/extractors/text-source-provenance.ts
  • src/lib/extractors/xlsx-extraction-budget.ts
  • src/lib/extractors/xlsx-original-zip.ts
  • src/lib/extractors/xlsx-source-model.ts
  • src/lib/extractors/xlsx-source-preflight.ts
  • src/lib/extractors/xlsx-source-provenance.ts
  • src/lib/ingestion-generation-evidence.ts
  • src/lib/ingestion-provider-egress.ts
  • src/lib/model-index-extraction.ts
  • src/lib/model-index-version.ts
  • src/lib/openai.ts
  • src/lib/privacy-page-content.tsx
  • src/lib/rag/rag-answer-instructions.ts
  • src/lib/rag/rag-candidate-sources.ts
  • src/lib/rag/rag-context-pack.ts
  • src/lib/rag/rag-extractive-answer.ts
  • src/lib/rag/rag-governed-search.ts
  • src/lib/rag/rag-original-section-delivery.ts
  • src/lib/rag/rag-original-section-eligibility.ts
  • src/lib/rag/rag-published-original-loader.ts
  • src/lib/rag/rag-route-budget.ts
  • src/lib/rag/rag-search-telemetry.ts
  • src/lib/rag/rag-source-block.ts
  • src/lib/rag/rag-source-bound-alcohol-summary.ts
  • src/lib/rag/rag-source-bound-policy-procedures.ts
  • src/lib/rag/rag.ts
  • src/lib/registered-public-answer-options.ts
  • src/lib/registered-published-admin-source-plan.ts
  • src/lib/trusted-document-admission.ts
  • src/lib/types.ts
  • supabase/applied-migration-hashes.json
  • supabase/drift-manifest.json
  • supabase/functions/_shared/document-ingress-safety.ts
  • supabase/functions/_shared/ingestion-attempt.ts
  • supabase/functions/_shared/ingestion-provider-egress.ts
  • supabase/functions/_shared/public-source-policy.ts
  • supabase/functions/_shared/source-anchored-summary.ts
  • supabase/functions/_shared/unavailable-ingestion-provider-egress.ts
  • supabase/functions/indexing-v3-agent/behavior.ts
  • supabase/functions/indexing-v3-agent/index.ts
  • supabase/functions/ingestion-worker/auth.ts
  • supabase/functions/ingestion-worker/index.ts
  • supabase/functions/ingestion-worker/retirement.ts
  • supabase/migrations/20261015185000_fenced_ingestion_reset_and_deep_memory.sql
  • supabase/migrations/20261015200000_fenced_ingestion_followups.sql
  • supabase/schema.sql
  • tests/account-access-model.test.ts
  • tests/admin-reference-preparation-receipt.test.ts
  • tests/answer-pilot-budget-ticket-fallback.test.ts
  • tests/api-validation-contract.test.ts
  • tests/challenger-m5-2-empirical-stress.test.ts
  • tests/chunking-native-table-link.test.ts
  • tests/deep-memory-transaction-sql.test.ts
  • tests/design-system-adoption.test.ts
  • tests/document-extraction-source-fidelity.test.ts
  • tests/document-ingress-safety.test.ts
  • tests/document-metadata-merge.test.ts
  • tests/docx-extraction-budget.test.ts
  • tests/edge-provider-egress.test.ts
  • tests/extracted-document-validation.test.ts
  • tests/helpers/module-graph.ts
  • tests/helpers/synthetic-ingestion-egress.ts
  • tests/indexing-v3-agent.test.ts
  • tests/indexing-v3-attempt.test.ts
  • tests/ingestion-generation-evidence.test.ts
  • tests/ingestion-provider-egress.test.ts
  • tests/ingestion-rpc-contract.test.ts
  • tests/original-source-applicability.test.ts
  • tests/pdf-source-provenance.test.ts
  • tests/pdf-table-extraction.test.ts
  • tests/privacy-page.dom.test.tsx
  • tests/privacy-readiness-contract.test.ts
  • tests/private-access-routes.test.ts
  • tests/production-dynamic-route-reachability.test.ts
  • tests/published-original-request-evidence.test.ts
  • tests/rag-answer-fallback.test.ts
  • tests/rag-original-section-mixed-selection.test.ts
  • tests/rag-routing.test.ts
  • tests/rag-unassessed-delivery-gap.test.ts
  • tests/registered-published-admin-source-plan.test.ts
  • tests/rsc-boundary.test.ts
  • tests/text-source-provenance.test.ts
  • tests/trusted-document-admission.test.ts
  • tests/worker-admin-source-runtime.test.ts
  • tests/worker-admin-text-inspection.test.ts
  • tests/worker-behavior.test.ts
  • tests/worker-candidate-chunks.test.ts
  • tests/worker-clamav-inspection.test.ts
  • tests/worker-generation-attempt-control.test.ts
  • tests/worker-generation-evidence.test.ts
  • tests/worker-generation-image-retention.test.ts
  • tests/worker-generation-quality-diagnostic.test.ts
  • tests/worker-ingestion-attempt.test.ts
  • tests/worker-lease-transition-contract.test.ts
  • tests/worker-malware-inspection.test.ts
  • tests/worker-managed-admin-text-terminal.test.ts
  • tests/worker-managed-local-memory-producer.test.ts
  • tests/worker-managed-native-parent-closure.test.ts
  • tests/worker-managed-prepared-supplier.test.ts
  • tests/worker-managed-private-terminal.test.ts
  • tests/worker-managed-provider-vectors.test.ts
  • tests/worker-managed-vector-final.test.ts
  • tests/worker-original-witness-readback.test.ts
  • tests/worker-page-serialization.test.ts
  • tests/worker-provider-authority-read.test.ts
  • tests/worker-published-admin-source-layout.test.ts
  • tests/worker-row-contract.test.ts
  • tests/worker-run-loop.test.ts
  • tests/worker-shadow-extraction.test.ts
  • tests/worker-visual-capture.test.ts
  • tests/worker-whole-original-supplier.test.ts
  • tests/xlsx-extraction-budget.test.ts
  • tests/xlsx-source-provenance.test.ts
  • tsconfig.json
  • worker/admin-source-runtime.ts
  • worker/admin-text-inspection.ts
  • worker/behavior.ts
  • worker/candidate-chunks.ts
  • worker/clamav-inspection.ts
  • worker/generation-evidence.ts
  • worker/index.ts
  • worker/main.ts
  • worker/malware-inspection.ts
  • worker/managed-admin-pdf-terminal.ts
  • worker/managed-admin-text-terminal.ts
  • worker/managed-local-memory-producer.ts
  • worker/managed-native-parent-closure.ts
  • worker/managed-prepared-supplier.ts
  • worker/managed-private-terminal.ts
  • worker/managed-provider-vectors.ts
  • worker/managed-vector-final.ts
  • worker/original-witness-readback.ts
  • worker/provider-egress.ts
  • worker/published-admin-source-layout-contract.ts
  • worker/published-admin-source-layout.ts
  • worker/python/extract_pdf_assets.py
  • worker/python/test_extract_pdf_assets_budget.py
  • worker/python/test_extract_pdf_assets_hard_tables.py
  • worker/python/test_extract_pdf_assets_hostile.py
  • worker/python/test_extract_pdf_assets_table_reading_order.py
  • worker/row-contracts.ts
  • worker/scoped-candidate-projection.ts
  • worker/table-facts.ts
  • worker/types.ts
  • worker/whole-original-supplier.ts
📝 Walkthrough

The pull request adds an administrator source-upload pilot, provenance-aware document extraction, fenced ingestion and provider-egress controls, managed generation flows, checked RAG answer delivery, and bounded PDF OCR and table handling. It also updates governance and audit records, route inventories, dependencies, and tests.

Changes: Ingestion and RAG pilot

Layer / File(s) Summary
Pilot records and repository references
docs/audit/*, docs/governance/*, docs/decisions/*, docs/branch-review-*, docs/design-system/*, docs/production-readiness.md, data/repo-awareness-snapshot.json
Adds audit inventories, review records, decisions, and pilot route and design-system references. Records the OpenAI and Railway agreement items as verified within the stated first-release scope.
Administrator source admission
src/app/admin/pilot/*, src/app/api/upload/route.ts, src/lib/trusted-document-admission.ts, src/lib/admin-reference-preparation-receipt.ts, tests/*upload*, tests/trusted-document-admission.test.ts
Adds an administrator pilot for approved TXT and PDF sources with MFA checks. The upload route uses idempotency, authority validation, trusted storage upload, and admission replay handling.
Source provenance and extraction
src/lib/extractors/*, src/lib/types.ts, worker/behavior.ts, worker/python/extract_pdf_assets.py, tests/*source-provenance*
Adds source identity validation for DOCX, PDF, TXT, and XLSX. XLSX extraction inspects archive structure and records source facts. PDF extraction updates bounded OCR, rotated-page coordinates, and table recognition.
Fenced ingestion and safety
supabase/functions/_shared/*, worker/behavior.ts, worker/provider-egress.ts, worker/admin-*, worker/generation-evidence.ts, supabase/schema.sql, supabase/migrations/*
Adds lease-bound attempts, retained generation evidence, provider-egress permits, bounded malware inspection, and generation-scoped persistence operations.
Managed generation workflows
worker/candidate-chunks.ts, worker/managed-*.ts, worker/scoped-candidate-projection.ts, worker/published-admin-source-layout.ts
Adds managed preparation, vector production, staged readback validation, source-layout projection, and terminal completion flows.
Enrichment and deep memory
src/lib/deep-memory*.ts, src/lib/document-enrichment.ts, supabase/functions/indexing-v3-agent/behavior.ts, supabase/migrations/*fenced_ingestion*
Adds generation-fenced enrichment and deep-memory staging, cleanup, and commit paths.
Checked RAG answer delivery
src/lib/rag/*, src/lib/registered-public-answer-options.ts, src/lib/registered-published-admin-source-plan.ts, src/lib/answer-pilot-*, src/app/api/answer/*
Adds checked published-source layouts and proof-bound original-section delivery. Answer routes apply pilot budgeting, and answer finalization validates delivered scope and completeness.
Supporting contracts and tests
worker/row-contracts.ts, worker/types.ts, scripts/*, tests/*, package.json, tsconfig.json
Updates worker contracts, build aliases, policy classification, route counts, and tests for the new flows.

Priority: ➖ Normal

Estimated code review effort: 5 (Critical) | ~120 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AdminPilot
  participant UploadRoute
  participant TrustedStorage
  participant AdmissionRPC
  AdminPilot->>UploadRoute: Submit source, authority, MFA session, and idempotency key
  UploadRoute->>TrustedStorage: Upload and verify source bytes
  UploadRoute->>AdmissionRPC: Dispatch trusted document admission
  AdmissionRPC-->>UploadRoute: Return admission outcome
  UploadRoute-->>AdminPilot: Return document and job identifiers or upload status
Loading
Merge Risk: 🟡 Moderate · up to cc21a

This change adds two live migrations. One of them can permanently delete repaired memory cards and index units when deep memory is committed. Large multi-sheet spreadsheets may also now fail ingestion permanently. Retrieval inputs changed without the required canary comparison. Resolve these issues before merging.

Pre-merge checks: Passed 4 · Inconclusive 1

Failed Failed checks: 1 inconclusive

Check name Status Explanation Resolution
Docstring Coverage Inconclusive Docstring coverage is 24.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 320 functions across 50 files. (148 skipp… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Title check Passed The title clearly summarizes the primary changes: administrator ingestion, native PDF extraction, and grounded Answer delivery.
Description check Passed The description is detailed and covers the summary, affected areas, verification results, migration risk, rollout limits, governance status, and unresolved follow-up work. It omits the template's sepa…
Full details: Description check

Explanation

The description is detailed and covers the summary, affected areas, verification results, migration risk, rollout limits, governance status, and unresolved follow-up work. It omits the template's separate Notes section, but the required information is otherwise substantially complete.

Full details: Docstring Coverage

Explanation

Docstring coverage is 24.69% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 320 functions across 50 files. (148 skipped: 42 unsupported, 106 over the file limit.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 3
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
⚔️ Resolve merge conflicts: 💡
  • Resolve merge conflict in branch codex/ingestion-rag-pilot-handover-20261014
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Comment @coderabbitai help to get the list of available commands.

@supabase

supabase Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

Updates to Preview Branch (codex/ingestion-rag-pilot-handover-20261014) ↗︎

Deployments Status Updated
Database ✅ Fri, 16 Oct 2026 11:07:20 UTC
Services ✅ Fri, 16 Oct 2026 11:07:20 UTC
APIs ✅ Fri, 16 Oct 2026 11:07:20 UTC

Tasks are run on every commit but only new migration files are pushed.
Close and reopen this PR if you want to apply changes from existing seed or migration files.

Tasks Status Updated
Configurations ✅ Fri, 16 Oct 2026 11:07:22 UTC
Migrations ✅ Fri, 16 Oct 2026 11:07:24 UTC
Seeding ✅ Fri, 16 Oct 2026 11:07:25 UTC
Edge Functions ✅ Fri, 16 Oct 2026 11:07:31 UTC

View logs for this Workflow Run ↗︎.
Learn more about Supabase for Git ↗︎.

Comment thread src/app/api/answer/route.ts Resolved
Comment thread src/lib/answer-pilot-budget.js Resolved
@codeant-ai

codeant-ai Bot commented Oct 14, 2026 •

Copy link
Copy Markdown

🏁 CodeAnt Quality Gate Results

Commit: ebbee0c4
Scan Time: 2026-10-16 11:27:06 UTC

✅ Overall Status: PASSED

Quality Gate Details

Quality Gate Status Details
Secrets ✅ PASSED 0 secrets found
Duplicate Code ✅ PASSED 3.3% duplicated
SAST ✅ PASSED No security issues
Bugs ✅ PASSED Rating S: No bugs
IAC ✅ PASSED Rating S: No issues

View Full Results

@BigSimmo BigSimmo changed the title feat(ingestion): preserve managed reference pipeline and RAG handover Reconcile admin ingestion, native PDF extraction and grounded Answer Oct 15, 2026
@BigSimmo
BigSimmo marked this pull request as ready for review October 15, 2026 18:00
Owner approved in the PR 3546 thread on 16 Oct 2026: "Yes, fix the deep memory
leftover facts in the PR 3546 migration."

Legacy index-health repair stamps producer-less cards, index units and
sections with the committed index generation. The fenced commit's legacy
branch only matched a null index generation, so those rows survived a
deep-memory refresh next to the new ones. The branch now also matches the
committed generation, keeping the local-worker and deep-memory version
markers that exclude db_repair rows.

Updates the unmerged migration, schema.sql, its seal hash, and the drift
manifest (schema hash, and the function fingerprint computed with a local
Postgres using the same normalisation CI uses).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
Comment thread worker/behavior.ts Resolved
@qodo-code-review

Copy link
Copy Markdown
Contributor

Code review by qodo was updated up to the latest commit 6668b99

@gitar-bot
gitar-bot Bot disabled auto-merge October 16, 2026 10:26
@qodo-code-review

Copy link
Copy Markdown
Contributor

Code review by qodo was updated up to the latest commit ad07275

The guard that refuses to null another producer's card-to-section link
now uses the same legacy predicate as the supersede deletes, including
rows health repair stamped with the committed generation. Reseals the
migration and refreshes the drift manifest.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95
@gitar-bot
gitar-bot Bot enabled auto-merge (squash) October 16, 2026 10:36
gitar-bot[bot]
gitar-bot Bot previously approved these changes Oct 16, 2026

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

@qodo-code-review

Copy link
Copy Markdown
Contributor

Code review by qodo was updated up to the latest commit 54f8559

The PR added the delivered-prose check that names required parts an
admitted-passage answer does not cover, without tests. Coverage for the
RAG answer files fell to 88.9% of functions against a 90% floor. These
tests pin the observed behaviour: a missing required part, a missing
facet detail, a negated safety constraint, a mismatched population, and
the no-gap case. No source change.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

gitar-bot[bot]
gitar-bot Bot previously approved these changes Oct 16, 2026

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

@qodo-code-review

Copy link
Copy Markdown
Contributor

Code review by qodo was updated up to the latest commit 6afd8bc

Addresses the Qodo review note that the facet test only checked one
missing detail. The fixture now delivers route and frequency, so the gap
must name exactly the missing dosing maximum and the absent monitoring.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01M6gRtN6r4vZDgDfzMtjs95

@gitar-bot gitar-bot Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gitar has auto-approved this PR (configure)

@gitar-bot

gitar-bot Bot commented Oct 16, 2026

Copy link
Copy Markdown
Contributor
Code Review ✅ Approved 8 closed / 8 findings

🔴 High risk · Adding fenced ingestion migrations and admin ingestion/Answer delivery could commit incorrect document generations or serve unsupported clinical guidance if provenance checks fail.

Reconciles admin ingestion, native PDF extraction and grounded Answer delivery, with fenced persistence and new migrations for the worker. Six earlier findings are resolved, including the missing Answer route module, the migration auto-apply concern, pilot ticket 503s, and the foreign-section guard now covering the committed-generation branch in 20261015185000. No open findings remain; one edge case (an orphaned private-bucket copy after an ambiguous dispatch retry) is recorded as a follow-up and does not block approval.

✅ 8 closed
✅ Bug: Answer routes import a module that does not exist

📄 src/app/api/answer/route.ts:21 📄 src/app/api/answer/route.ts:77 📄 src/app/api/answer/stream/route.ts:36 📄 src/app/api/answer/stream/route.ts:333
Both src/app/api/answer/route.ts:21 and src/app/api/answer/stream/route.ts:36 import rejectPatientDetailQuestion from @/lib/clinical-question-patient-check-server. That file is not in the repo and this PR does not add it. A repo-wide search finds the symbol only at these two call sites. The build/typecheck fails, so neither answer endpoint can ship. Either add the server helper in this PR or remove the import and calls until it exists.

✅ Quality: Migrations say 'CANDIDATE ONLY / no production application' but auto-apply

📄 supabase/migrations/20261013075944_administrator_txt_fenced_runtime_cutover.sql:1-2 📄 supabase/migrations/20261013075944_administrator_txt_fenced_runtime_cutover.sql:167-170 📄 supabase/migrations/20261013144347_managed_generation_quality_repair_ownership.sql:1-3 📄 supabase/migrations/20261013200007_managed_native_pdf_terminal_append.sql:1
20261013075944_administrator_txt_fenced_runtime_cutover.sql starts with "CANDIDATE ONLY... reviewed operator cutover required. No production application... authority". 20261013144347 calls itself a "Proposed function-only migration". Under the Supabase GitHub integration, any file in supabase/migrations/ is applied to the live clinical database when it merges to main. These headers promise a deferred, operator-gated deploy, which is the misleading pattern this repo forbids. The cutover migration also aborts with ingestion_lease_migration_drain_required if any job is processing. It also checks md5 hashes of more than 60 live function bodies against fixed preimages, and the PR says those functions already conflict with newer main. So it is likely to fail at merge time and leave silent drift. Fix: keep these files out of supabase/migrations/ until the operator window, or replace the deferred-deploy wording with an accurate merge-gate statement and rebase the preimage hashes onto current main.

✅ Bug: Expired pilot ticket makes every Answer request return 503

📄 src/lib/answer-pilot-budget.js:87-97 📄 src/lib/answer-pilot-ticket.js:131-140 📄 src/app/api/answer/route.ts:149-163
runAnswerWithPilotBudget runs on every request to /api/answer and /api/answer/stream. It turns any failure from loadServerAnswerPilotTicket() into AnswerPilotBudgetError("answer_pilot_ticket_load_failed"). The loader calls fail() (a throw) when the env ticket is set but Date.now() >= pinned.expiresAtMs (answer-pilot-ticket.js:136). So once a configured pilot ticket expires, or is malformed, every user's Answer request gets a 503 until someone removes the env var. The 'closed default' only covers an unset variable. Also, pilotIdentityMatches treats a request as related when only the owner, only the document, or only the query matches. While a ticket is active, other questions from the pilot owner and other users' questions about the pilot document are refused as near_match_refused. Fix: when the ticket is expired or invalid, fall through to answerWork() (fail closed for pilot dispatch only), and refuse only requests that match the exact pilot identity.

✅ Bug: Expired pilot ticket again returns 503 for default All-documents Answers

📄 src/lib/answer-pilot-ticket.js:13-15 📄 src/lib/answer-pilot-ticket.js:21-23 📄 src/lib/answer-pilot-budget.js:67-74 📄 src/lib/answer-pilot-budget.js:98-101
With no filters and no selected documents, resolveSearchScope returns documentIds: undefined (src/lib/search-scope.ts). The route passes that through as resolvedDocumentIds, and the new line 23 returns true for any non-array scope. So whenever the pilot ticket is expired or unreadable, every ordinary unfiltered Answer from any user throws answer_pilot_ticket_load_failed. This is the most common kind of request, so the change brings back most of the outage that the earlier 'expired ticket makes every Answer 503' finding fixed.

It also contradicts the docstring at lines 14-15 and the PR description, which both say every other Answer keeps working. And it is stricter than the valid-ticket path: with a usable ticket, pilotIdentityMatches treats the same unrestricted request (different owner, different query) as unrelated and answers it.

Suggested fix: drop the line 23 short-circuit so unrestricted scopes go through the same owner and query-digest check a valid ticket would apply. If corpus-wide retrieval of the pilot document is a real concern, add that rule to pilotIdentityMatches too, so both paths behave the same, and update the docstring and PR text.

✅ Bug: Image-free PDF path completes without publication or fidelity gate

📄 worker/main.ts:1985-1996
After stageManagedVectorFinal, the new else branch calls the generic completeJob → complete_ingestion_job. The SQL version of that function (20261013200007, line 1562) returns {ok:false, reason:'technical_supplier_not_implemented'} whenever a candidate row exists for the job and neither admin_text_commit_allowed nor admin_pdf_commit_allowed holds. runManagedPrivateSupplier creates that candidate row, and those predicates only pass inside the dedicated terminal functions. So parseIngestionMutationResult throws, the attempt is cancelled, and the job goes to fail/retry. The code also has a deeper problem: this branch skips the loadManagedAdminPdfCompletionEvidence/evaluateManagedAdminPdfCompletion gate that the image branch enforces, and it never commits or publishes the generation. Even if the RPC were accepted, the job would read completed while the document has no served generation. The comment just below still says completeness and fidelity 'prevent candidate admission, completion and publication'. The fix is to route image-free PDFs through completeManagedAdminPdfAttempt (or a fenced terminal RPC of the same kind), with the same completion-evidence check. Until then, keep the technical_supplier_not_implemented throw.

...and 3 more closed from earlier reviews

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Approved

For more verification details, comment gitar display:verbose.

Options

Auto-apply is off → Gitar will not commit updates to this branch.
Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Auto-apply Compact
gitar auto-apply:on         
gitar display:verbose         

Important

Your trial ends in 5 days — upgrade now to keep code review, CI analysis, auto-apply, custom automations, and more.

Was this helpful? React with 👍 / 👎 | Gitar

@qodo-code-review

Copy link
Copy Markdown
Contributor

Code review by qodo was updated up to the latest commit ebbee0c

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants