Add frozen retrieval abstention experiment

[patschmittdev]

Summary

Add a reproducible, evaluation-only experiment for abstaining on retrieval queries the vault cannot answer. Production BM25, query behavior, MCP, and admission policy remain unchanged.

• Freeze 48 separately authored synthetic queries over the unchanged 20-page corpus, with strict manifest/digest validation and explicit AI-authored, human-unverified provenance.
• Compare fixed query-token-coverage thresholds using tuning-only selection: minimize false positives without category-level Recall@5/MRR@5 or identifier regression. Reserve the 50% reduction target for the selected holdout candidate.
• Add npm run eval:abstention, real-file CLI tests, and exact scorer/signed-vault parity. Reuse Gold chunk identities so tied rankings match production.
• Preserve the measured negative result: every threshold was ineligible. At 0.5 coverage, tuning false positives fell from 4/8 to 1/8, but useful-query metrics regressed. No candidate advanced to holdout.
• Document reproduction, compatibility, measurements, and evidence limits in docs/retrieval-evaluation.md. The legacy v1 benchmark remains unchanged.

This work was initiated directly, not from a tracked issue. No closing issue references or changelog update apply; the repository does not keep a changelog.

Trust-boundary impact

None to runtime admission or retrieval authority. The only existing code changes are export visibility for chunkId and the manual evaluator's cwdPath; their implementations and existing callers are unchanged.

The evaluator uses existing ephemeral test keys in disposable vaults. It adds no signer, admission command, runtime rejection policy, model capability, dependency, or index-format change. Shipped MCP remains Gold-only and read-only. Retrieved content remains non-instructional reference data.

Synthetic labels, byte-valid citations, and software signing tests do not establish human review, semantic truth, real-world generalization, or production readiness.

Validation

Validated commit 3d89ecf7c5847369084047f0f011bc0958a1a927, source tree e0676b21cf1156c7da621cb6be3e0201d7549637, after rebasing on current main.

• npm run check, both before commit and on the exact committed source export: tests 746, pass 743, fail 0, skipped 3; all 11 script tests passed. The three skips concern filesystem permissions on Windows. Link and style checks passed.
• node dist/src/cli/main.js check --root . --audit-clean-room, from the 229-file committed source export: PASS, No findings. No audit exclusions or scanner rules were changed.
• Signed external-directory run of the compiled test/manual/retrieval-abstention.js runner: exit 0, exact Gold-ID parity, all seven authorization-boundary checks true, no_eligible_candidate, candidate and decisions null, holdout not_run.
• npm run eval:abstention -- --scorer-only: exit 0 with explicit authorized: null and the same frozen negative result.
• Candidate-bound focused gate: 24/24 tests, exit 0; deterministic verification passed. Independent AI task and final reviews found no blocking defects.
• git diff --check passed before committing; the branch is clean.

The raw app-managed worktree audit still flags its pre-existing .git pointer, which contains a machine path. That pointer is not committed. The source-export audit above is not described as a passing raw-worktree audit.

The frozen fixture does not exercise candidate-present runner parity end to end; that branch has pure-policy unit coverage and inspection evidence. No fixture was retuned to manufacture a passing research result.

No separate packaging or site-visual smoke was run: the package remains private and source-distributed, and no installer or UI files changed. Build, CLI wiring, and the real signed-vault workflow are covered by the checks above.

Checklist

• Behavior changes include tests that use real temporary files.
• Model pathways still cannot write Bronze, knowledge, trust, receipts, reviewed metadata, or indexes.
• No signer, apply, approve, promote command, or --promote flag was added.
• Retrieved content remains reference-only with instruction_authority: none.
• Documentation and compatibility notes match the implementation.
• npm run check passes.
• node dist/src/cli/main.js check --root . --audit-clean-room passes. (Exact committed source export; raw-worktree limitation documented above.)
• No private key, credential, personal path, or private vault content is included.