Skip to content

Fix document evidence validation rejecting exact source quotes - #1737

Merged
Paul Lizer (paullizer) merged 2 commits into
paullizer-react-v2-uifrom
paullizer-document-analysis-validation
Oct 9, 2026
Merged

Paul Lizer (paullizer) merged 2 commits into
paullizer-react-v2-uifrom
paullizer-document-analysis-validation

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

  • A document-to-Word request read every page but stopped before drafting because normalization erased a verbatim footer quote stored as an extraction comment. One rejected citation made the entire analysis partial: 24 of 25 findings were retained.
  • Try exact matching against the original cited chunks before presentation normalization. This shared correction handles literal comments, markup, code and annotations without a footer-specific exception. Empty normalized needles cannot match unrelated source text.
  • Preserve source selectors, ambiguity rejection, original evidence offsets, normalized word/number/sign protections and complete-only composition. No new model calls, repair loops, planner changes or old-run recovery.
  • Add focused regressions and fresh Analyze -> Compose -> Word publication cases. Replaying the unchanged saved responses against fingerprint-matched original chunks now yields all 25 findings with no unresolved evidence. Nothing was deployed or written to production.

Linked issue

N/A - the user requested investigation without creating a tracking issue. This addresses a different validation failure from #1733; its malformed-metadata repair remains unchanged.

Release Notes & Latest Features

  • New Feature
  • Bug Fix
  • UI Enhancement
  • Breaking Change
  • Internal only

Is this visible to end users?

  • Yes
  • No

Is this admin-facing (Admin Settings, governance, deployment, config)?

  • Yes
  • No

Should this become a Latest Feature card?

  • Yes
  • No
  • Already added

Screenshot needed for the card?

  • Yes
  • No
  • Attached

Version bump

  • application/single_app/config.py VERSION third segment bumped, or not needed because this is docs-only
  • deployers/version.txt bumped, or not needed because deployers/ was not changed

Application version: 0.261.309 -> 0.261.310. No deployer changes.

Testing / validation

Offline tests ran with PYTHONPATH including application\single_app and functional_tests, PYTHONDONTWRITEBYTECODE=1, and PYTHONIOENCODING=utf-8. An isolated Python 3.12 environment used repository-pinned Flask 3.1.3/Werkzeug 3.1.6 and cryptography 46.0.7 compatible with the installed pyOpenSSL 25.3.0. Shared/global packages and application dependency manifests were not changed; the temporary environment was removed afterward.

  • python -m pytest .\functional_tests\test_document_analysis_evidence_matching.py .\functional_tests\test_document_analysis_final_results.py .\functional_tests\test_orchestration_document_derivation_reliability.py -q --tb=short --disable-warnings -p no:cacheprovider - 201 passed. Both new Word cases execute real producers, composition, rendering, durable publication and download, with two extraction calls and one composition call; no correction calls.
  • python -m pytest .\functional_tests\test_document_analysis_work_recovery.py .\functional_tests\test_document_analysis_attempt_fences.py .\functional_tests\test_workflow_durable_analysis.py .\functional_tests\test_analyze_orchestration_saved_integration.py -q --tb=short --disable-warnings -p no:cacheprovider - 74 passed.
  • python .\functional_tests\test_docs_app_surface_coverage.py - 7/7 checks passed, committed surface inventory current.
  • python .\functional_tests\test_docs_site_quality.py - 6/6 checks passed.
  • git diff --check - passed.
  • Private session-local probe: python reproduce_analysis_validation.py --expect-fixed and python -O reproduce_analysis_validation.py --expect-fixed - both passed. The unchanged saved responses finalize 25/25 findings against the original 20 chunks with the saved content fingerprint verified. No live model calls or Azure writes; private inputs are not committed.
  • Before the fix, the new focused cases failed on the expected evidence-validation and downstream composition refusal. The original shared Python environment could not collect the derivation suite because its OpenSSL/cryptography combination lacks lib.GEN_EMAIL; the isolated environment resolved this test-only blocker.
  • Not run: production deployment or a live end-to-end request. Existing failed runs are deliberately not migrated; after deployment, test with a fresh request.

Documentation

  • Release notes updated, or not needed
  • Feature documentation updated, or not needed
  • Fix documentation updated, or not needed

Added DOCUMENT_ANALYSIS_LITERAL_EVIDENCE_FIX.md and linked the exact-first correction from the existing evidence-matching fix documentation. No new surface or capability was added. Release-note approval was offered, but the user was unavailable; release notes remain unchanged pending approval.

Security checklist

  • New Flask routes include @swagger_route(security=get_auth_security())
  • Settings sent to non-admin frontends use sanitize_settings_for_user()
  • Browser JavaScript is served from local SimpleChat static assets only; no CDN-hosted JS
  • No secrets, keys, connection strings, or local-only artifacts are included

No routes, frontend settings payloads or browser assets changed. Evidence location is not a claim of factual accuracy or entailment; all existing authorization and completeness boundaries remain intact.

Paul Lizer (paullizer) and others added 2 commits October 9, 2026 09:34
Preserve literal source comments and markup without relaxing citation or completeness safeguards. Add Analyze-to-Word regressions and bump to 0.261.310.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Merge origin/paullizer-react-v2-ui, retain the exact-source validation fix, and align its version references with 0.261.314. Verified 275 relevant tests and documentation checks.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@paullizer
Paul Lizer (paullizer) merged commit 9f446ab into paullizer-react-v2-ui Oct 9, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant