Skip to content

Batch large document metadata chunk sync - #1645

Merged
Paul Lizer (paullizer) merged 1 commit into
microsoft:Developmentfrom
Xeelee33:fix-large-document-tag-processing
Oct 6, 2026
Merged

Paul Lizer (paullizer) merged 1 commit into
microsoft:Developmentfrom
Xeelee33:fix-large-document-tag-processing

Conversation

@chadpalmer

Copy link
Copy Markdown
Contributor

Summary

  • Adds resumable batching for document metadata search chunk synchronization so large uploaded documents do not require one long metadata-save request.
  • Updates the personal workspace Save Metadata flow to continue short JSON PATCH requests until chunk sync is complete.
  • Removes duplicate tag search chunk propagation while preserving blob metadata tag updates.

Why

Saving tags or other metadata for large XML/large document uploads could time out while synchronizing every search chunk. The browser then tried to parse the timeout HTML response as JSON, surfacing an invalid JSON error even though the underlying issue was the long-running metadata update.

Changes

  • update_document() now supports chunk_sync_limit, chunk_sync_offset, and chunk_sync_fields, returning progress when search chunk metadata sync needs continuation.
  • The personal document metadata PATCH route now applies metadata updates once, returns chunk-sync continuation state, validates continuation offsets, and returns a stable JSON error on failure.
  • workspace-documents.js now retries/continues the metadata PATCH flow until chunk sync completes.
  • Added fix documentation and release notes for v0.261.051.
  • Added a focused functional regression test for batch-and-resume chunk sync.

Validation

  • git diff --check upstream/Development...HEAD
  • python -m py_compile application/single_app/config.py application/single_app/functions_documents.py application/single_app/route_backend_documents.py functional_tests/test_document_metadata_chunk_sync_batching.py
  • python functional_tests/test_document_metadata_chunk_sync_batching.py
  • node --check application/single_app/static/js/workspace/workspace-documents.js
  • python scripts/check_swagger_routes.py application/single_app/route_backend_documents.py
  • python functional_tests/route_tests/test_route_blueprint_policy_inventory.py
  • python functional_tests/route_tests/test_route_unauthenticated_policy_contract.py
  • python functional_tests/route_tests/test_route_policy_test_coverage.py
  • python scripts/check_xss_sinks.py --base-sha upstream/Development --head-sha HEAD application/single_app/functions_documents.py application/single_app/route_backend_documents.py application/single_app/static/js/workspace/workspace-documents.js
  • python scripts/check_broken_access_control.py --base-sha upstream/Development --head-sha HEAD application/single_app/functions_documents.py application/single_app/route_backend_documents.py
  • python functional_tests/test_docs_site_quality.py

Notes

  • Local browser UI tests were not run because the app/authenticated browser state was not started during validation.
  • Recommended manual smoke test: upload or select a large processed document, add/select a tag such as bills, click Save Metadata, and confirm the dialog completes without an invalid JSON error.

@chadpalmer

Copy link
Copy Markdown
Contributor Author

Paul Lizer (@paullizer) Bionic711 This PR fixes an issue where a user adding a metadata tag to a large document causes a invalid json error due to an html error page being sent back that is caused by a long request timing out during the metadata update process.

@paullizer
Paul Lizer (paullizer) merged commit 2ab4cf4 into microsoft:Development Oct 6, 2026
12 checks passed
@paullizer

Copy link
Copy Markdown
Contributor

Great! Thanks for sharing, I didnt know this was a bug. I will investigate improving the underlying code to better support large files.

Chad Palmer (chadpalmer) pushed a commit to Xeelee33/simplechat that referenced this pull request Oct 6, 2026
- Replace per-chunk AI Search GET + full re-upload with key-paged
  merge_documents batches that send only the changed fields, so
  embeddings never round-trip and Search/Cosmos calls drop ~500x.
- Record a durable sync request with each metadata save; a leased
  background worker projects pending fields, retries with backoff, and
  a reconciler loop resumes failed or orphaned syncs.
- Project shared_user_ids/shared_group_ids changes to chunks before
  saving them (fail closed), fixing group sharing changes that never
  reached Search.
- Save metadata PATCH fields in one update in all workspaces, drop the
  duplicate tag passes, and retire the microsoft#1645 browser continuation loop.

Refs microsoft#1657

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants