Sync document search metadata in the background with batched merges by paullizer · Pull Request #1658 · microsoft/simplechat

Paul Lizer (paullizer) · 2026-10-06T17:39:23Z

Summary

Fixes the root cause behind #1645 (now merged). Metadata, tag, and sharing changes no longer do per-chunk Azure AI Search work inside the HTTP request.

Merge engine: chunk keys are listed with key paging (select=id, ordered by id, exact workspace scope; the same keyset approach Data Management migrations use). Changed fields are written with merge_documents batches (default 500) through the existing Search write fence. Embeddings never round-trip.
Durable background sync: each metadata save records a sync request (in the settings container) before saving the document. The save stores the request's unique token on the document. A small dedicated worker, leased per document, projects the latest value of every pending field. It completes only once the document carries the latest request's token, so a save that hasn't landed yet is never skipped.
Failure handling: failures back off (1/5/15/60 min, then every 6 h). Search writes frozen by a Data Management migration retry every 5 minutes without escalating the backoff. A reconciler loop in background_tasks.py resumes failed or orphaned syncs. The worker never writes back to the document, so list sort order (_ts) and etags only change on user saves.
ACL projections stay synchronous and fail closed: shared_user_ids / shared_group_ids changes reach every chunk before they're saved. This also fixes group share/approve/unshare/remove-self, which never reached Search before. Only real access changes are projected: a missing list and an empty list count as equal, so the legacy-document upgrade never depends on Search. A frozen Search fence returns a retryable 503; any other Search failure returns a safe 500. In both cases nothing is saved.
Routes and UI:
- Personal, group, and public metadata PATCH routes save in one update_document() call and return search_sync.
- Bulk tag, tag rename, and tag delete drop their duplicate propagate_tags_to_chunks() pass; the function is removed.
- Batch large document metadata chunk sync #1645's browser continuation loop is retired.
- All three workspace dialogs show a brief info toast while the index updates.

Refs #1657 · Follow-up: separate port PR for paullizer-react-v2-ui.

What users will notice

Area	Before	After
Azure load	Every chunk's embedding was downloaded and re-uploaded to change a tag, at about 2 Search calls and 4 Cosmos fence ops per chunk (10k chunks ≈ 20k Search calls). Bulk tag ops doubled that.	A merge of a few hundred bytes per chunk. 10k chunks ≈ 40 Search calls, about 500x fewer.
Metadata/tag save on a large doc	Personal: many sequential requests. Group/public: timeout with "invalid JSON".	Returns immediately.
Bulk tag, tag rename, tag delete	Every chunk synced twice inside one request.	Only Cosmos writes run in the request.
Chat tag filters and citations	Partially synced docs stayed partly findable and showed stale titles or classifications, with no retry.	Chunks converge automatically.
Group sharing in search	Approvals and revocations never reached the chunks.	Approvals become searchable; revocations are enforced before success is reported.

Trade-offs:

Tag filters can lag a few seconds after a save, up to a minute or two on huge docs.
Sharing changes now return an error instead of a silent "success" when Search can't enforce them.
Group shares approved before this change are projected the next time that document's sharing changes; existing chunk access lists are not backfilled.

Validation

python -m pytest functional_tests/test_document_search_metadata_sync.py functional_tests/test_document_metadata_chunk_sync_batching.py functional_tests/test_data_management_search_write_fence_authorization.py -q: 27 passed. This includes regression tests for a stale-token race, a record removed mid-write, frozen-fence retry, and equal-but-differently-shaped access lists.
python functional_tests/test_document_metadata_chunk_sync_batching.py: passed (a 5,000-chunk tag save does no Search work in the request; 10 merge batches)
python functional_tests/test_document_auto_metadata_extraction_consistency.py: 3/4 passed. test_dispatcher_owns_final_metadata_extraction fails identically on the base commit 2ab4cf43d (pre-existing, unrelated).
python functional_tests/test_logging_tag_standardization.py: passed (267 tags; added [DOCUMENT_SEARCH_SYNC], [DOCUMENT_SHARING], and the previously undocumented [DOCUMENT_METADATA])
python functional_tests/test_file_sync_tag_definition_application.py: passed
python functional_tests/route_tests/test_route_blueprint_policy_inventory.py: 6/6
python functional_tests/route_tests/test_route_unauthenticated_policy_contract.py: 4/4
python functional_tests/route_tests/test_route_policy_test_coverage.py: 2/2
python scripts/check_swagger_routes.py application/single_app/route_backend_documents.py application/single_app/route_backend_group_documents.py application/single_app/route_backend_public_documents.py: passed
python scripts/check_xss_sinks.py --base-sha 2ab4cf43d --head-sha HEAD <8 changed app files>: passed
python scripts/check_broken_access_control.py --base-sha 2ab4cf43d --head-sha HEAD <5 changed Python files>: passed
python functional_tests/test_docs_site_quality.py: 6/6
python functional_tests/test_docs_app_surface_coverage.py: 7/7
python functional_tests/test_release_notes_check_fork_permissions.py: 2/2
python -m py_compile on the touched Python files, node --check on workspace-documents.js and public_workspace.js, and git diff --check: clean
functional_tests/test_backend_documents_swagger_integration.py and test_backend_group_documents_swagger_integration.py: 3/6 on both this branch and base 2ab4cf43d. They fail on stale hard-coded route counts; this PR adds no routes.

Not run: ui_tests/test_workspace_metadata_search_sync_toast.py (new). It needs SIMPLECHAT_UI_BASE_URL and authenticated storage state. It checks for one PATCH and that the toast appears only when a sync is queued.

Notes

Version 0.261.052. Fix doc: docs/explanation/fixes/DOCUMENT_SEARCH_METADATA_SYNC_FIX.md. The Batch large document metadata chunk sync #1645 fix doc is marked superseded.
No new routes, settings, containers, or Azure resources, and no migration. Existing docs without a sync record are treated as in sync.
Suggested smoke test:
1. Tag a large (thousands of chunks) group or public document. The dialog should close immediately.
2. In the settings container, a document_search_metadata_sync record appears and is then deleted.
3. Chat with that tag filter and confirm the doc's chunks show up.
Follow-up candidates (out of scope):
- Backfill existing group chunk access lists, and serialize concurrent access changes per document (the concurrent read-modify-write race in update_document() predates this PR).
- Move set_document_chunk_visibility() (revision archive/restore) to merges.

- Replace per-chunk AI Search GET + full re-upload with key-paged merge_documents batches that send only the changed fields, so embeddings never round-trip and Search/Cosmos calls drop ~500x. - Record a durable sync request with each metadata save; a leased background worker projects pending fields, retries with backoff, and a reconciler loop resumes failed or orphaned syncs. - Project shared_user_ids/shared_group_ids changes to chunks before saving them (fail closed), fixing group sharing changes that never reached Search. - Save metadata PATCH fields in one update in all workspaces, drop the duplicate tag passes, and retire the #1645 browser continuation loop. Refs #1657 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>


            return jsonify({'message': 'Share approved' if updated else 'Already approved'}), 200
+        except DocumentSearchAclProjectionDeferredError as exc:
+            response = jsonify({'error': str(exc)})


+            response.headers['Retry-After'] = '150'
+            return response, 503
+        except DocumentSearchAclProjectionError as exc:
+            return jsonify({'error': str(exc)}), 500



+def _group_share_search_acl_error_response(error):
+    """Return the safe response for a sharing change the search index could not enforce."""
+    response = jsonify({'error': str(error)})


+    return 'user_id', user_id
+
+
+def get_document_search_acl_fields(group_id=None, public_workspace_id=None):


- Match a per-request token stored on the document instead of comparing revisions, so a stale value from an earlier request can never complete a newer sync before its save lands. - Retry a sync request when a finishing worker removes the record between read and replace, instead of failing the user's save. - Retry Search writes frozen by a Data Management migration on a short fixed delay without climbing the failure backoff. - Project only real access changes; missing and empty access lists are equivalent, so the legacy document upgrade never depends on Search. Refs #1657 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Port microsoft#1658 (merged into Development as 900a3e4, including hardening commit 9c156a4) and its follow-up microsoft#1673 (df3879c) to the React V2 branch. Re-implemented against V2's strict update_document(), group projection fence, content screening and native group/public document APIs rather than cherry-picked. - Mirrored metadata (title, authors, file name, classification, tags) is projected by a durable background sync. The request is recorded in the settings container before the save, the save carries its token, and a worker merges only the changed fields in key-paged batches of 500. The worker never writes to the document. - Access-control lists are projected synchronously and fail closed. Group revocations go through the projection fence before the save, grants after it. Re-approving an approved share repairs its search access. - Native group/public receipts add search_sync. Metadata projection failures no longer produce document_propagation_incomplete; strict group edits on multi-chunk documents no longer fail on the fence etag. - Legacy routes save every field in one update_document() call and drop propagate_tags_to_chunks(), which is removed. Errors are fixed messages. - Blob tag refresh moves to the worker, best effort and conditioned on the blob ETag. After a pass that changed chunks, the worker clears cached search results for the workspace and every approved share recipient. - React and classic workspace UIs show a notice while the index updates. - Harnesses that exercised the per-chunk loop now drive the sync worker. Refs microsoft#1657 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Paul Lizer (paullizer) mentioned this pull request Oct 6, 2026

Large document metadata, tag, and sharing changes don't scale to AI Search chunks #1657

Closed

6 tasks

github-advanced-security Bot found potential problems Oct 6, 2026

View reviewed changes

Paul Lizer (paullizer) merged commit 900a3e4 into Development Oct 6, 2026
13 checks passed

This was referenced Oct 6, 2026

Clear cached search results after a background metadata sync #1673

Merged

Port durable batched search metadata sync to React v2 #1674

Merged

Navigation Menu

Repository navigation

Sync document search metadata in the background with batched merges - #1658

Sync document search metadata in the background with batched merges#1658
Paul Lizer (paullizer) merged 2 commits into
Developmentfrom
fix-large-document-tag-processing

Paul Lizer (paullizer) commented Oct 6, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return 'user_id', user_id


		def get_document_search_acl_fields(group_id=None, public_workspace_id=None):

Conversation

Paul Lizer (paullizer) commented Oct 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

What users will notice

Validation

Notes

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Paul Lizer (paullizer) commented Oct 6, 2026 •

edited

Loading