Skip to content

Clear cached search results after a background metadata sync - #1673

Merged
Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
fix-large-document-tag-processing
Oct 6, 2026
Merged

Paul Lizer (paullizer) merged 1 commit into
Developmentfrom
fix-large-document-tag-processing

Conversation

@paullizer

Copy link
Copy Markdown
Contributor

Summary

Follow-up to #1658 (merged as 900a3e431). The React v2 port surfaced this gap during its review.

Routes clear the search cache when a metadata or tag change is saved, but since #1658 the chunk merge finishes in the background. A chat or search that runs during the sync can cache the old chunk values and keep serving them for the cache lifetime (search_cache_ttl_seconds, 300 seconds by default; caching is on by default) after the sync completes. That's most noticeable on exactly the large documents #1658 targets, where the sync takes longest.

  • After a pass that changed chunks, run_document_search_metadata_sync() clears cached search results for:
    • the document's workspace (personal owner, group, or public workspace);
    • every user (shared_user_ids) or group (shared_group_ids) with an approved share, since recipients cache their own results.
  • Best effort: invalidation failures are logged under [DOCUMENT_SEARCH_SYNC] and never fail the sync.
  • No invalidation when a pass changed no chunks.
  • functions_documents.py now imports utils_cache directly. It was already a transitive import through functions_search, so the import graph is unchanged.

Refs #1657. The React v2 port has the same invalidation.

Validation

  • python -m pytest functional_tests/test_document_search_metadata_sync.py functional_tests/test_document_metadata_chunk_sync_batching.py functional_tests/test_data_management_search_write_fence_authorization.py -q: 28 passed (adds test_worker_clears_cached_search_results_for_the_owner_and_approved_recipients)
  • python functional_tests/test_codeql_import_cycle_lazy_imports.py: 2/2
  • python functional_tests/test_logging_tag_standardization.py: passed
  • python functional_tests/test_file_sync_tag_definition_application.py: passed
  • python functional_tests/route_tests/test_route_blueprint_policy_inventory.py: 6/6
  • python functional_tests/route_tests/test_route_unauthenticated_policy_contract.py: 4/4
  • python functional_tests/route_tests/test_route_policy_test_coverage.py: 2/2
  • python functional_tests/test_docs_site_quality.py: 6/6
  • python functional_tests/test_docs_app_surface_coverage.py: 7/7
  • python scripts/check_xss_sinks.py --base-sha 900a3e431 --head-sha HEAD application/single_app/functions_documents.py: passed
  • python scripts/check_broken_access_control.py --base-sha 900a3e431 --head-sha HEAD application/single_app/functions_documents.py: passed
  • python -m py_compile application/single_app/functions_documents.py and git diff --check: clean

Notes

  • Version 0.261.053, with a release notes entry. The fix doc docs/explanation/fixes/DOCUMENT_SEARCH_METADATA_SYNC_FIX.md is updated.
  • No routes, settings, or containers changed.

Routes clear the search cache when a metadata change is saved, but the
chunk merge now finishes in the background. A search that ran during
the sync could cache the old values and keep serving them for the cache
lifetime (300 seconds by default) after the sync completed. After a
pass that changed chunks, the worker now clears cached results for the
document's workspace and every user or group with an approved share.

Refs #1657

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Comment on lines +31 to +35
from utils_cache import (
invalidate_group_search_cache,
invalidate_personal_search_cache,
invalidate_public_workspace_search_cache,
)
@paullizer
Paul Lizer (paullizer) merged commit ab21fbb into Development Oct 6, 2026
12 checks passed
Paul Lizer (paullizer) added a commit to paullizer/simplechat that referenced this pull request Oct 7, 2026
Port microsoft#1658 (merged into Development as 900a3e4, including hardening
commit 9c156a4) and its follow-up microsoft#1673 (df3879c) to the React V2
branch. Re-implemented against V2's strict update_document(), group
projection fence, content screening and native group/public document
APIs rather than cherry-picked.

- Mirrored metadata (title, authors, file name, classification, tags) is
  projected by a durable background sync. The request is recorded in the
  settings container before the save, the save carries its token, and a
  worker merges only the changed fields in key-paged batches of 500. The
  worker never writes to the document.
- Access-control lists are projected synchronously and fail closed. Group
  revocations go through the projection fence before the save, grants
  after it. Re-approving an approved share repairs its search access.
- Native group/public receipts add search_sync. Metadata projection
  failures no longer produce document_propagation_incomplete; strict
  group edits on multi-chunk documents no longer fail on the fence etag.
- Legacy routes save every field in one update_document() call and drop
  propagate_tags_to_chunks(), which is removed. Errors are fixed messages.
- Blob tag refresh moves to the worker, best effort and conditioned on the
  blob ETag. After a pass that changed chunks, the worker clears cached
  search results for the workspace and every approved share recipient.
- React and classic workspace UIs show a notice while the index updates.
- Harnesses that exercised the per-chunk loop now drive the sync worker.

Refs microsoft#1657

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants