Large document metadata, tag, and sharing changes don't scale to AI Search chunks #1657

Issue

Saving document metadata or tags does work proportional to the document's chunk count inside the HTTP request. For every chunk, update_document() → update_chunk_metadata() performs an Azure AI Search GET of the full chunk (including its embedding vector), a full upload_documents rewrite, and a Data Management Search write-fence slot acquire/release in Cosmos. Large documents (large XML, long PDFs) exceed the front-end request timeout, and the browser receives an HTML error page that surfaces as an "invalid JSON" error.

#1645 mitigated the personal workspace Save Metadata dialog by splitting the same per-chunk work across browser-driven PATCH continuations (75 chunks per request). The per-chunk cost, the other entry points, and the failure handling are unchanged.

Steps to Reproduce

1. Upload a document that produces thousands of chunks to a group or public workspace.
2. Add a tag or edit metadata and save. Alternatively, rename or delete a tag used by many documents, or bulk-tag several large documents.
3. The request runs until it times out, and some chunks keep the old metadata.

Expected Behavior

• Metadata and tag saves return quickly regardless of document size.
• Search chunks converge to the saved metadata automatically, including after transient failures or worker restarts.
• Sharing approvals and revocations reach the search index, and revocations are enforced before success is reported.

Actual Behavior

• Per-chunk round-trips run inside the request. Bulk tag, tag rename, and tag delete sync tags twice (update_document() plus propagate_tags_to_chunks()) in personal, group, and public workspaces. The group and public metadata PATCH routes call update_document() once per field, so a multi-field save makes several full passes.
• update_document() swallows chunk sync errors and reports success, and partially synced documents are never retried. The Batch large document metadata chunk sync #1645 continuation state lives only in the browser.
• Group shared_group_ids changes made through update_document() (share, approve, unshare, remove-self) are never projected to chunks, because the sync loop skips the chunk write when no non-ACL field changed. Chunks rebuilt by reprocessing or a new revision copy the document's shared_group_ids, so a later unshare can leave a revoked group with search access, and approved shares may never become searchable.
• The full-document GET + upload pattern can re-upload a stale shared_user_ids or shared_group_ids value if it races with a concurrent sharing change.

Impact: what users will notice

The biggest wins are for large documents, large workspaces, and group sharing. Small documents mostly gain reliability.

Unchanged: search ranking and relevance, embeddings, semantic and hybrid search behavior, upload-time chunking and ingestion speed, and the UI layout.

Trade-offs:

• Chat tag filters can lag a few seconds after a save, up to a minute or two on very large documents.
• Sharing actions can now return an error instead of a silent success when Search writes fail or are frozen during a Data Management migration.

Notes

Proposed fix, delivered as two PRs (Development and paullizer-react-v2-ui):

• Replace per-chunk GET + upload with key-only keyset paging (select=id, ordered by id, exact workspace scope filter) and merge_documents batches through the existing fenced Search writer. Only the changed fields are sent; embeddings never round-trip.
• Add a durable background sync for non-ACL metadata (title, authors, file name, classification, tags). A per-document sync revision and status is stored on the Cosmos document in the same write as the edit; a background worker is serialized per document; and a reconciler loop in background_tasks.py retries with backoff.
• Keep ACL projections (shared_user_ids, shared_group_ids) synchronous and fail-closed, using the same merge engine.
• Consolidate the metadata PATCH routes into a single update_document() call, remove the duplicate tag passes, move blob-metadata tag updates into the sync worker, and supersede the Batch large document metadata chunk sync #1645 client continuation protocol.
• The React v2 port preserves the strict / expected_etag / operation_guard / content-screening semantics of its update_document() and the V2 UI receipt contracts.

Acceptance Criteria

• No per-chunk Search or Cosmos work happens inside metadata or tag save requests.
• Chunk metadata converges automatically; failures are retried with backoff and visible on the document.
• Personal and group share approvals and revocations are projected to search, and revocations fail closed.
• Bulk tag, tag rename, and tag delete no longer double-sync.
• Functional tests, fix documentation, and release notes on each branch.
• Separate PRs merged into Development and paullizer-react-v2-ui.

Refs #1645