Data Management backups silently skip every Cosmos container larger than one page (Invalid Continuation Token) #1258

Summary

Scheduled and manual Data Management backups silently drop every Cosmos container that holds more than one page of documents. Affected containers are recorded as failed resources and are not present in the backup artifact set, so the backup completes "with warnings" while the largest containers — typically personal_conversations and personal_messages — are never backed up.

This is a data-protection defect: any deployment with more than 100 conversations or 100 messages is producing incomplete backups.

Impact

• Severity: P0 — silent, partial data loss in backups.
• Blast radius: every deployment whose conversations or messages exceed 100 documents.
• Backups report completed_with_warnings, which is easy to overlook. The job summary shows the affected containers missing entirely rather than showing a zero-record export.
• Reproduced on both a full backup (data_management_full_20260817T0300Z) and a partial backup (data_management_partial_20260817T0300Z) within the same night.

Observed behavior

App Service console logs:

[DATA_MANAGEMENT] Cosmos backup source page read failed. -- {'container': 'conversations', 'status_code': 400,
 'error': '(BadRequest) Invalid Continuation Token ActivityId: d009399c-468b-44b2-94a8-8c4515e0cf79,
 Microsoft.Azure.Documents.Common/2.14.0'}

[DATA_MANAGEMENT] Cosmos backup resource failed. -- {'job_id': 'data_management_partial_20260817T0300Z',
 'resource': 'personal_conversations', 'error': '(BadRequest) Invalid Continuation Token ...'}

The 100-document boundary matches DATA_MANAGEMENT_BACKUP_MANIFEST_BATCH_SIZE exactly:

Every container that fit in a single page succeeded. Only the two containers requiring a second page failed. 400 is non-retryable in _is_retryable_backup_cosmos_error, so the failure is immediate — job telemetry reports Retries / throttles: 0 / 0.

Root cause

_iter_backup_cosmos_source_items in application/single_app/functions_data_management.py rebuilt the query for every page and replayed the previous pager's continuation token:

continuation_token = None
while True:
    source_iterable = container.query_items(          # new query object per page
        query="SELECT * FROM c ORDER BY c.id",
        enable_cross_partition_query=True,
        max_item_count=DATA_MANAGEMENT_BACKUP_MANIFEST_BATCH_SIZE,  # 100
        ...)
    page_iterator = source_iterable.by_page(continuation_token=continuation_token)
    ...
    continuation_token = getattr(page_iterator, "continuation_token", None)

A cross-partition query is merged client side by the Cosmos SDK. The continuation token surfaced by the pager belongs to the underlying per-partition request and is not valid to resume the merged query. Rebuilding query_items(...) each iteration also discards the SDK's cross-partition execution context, so the merge state cannot be restored regardless of the token.

Page 1 therefore always succeeds; page 2 always fails with BadRequest: Invalid Continuation Token.

Secondary problem: the failure was effectively invisible

While diagnosing this, two observability gaps made the investigation far harder than it should have been:

1. Per-item source blob failures are never logged. _run_backup_source_blob_transfer converts every exception into a failure_summary string that is written only to the backup state document (last 50 entries) and the encrypted manifest. A run with 19,978 failed blobs produced zero log lines. A 47 MB console log export covering the full backup window contained 72,465 [DEBUG][Log] lines and not one containing "backup".
2. Application Insights carries no message text. _build_logger_extra in functions_appinsights.py reduced every string property to a length (job_id -> sc_job_id_length) and the emitted trace message was always the constant [SIMPLE_CHAT_LOG_EVENT]. App Insights could not identify which event fired, let alone why.

Expected behavior

• Cosmos backup source reads stream every document in a container regardless of size.
• Containers larger than one page appear in the backup artifact set with a full record count.
• Source blob transfer failures produce bounded, actionable log output.
• Application Insights traces carry enough sanitized context to diagnose a backup failure without a console log export.

Proposed fix

• Drain a single pager to exhaustion instead of rebuilding the query and replaying continuation tokens. Retries operate on the same pager so the cross-partition execution context is preserved.
• Log the first source blob failure per resource plus a bounded rollup of distinct failure reasons with counts when the resource finishes, so 19k failures do not become 19k log lines.
• Retain the sanitized message text and an allowlist of non-sensitive diagnostic keys (job_id, resource, container, error, status_code, ...) in App Insights properties. Sensitive keys continue to collapse to a presence flag and secret redaction is unchanged.

Validation

New functional test functional_tests/test_data_management_backup_cosmos_pagination.py covers:

• multi-page streaming where the fake pager raises 400 Invalid Continuation Token if any token is replayed, and asserts the query is built exactly once
• cutoff filtering and the non-paged test-double fallback
• bounded failure-reason rollup ordering and overflow aggregation
• App Insights extras retaining message text and allowlisted diagnostics while still redacting secrets

Environment

• SimpleChat 0.250.208
• Azure Cosmos DB (container-level autoscale), azure-cosmos==4.9.0
• Azure App Service, 3 instances