Fix Cosmos backup continuation token failure dropping large containers

[Paul Lizer (paullizer)]

Fixes #1258

Problem

Data Management backups silently omitted every Cosmos container holding more than one page of documents. Affected containers were recorded as failed resources and dropped from the backup artifact set entirely, while the job still reported completed_with_warnings.

In practice this meant personal_conversations and personal_messages — the two largest containers in most deployments — were never backed up. Reproduced on both a full and a partial backup on the same night.

[DATA_MANAGEMENT] Cosmos backup source page read failed. -- {'container': 'conversations', 'status_code': 400,
 'error': '(BadRequest) Invalid Continuation Token ActivityId: d009399c-468b-44b2-94a8-8c4515e0cf79,
 Microsoft.Azure.Documents.Common/2.14.0'}

The failure boundary matched DATA_MANAGEMENT_BACKUP_MANIFEST_BATCH_SIZE (100) exactly. Every container that fit in one page succeeded; only the two needing a second page failed.

Container	Documents	Pages	Result
public_documents	84	1	completed
personal_documents	59	1	completed
group_documents	39	1	completed
all other containers	<= 21	1	completed
personal_conversations	> 100	2+	failed
personal_messages	> 100	2+	failed

Root cause

_iter_backup_cosmos_source_items rebuilt the query for every page and replayed the previous pager's continuation token:

continuation_token = None
while True:
    source_iterable = container.query_items(          # new query object per page
        query="SELECT * FROM c ORDER BY c.id",
        enable_cross_partition_query=True,
        max_item_count=DATA_MANAGEMENT_BACKUP_MANIFEST_BATCH_SIZE,
        ...)
    page_iterator = source_iterable.by_page(continuation_token=continuation_token)
    ...
    continuation_token = getattr(page_iterator, "continuation_token", None)

A cross-partition query is merged client side by the Cosmos SDK. The surfaced continuation token belongs to the underlying per-partition request and cannot resume the merged query, and rebuilding query_items(...) each iteration discarded the SDK's cross-partition execution context anyway. Page one always succeeded; page two always failed. 400 is non-retryable, so the resource failed immediately with zero retries.

Changes

1. Cosmos source pagination (functions_data_management.py)

The query is built once and a single pager is drained to exhaustion. Retries call next() on the same pager so the execution context is preserved, and no continuation token is ever passed to by_page(). Page normalization, client-side sorting, cutoff filtering, cancellation checks, and RU/retry telemetry are unchanged. The non-paged fallback used by lightweight test doubles is retained and shares the new page-finalization helper.

2. Source blob failure logging (functions_data_management.py)

Per-item blob failures previously produced no log output at all — a run with 19,978 failed blobs left zero trace across a 47 MB console log export containing 72,465 [DEBUG][Log] lines. Now logs the first failure per resource, logs enumeration failures, and emits one rollup at resource completion with counts and distinct reasons. Two helpers keep it bounded: _record_backup_failure_reason caps distinct reasons at 10 and aggregates the rest into an Other backup failures. bucket; _summarize_backup_failure_reasons renders a frequency-ordered string.

3. Application Insights message text (functions_appinsights.py)

Traces arrived as the constant [SIMPLE_CHAT_LOG_EVENT] with every string property reduced to a length (job_id -> sc_job_id_length), making App Insights useless for diagnosis. Now emits sc_message with the sanitized message plus the sanitized values of an explicit allowlist of non-sensitive diagnostic keys. Sensitive keys still collapse to a _present boolean, all other strings still reduce to a length only, and sanitize_log_message redaction is unchanged.

Validation

New regression test functional_tests/test_data_management_backup_cosmos_pagination.py — the fake pager raises 400 Invalid Continuation Token if any token is supplied, reproducing the production failure directly.

5 passed in 2.93s

Test	Assertion
test_multi_page_cosmos_source_drains_a_single_pager	300 docs across 3 pages stream fully; query built exactly once; by_page only ever receives None
test_cutoff_and_unpaged_fallback_still_stream	Cutoff drops exactly the out-of-range doc; non-paged fallback streams everything
test_failure_reason_rollup_is_bounded	Distinct reasons bounded, overflow aggregated, summary frequency-ordered
test_logger_extra_retains_sanitized_diagnostics	Message text and allowlisted diagnostics retained; secret value never emitted
test_version_is_at_least_fix_version	config.py version floor

Full Data Management regression suite (functional_tests/test_data_management_*.py): 147 passed, 1 failed.

The single failure — test_data_management_backup_durability.py::test_backup_recovery_and_admin_progress_are_bounded_and_sanitized — was confirmed pre-existing on origin/Development by stashing these changes and re-running it on the clean base. It is unrelated to this PR.

Security notes

• The App Insights allowlist is an explicit key set, not a fragment match. Sensitive-key detection runs first and short-circuits, so no allowlisted key can bypass redaction.
• All retained values pass through sanitize_log_message and are truncated (1024 chars for properties, 8192 for the message).
• Failure summaries already pass through _sanitize_data_management_backup_text before reaching logs.

Version

0.250.208 -> 0.250.209

Documentation

• docs/explanation/fixes/COSMOS_BACKUP_CONTINUATION_TOKEN_FIX.md
• docs/explanation/release_notes.md