Accelerate source blob backups with high-throughput resumable transfers #1095

Summary

Accelerate Data Management source-blob backups with high-throughput bounded transfers, durable per-file checkpoints, retry/skip behavior, and an evidence-based choice among AzCopy, Azure Blob server-side copy, and Azure SDK block transfer.

User Value

Large Enhanced Citations storage sets should finish within a practical backup window. The current implementation downloads each complete blob with readall() and uploads it serially, so tens of gigabytes can take hours and consume unnecessary App Service memory.

Proposed Behavior

• Benchmark AzCopy, Blob server-side copy, and bounded Azure SDK block upload for the supported authentication, deployment, source/target account, and encryption configurations.
• Document and implement the fastest safe transport per mode. AzCopy or server-side copy may be used when compatible; encrypted artifacts must retain the backup encryption contract through a bounded streaming fallback rather than silently disabling encryption.
• Transfer blobs concurrently within configurable service and memory bounds, with chunked streaming or staged blocks when bytes must pass through the app.
• Persist a durable per-file checkpoint containing container/blob identity, source ETag/Last-Modified/length, target artifact identity, bytes, status, retries, and latest-only backup state.
• On retry/resume, validate source version and target checkpoint before skipping completed work; changed source blobs follow the stable backup cutoff policy.
• Handle 408/409/412/429/5xx and transport timeouts with bounded retry/backoff, honor service guidance, and adapt concurrency under throttling.
• Skip independent unreadable or failed files after the retry budget, continue the backup, and record sanitized failure detail for focused retry. Fail only when manifest integrity cannot be established.
• Report per-container and aggregate blob count, bytes, completed/skipped/failed files, retries, elapsed time, bytes/sec, current file, and last durable checkpoint.
• Never persist or return account keys, connection strings, SAS URLs, source contents, or raw sensitive exception data.

Acceptance Criteria

• A documented benchmark compares AzCopy, server-side copy, and SDK transfer under representative SimpleChat constraints and records the chosen path for encrypted and unencrypted backups.
• Backup transfer uses bounded concurrency and bounded memory; no file requires whole-blob readall() buffering.
• A representative 50 GB test set demonstrates a material elapsed-time improvement over the serial baseline, with the achieved throughput and environment documented rather than treated as a universal guarantee.
• Every file reaches a durable completed, skipped, or failed checkpoint with source-version and target-artifact identity sufficient for safe resume.
• 429/5xx/timeouts use bounded adaptive retry, and one failed file does not abort independent file transfers.
• Cancellation, worker restart, and Retry/Resume reuse verified files without duplicate target artifacts or silent omission.
• Existing metadata/content-type requirements and optional backup encryption remain correct for every supported transfer path.
• Progress and completion reports expose counts, bytes, rates, retries, and skip/failure summaries without secrets.
• Focused tests cover chunking, concurrency limits, encryption, source mutation, checkpoint resume, throttling, skip behavior, cancellation, and secret safety.

Notes

• The current _copy_source_blob() path downloads each blob with readall() and _export_source_blob_artifacts() loops serially.
• AzCopy is a required evaluation candidate because operational testing has moved roughly 50 GB in minutes, but bundling or invoking it from App Service must be justified for all supported platforms and identities.
• Depends on Make Data Management backup jobs durable, cancellable, and overlap-safe #1092. Supports restore capability in Add Data Management backup restore capability #1091.