Make Data Management backup jobs durable, cancellable, and overlap-safe #1092

Summary

Make full and partial Data Management backup jobs durable, cancellable, recoverable, and mutually exclusive, with a latest-only item state contract that supports reliable differential backup and restore workflows.

User Value

Administrators need long-running backups to survive worker restarts, avoid overlapping scheduled/manual runs, expose useful progress, and stop safely on request. They also need to know the latest backup disposition for each Cosmos record, AI Search document, and source blob without retaining duplicate per-item history outside the existing job history.

Proposed Behavior

• Persist an immutable normalized backup plan, source cutoff, attempt/lease generation, phase state, per-resource checkpoints, and bounded per-item or per-batch checkpoints.
• Coordinate backup execution with a source-scoped lock so full and partial jobs cannot overlap. Scheduled occurrences should be deferred or recorded as skipped while an active backup owns the scope, rather than starting another worker.
• Add admin-only cancellation for queued and running backups. Queued jobs cancel immediately; running workers stop at durable boundaries and cannot write after terminal cancellation.
• Recover queued or stale backup jobs after restart or deployment and expose Retry/Resume for incomplete work without replaying verified units.
• Define latest-only backup item state for Cosmos records, AI Search documents, and source blobs. Track source identity/version, job ID, artifact/checkpoint identity, timestamp, status, and failure/skip summary while leaving attempt history in job records.
• Store latest-item state in a sidecar/manifest design that does not mutate source records in a way that advances _ts, ETag, or Last-Modified and makes every item appear changed.
• Make full, partial, and differential semantics explicit, including a stable source cutoff and non-destructive default deletion policy.
• Treat item failures as reportable warnings where safe: continue independent work, record skipped/failed items, complete with warnings, and permit focused retry.
• Define a shared progress contract for counts, bytes, RU, retries, skip/failure counts, rates, heartbeat, lease, current resource, and last successful checkpoint.

Acceptance Criteria

• At most one active full or partial backup owns the configured source scope across manual requests, scheduler ticks, workers, and App Service instances.
• Queued and running backups can be canceled from authenticated admin API/UI controls, and no artifact or state writes occur after a job reaches canceled.
• Failed, canceled, stale, and interrupted backups can resume from verified durable checkpoints after worker or App Service restart.
• Cosmos records, AI Search documents, and source blobs have a latest-only backup state contract that supports differential decisions without causing source-version churn.
• Full/partial/differential cutoffs and deletion semantics are stable, explicit, and represented in the backup manifest.
• Independent item failures are skipped and reported without aborting the whole backup unless manifest integrity or another safety invariant is violated.
• Job detail exposes bounded, sanitized progress and failed/skipped item summaries without settings, credentials, SAS URLs, or content.
• Functional, scheduler/recovery, concurrency/fencing, and authenticated UI tests cover cancellation, overlap prevention, resume, and latest-item state.
• Data Management backup and recovery documentation explains lifecycle states, differential semantics, and operator recovery.

Notes

• Reuse and generalize the durable state, provenance, fencing, cancellation, and recovery patterns developed under Make Data Management migrations resilient for large blob transfers with cancel, resume, and incremental sync #1043 rather than creating a second incompatible framework.
• This is the shared foundation for Accelerate Cosmos DB backups with bounded parallelism and temporary RU scaling #1093 (Cosmos), Accelerate exhaustive Azure AI Search backups with adaptive parallel export #1094 (AI Search), and Accelerate source blob backups with high-throughput resumable transfers #1095 (Blob), and for restore capability in Add Data Management backup restore capability #1091.