Evaluate Pod checkpoint/restore for warm scale-from-zero #3420

🤖 Generated by the Agentic Engineer

Parent: #3418

Objective

Evaluate Kubernetes Pod-level checkpoint/restore as a future warm-start accelerator for genuinely slow-initializing Platform workloads. Record an adopt/defer/reject decision; do not treat Kubernetes 1.37 Alpha support as a production commitment.

Why this may matter

KEP-5823 names warm-starting slow services as the primary Alpha use case. A checkpoint can preserve process memory, file descriptors, writable runtime state, and initialized application state, potentially reducing restore time for JVM, model-serving, or similarly expensive startup paths.

It does not currently integrate with the production scale-to-zero path:

• Alpha restore is node-local and same-node only.
• Alpha leaves the source Pod running; stop-and-release semantics are not implemented.
• HPA and Deployment controllers do not automatically create restored Pods.
• spec.restoreFrom is intended for directly created, one-shot Pods; restoring one captured memory image into multiple replicas is unsupported.
• Shared volumes/devices, checkpoint transport, lifecycle/retention safety, and application refresh hooks are incomplete or deferred.
• Kubelet support is insufficient unless the Talos-provided CRI runtime implements CheckpointPod and RestorePod.

Strong-fit gate

Proceed to an implementation follow-up only if all of these become true:

• A measured Platform workload has initialization expensive enough that response caching, image caching, startup work, and request holding do not meet its cold-start budget.
• The then-current Kubernetes feature is production-ready enough for the proposed scope, with compatible released Talos and CRI runtime support proven end to end.
• Checkpoint storage works across the node lifecycle required by min-zero autoscaler pools, or the use case is deliberately and safely node-pinned.
• HPA/workload-controller integration has a supported design; no bespoke controller is accepted merely to make Alpha usable.
• Applications can refresh stale database connections, DNS state, tokens, certificates, randomness, and other captured state correctly after restore.
• Checkpoint artifacts have Secret-equivalent encryption, authorization, retention, garbage collection, disk-pressure protection, and auditability.
• Restore beats an image-cached ordinary cold start by a material amount under the same user-path test.

If any gate fails, record the named missing capability and defer or reject the approach.

Evaluation tasks

• Refresh Kubernetes/KEP, Talos, containerd/CRI-O, CRIU/runtime, and controller integration support at investigation time; do not rely on the Alpha limitations recorded here as permanent facts.
• Prove whether the deployed Talos/containerd combination exposes both required CRI RPCs; absence is QUERY-UNKNOWN until a direct compatibility or runtime test resolves it.
• Use a synthetic non-sensitive slow-start workload in an isolated non-production environment—never a checkpoint containing real wedding, authentication, database, or production secret state.
• Benchmark ordinary uncached start, image-cached start, and checkpoint restore through readiness and first response.
• Exercise node restart/deletion, checkpoint loss, incompatible runtime/kernel versions, artifact cleanup, rollback, and failed restore behavior.
• Record security and correctness risks, operational overhead, and the exact applications—if any—that justify the mechanism.
• Finish with an explicit decision and create typed implementation issues only if the strong-fit gate passes.

Out of scope

• Production rollout from this spike.
• Using checkpoint/restore to compensate for avoidable application startup work.
• Restoring one captured image as a generic Deployment replica factory.
• Cross-node or live migration claims not supported by the then-current feature.

Acceptance Criteria

• The decision cites current upstream versions and direct runtime evidence.
• Normal cold start, image-cached cold start, and restore have comparable numbers.
• Security, lifecycle, scheduler, autoscaler, controller, and application-state implications are explicitly resolved or named as blockers.
• The outcome is adopt through follow-up issues, defer until named upstream triggers, or reject for the Platform workload profile.

References

• Kubernetes 1.37 Pod checkpoint/restore announcement
• KEP-5823: Pod-level Checkpoint/Restore

Generated by Claude Code