Add opt-in HTTP scale-to-zero to the WebApp abstraction #3419

🤖 Generated by the Agentic Engineer

Parent: #3418

User Story

As a platform operator,
I want the KRO WebApp abstraction to offer request-buffered scale-to-zero as an explicit opt-in,
so that idle non-critical HTTP workloads can release replicas without failing the request that wakes them.

Evidence and constraints

• Kubernetes 1.37 enables HPA scale-to-zero by default for Object and External metrics; CPU and memory cannot activate a workload from zero.
• Platform already installs KEDA 2.20.2, but currently renders no ScaledObject, HTTP activation route, or zero-replica WebApp.
• The current WebApp RGD deliberately leaves KEDA HTTP scale-to-zero app-specific until it has been piloted.
• A native HPA or KEDA metric alone does not hold synchronous HTTP traffic while the first Pod becomes Ready. The request path needs an interceptor or an equivalently supported activation mechanism.
• GitOps must not continually reapply spec.replicas to an HPA-owned Deployment.

Proposed Solution

Refresh upstream support at implementation time, then add the smallest supported request-holding path—currently expected to be the KEDA HTTP Add-on—and expose it through a default-off WebApp field. Keep controller installation, activation resources, and application routing in the correct Flux dependency layers.

The opt-in shape should configure only stable user intent: enabled state, safe replica bounds, activation/cooldown policy, and the external/object request metric. Do not expose controller-internal details without a demonstrated tenant need.

Use whoami as the plumbing canary. A user-facing application must not opt in through this issue until its own cache, startup, and first-render gate is complete.

Acceptance Criteria

• The then-current Kubernetes, Talos, KEDA, KEDA HTTP activation, Cilium Gateway API, and WebApp RGD contracts are re-verified and source-linked before implementation.
• Scale-to-zero is disabled by default; render and behavior in the disabled state are unchanged, and automated tests cover both disabled and enabled states.
• The enabled WebApp renders the request-holding route, external/object activation metric, zero-capable scaling resource, readiness-safe backend path, and explicit rollback configuration.
• whoami reaches zero replicas when idle, and the first request is held until a Ready replica serves it with no 502/503 response.
• HPA/KEDA owns replica count without Flux or another apply path resetting spec.replicas; the ownership rule has a regression test.
• Cached-image and uncached-image 0 → Ready, first-request TTFB, and first-render timing are captured against a warm baseline.
• Cooldown is derived from observed request gaps and starts conservatively enough to avoid repeated cold starts.
• Coroot/OpenCost plus Cluster Autoscaler evidence distinguishes reclaimed pod allocation from an actually avoided or removed overflow node; no private billing values enter the public issue or PR.
• Databases, Gateway, DNS, identity, observability, Flux, KEDA, and other critical controllers are excluded by construction and documented.
• Static validation passes for local and production overlays, and activation is proven in a permitted non-production environment before any production workload opts in.
• Operator documentation covers eligibility, cache/readiness prerequisites, metrics, diagnostics, rollback, and ScaledToZero status.

Alternatives Considered

• Plain HPA on CPU/memory: cannot wake from zero.
• Native HPA without request holding: suitable for queued/event workloads, not a synchronous web request path.
• Keep one replica permanently: remains the rollback and the right choice for critical or cold-start-sensitive workloads.
• Checkpoint/restore: tracked separately as an alpha research path; it is not a prerequisite for this feature.

References

• Kubernetes HPA scale-to-zero
• KEDA scaling deployments
• KEDA HTTP Add-on scaling