Resolve policy per pod and make it part of profile identity (#87)

[nickmarden]

Closes #87.

The defect

policy.Resolver.Resolve served two callers with incompatible contracts. The metrics collector and resource adjuster both called it with only a WorkloadProfile's identity tuple:

resolved, err := r.resolver.Resolve(ctx, policy.Input{Labels: profile.Status.TupleLabels})

A WorkloadProfile is cluster-scoped and pools pods across namespaces, so that Input has no namespace, no owner kind, and no annotations. Two silent failures followed from one cause:

Over-reach. client.InNamespace("") means all namespaces, and Resolve ranks namespace-scoped above cluster-scoped before it compares priority. One ResourcePolicy with a loose selector therefore governed recommendations and resizes for every matching profile in the cluster, outranking policies that explicitly set a higher priority.

Split resolution. A policy selecting on kinds, annotations, or namespaces.include matched at admission (which has the pod) but could never match in the controllers. Pods were admitted with one policy's values and then resized toward another's, with nothing logging a conflict because each path resolved successfully on its own terms.

The fix

Policy is resolved once per pod, by the workloadwatcher — the only component holding a pod, and therefore the only one that can evaluate a selector correctly. The result is recorded on workloadprofile.status.policyRef; the collector and adjuster Load it instead of resolving again, so all three paths agree by construction. A Resolver Input with no namespace can no longer match a ResourcePolicy at all.

The governing policy is now part of profile identity. A profile holds one set of recommendations per container, and the policy chooses the metrics sources, poll cadence, tracked resources, aggregation, and headroom that produce them, so pods resolving to different policies cannot share a profile. Two keys that were one are now separate:

• Profile identity — name plus status.policyRef. Names carry a policy token: checkout--server--fleet-a1b2c3d4.
• Measurement namespace — status.measurementHash, covering tuple and policy.

Per-policy measurement namespaces (rather than a shared series) mean sibling profiles never RPUSH into one list. Samples carry no per-sample timestamps, so shared writes would inflate counts, distort the distribution, and halve the effective retention window. It also lets the finalizer purge its own keys with no reference counting.

The policy is deliberately not folded into tupleLabels. That map also produces status.selectorLabels, the server-side label query that finds a profile's pods, and a policy name is not a pod label — folding it in would rename profiles without changing which pods they measure.

Runtime policy changes

Nothing watched either policy kind before. That was survivable only because resolution was never cached; caching it on the profile makes a watch mandatory. Both kinds are now watched, so a policy applied to a running cluster takes effect in seconds.

Only selector and priority changes re-scope identity. Every other spec field (aggregation, headroom, thresholds, sources, cadence) is read live on the next collection or resize cycle, so tuning a policy does not re-key measurement history: the sample already recorded does not change meaning because the headroom applied to it did.

The fan-out is deliberately indiscriminate (every enrolled pod). The affected set is not "the pods this policy matches" — on a delete the matching spec no longer exists, on a narrowed selector the affected pods are the ones that stopped matching, and a new high-priority policy flips pods that previously matched nothing. Each reconcile is a cache read plus a resolve and writes nothing unless identity actually changed.

Traceability

status.profileDiscriminator on both policy kinds, with a DISCRIMINATOR print column, so profile names trace back to their policy without computing a hash by hand:

kubectl get workloadprofiles | grep "$(kubectl get crp fleet \
  -o jsonpath='{.status.profileDiscriminator}')"

The token hashes kind + namespace + name, not the name alone: two ResourcePolicies named defaults in different namespaces are different policies, and a name-only hash would give them one token and make two distinct profiles contend for one object name. It is published on status rather than as a label because it is operator-derived data about a user-owned object.

workloadprofiles also gains a POLICY column, and the workloadwatcher now refreshes the pod's policy-ref annotation, which previously kept advertising whatever admission resolved.

Upgrade impact

Every existing WorkloadProfile is replaced by a newly-named one, and the fleet accrues from zero for one readiness.minTimeSpan (24h by default) before resizes resume. No resizes are issued in that window and admission applies nothing, so pods keep the requests they were created with. Old profiles orphan, age out over orphanTTL (168h), and their Redis history is purged by the finalizer as they go — the finalizer falls back to the bare tuple hash for profiles that predate measurementHash, so nothing is stranded. Nothing needs doing by hand.

Notes on scope

The issue proposed moving Resolve into the resource adjuster's per-pod loop. That is now unnecessary: pods in a profile share a policy by construction, so reading the profile's policyRef is exactly per-pod resolution, done once.

The issue's "annotations cannot scope measurement" consequence is now documented on the PolicySelector CRD type rather than left implicit.

Verification

• The issue's reproduction is included as TestResolve_NoNamespace_ExcludesResourcePolicies (three cases, including the higher-priority ResourcePolicy), and passes.
• New coverage for profile splitting across policies, runtime policy apply/remove migration, sibling-purge isolation, the legacy purge path, the watch predicate (including that a headroom-only edit is not admitted), and the discriminator controller end to end under envtest.
• make lint — 0 issues. make test-coverage-check — passes at 79.1%. make helm-lint — passes.

[github-actions]

Docker Image Built

docker pull ghcr.io/tight-line/ballast:pr-88-f2464d7

Helm (values override)

image:
  repository: ghcr.io/tight-line/ballast
  tag: "pr-88-f2464d7"

Image expires ~15 days after PR closes.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!