Summary
Since the refactor in c6e4606 ("Add capacity management for target distribution in
ClusterReconciler"), any Cluster whose Pipelines have zero gNMI targets — i.e.
the "Centralized Processing" / relay pattern documented in
docs/content/docs/user-guide/input.md (Input: Kafka → Output: Prometheus, no
Target CRs at all) — never receives its Inputs/Outputs config. The reconciler
logs a false success ("successfully applied config to gNMIc cluster"), so there's no
visible error; the gNMIc pods simply sit idle forever with an empty running config.
Environment
- Operator image:
ghcr.io/gnmic/operator:latest (also reproduced by building
current main, commit past v0.3.0)
- Confirmed working correctly on
v0.2.0
- gNMIc version in pods: 0.46.0
Steps to reproduce
- Deploy a
Cluster with no Target CRs, and a Pipeline referencing only an
Input (e.g. type: kafka) and an Output (e.g. type: prometheus_write) —
exactly the pattern shown in docs/content/docs/user-guide/input.md under
"Centralized Processing".
- Wait for the
Cluster/Pipeline to report Ready.
- Query the running gNMIc pod's own API:
GET https://<pod>:7890/api/v1/config —
inputs and outputs are both {}.
- Check the Kafka consumer group for the Input's
group-id — it never gets created.
- Operator logs still show
"successfully applied config to gNMIc cluster" for
every reconcile of this Cluster.
Root cause
In internal/gnmic/placement_blrh.go:
func boundedLoadRendezvousHash(targets map[string]*gapi.TargetConfig, options *PlacementStrategyOpts) Assignment {
numTargets := len(targets)
if numTargets == 0 {
return make(Assignment) // empty assignment when there are no targets
}
...
In internal/gnmic/distribute.go, DistributeTargets builds its PerPodPlans map by
iterating that (now-empty) assignment:
newAssignment := placement.distributeTargets(plan.Targets, placementOptions)
result := make(map[int]*ApplyPlan)
for podIndex, targets := range newAssignment { // never executes when targets == 0
result[podIndex] = &ApplyPlan{ Outputs: plan.Outputs, Inputs: plan.Inputs, ... }
}
So PerPodPlans ends up with no entry for any pod when a Cluster has zero
targets — even though plan.Outputs/plan.Inputs are populated correctly upstream
in plan.go.
In internal/controller/cluster_controller.go, applyConfigToPods then skips every
pod silently:
for podIndex := 0; podIndex < numPods; podIndex++ {
podPlan, ok := distResult.PerPodPlans[podIndex]
if !ok {
continue // hit for every pod; POST to /api/v1/config/apply is never sent
}
...
}
...
return unassigned, nil // no error — reconciler logs a false "success"
Prior to c6e4606, DistributeTargets was called once per podIndex and always
returned a valid *ApplyPlan with Outputs/Inputs copied in regardless of target
count — only Targets itself was conditionally empty. The refactor to a
capacity-aware, map-returning API changed that guarantee as a side effect, and no
test in distribute_test.go or placement_blrh_test.go covers the zero-target case
(every test seeds at least one target).
Suggested fix
In DistributeTargets, build result[podIndex] for every podIndex in
[0, numPods) up front (with Outputs/Inputs/Subscriptions/Processors
attached), and only use newAssignment to populate Targets per pod — mirroring
the pre-c6e4606 behavior. Alternatively, applyConfigToPods could special-case
numTargets == 0 to still push the shared plan to every pod once.
Impact
Breaks the documented "Centralized Processing"/relay pattern (Kafka/NATS Input →
any Output, no gNMI targets) for any user upgrading past v0.2.0. No error surfaces
anywhere — the Cluster/Pipeline both report Ready, and the only symptom is silent
data loss downstream.
Summary
Since the refactor in c6e4606 ("Add capacity management for target distribution in
ClusterReconciler"), any
Clusterwhose Pipelines have zero gNMI targets — i.e.the "Centralized Processing" / relay pattern documented in
docs/content/docs/user-guide/input.md(Input: Kafka → Output: Prometheus, noTargetCRs at all) — never receives itsInputs/Outputsconfig. The reconcilerlogs a false success ("successfully applied config to gNMIc cluster"), so there's no
visible error; the gNMIc pods simply sit idle forever with an empty running config.
Environment
ghcr.io/gnmic/operator:latest(also reproduced by buildingcurrent
main, commit pastv0.3.0)v0.2.0Steps to reproduce
Clusterwith noTargetCRs, and aPipelinereferencing only anInput(e.g.type: kafka) and anOutput(e.g.type: prometheus_write) —exactly the pattern shown in
docs/content/docs/user-guide/input.mdunder"Centralized Processing".
Cluster/Pipelineto reportReady.GET https://<pod>:7890/api/v1/config—inputsandoutputsare both{}.group-id— it never gets created."successfully applied config to gNMIc cluster"forevery reconcile of this Cluster.
Root cause
In
internal/gnmic/placement_blrh.go:Prior to c6e4606, DistributeTargets was called once per podIndex and always
returned a valid *ApplyPlan with Outputs/Inputs copied in regardless of target
count — only Targets itself was conditionally empty. The refactor to a
capacity-aware, map-returning API changed that guarantee as a side effect, and no
test in distribute_test.go or placement_blrh_test.go covers the zero-target case
(every test seeds at least one target).
Suggested fix
In DistributeTargets, build result[podIndex] for every podIndex in
[0, numPods) up front (with Outputs/Inputs/Subscriptions/Processors
attached), and only use newAssignment to populate Targets per pod — mirroring
the pre-c6e4606 behavior. Alternatively, applyConfigToPods could special-case
numTargets == 0 to still push the shared plan to every pod once.
Impact
Breaks the documented "Centralized Processing"/relay pattern (Kafka/NATS Input →
any Output, no gNMI targets) for any user upgrading past v0.2.0. No error surfaces
anywhere — the Cluster/Pipeline both report Ready, and the only symptom is silent
data loss downstream.