Skip to content

DistributeTargets silently drops Inputs/Outputs for zero-target Clusters (relay/processing pattern broken since v0.3.0) #114

Description

@EwaldvanGeffen

Summary

Since the refactor in c6e4606 ("Add capacity management for target distribution in
ClusterReconciler"), any Cluster whose Pipelines have zero gNMI targets — i.e.
the "Centralized Processing" / relay pattern documented in
docs/content/docs/user-guide/input.md (Input: Kafka → Output: Prometheus, no
Target CRs at all) — never receives its Inputs/Outputs config. The reconciler
logs a false success ("successfully applied config to gNMIc cluster"), so there's no
visible error; the gNMIc pods simply sit idle forever with an empty running config.

Environment

  • Operator image: ghcr.io/gnmic/operator:latest (also reproduced by building
    current main, commit past v0.3.0)
  • Confirmed working correctly on v0.2.0
  • gNMIc version in pods: 0.46.0

Steps to reproduce

  1. Deploy a Cluster with no Target CRs, and a Pipeline referencing only an
    Input (e.g. type: kafka) and an Output (e.g. type: prometheus_write) —
    exactly the pattern shown in docs/content/docs/user-guide/input.md under
    "Centralized Processing".
  2. Wait for the Cluster/Pipeline to report Ready.
  3. Query the running gNMIc pod's own API: GET https://<pod>:7890/api/v1/config
    inputs and outputs are both {}.
  4. Check the Kafka consumer group for the Input's group-id — it never gets created.
  5. Operator logs still show "successfully applied config to gNMIc cluster" for
    every reconcile of this Cluster.

Root cause

In internal/gnmic/placement_blrh.go:

func boundedLoadRendezvousHash(targets map[string]*gapi.TargetConfig, options *PlacementStrategyOpts) Assignment {
	numTargets := len(targets)
	if numTargets == 0 {
		return make(Assignment)   // empty assignment when there are no targets
	}
	...
In internal/gnmic/distribute.go, DistributeTargets builds its PerPodPlans map by
iterating that (now-empty) assignment:


newAssignment := placement.distributeTargets(plan.Targets, placementOptions)
result := make(map[int]*ApplyPlan)
for podIndex, targets := range newAssignment {   // never executes when targets == 0
    result[podIndex] = &ApplyPlan{ Outputs: plan.Outputs, Inputs: plan.Inputs, ... }
}
So PerPodPlans ends up with no entry for any pod when a Cluster has zero
targetseven though plan.Outputs/plan.Inputs are populated correctly upstream
in plan.go.

In internal/controller/cluster_controller.go, applyConfigToPods then skips every
pod silently:


for podIndex := 0; podIndex < numPods; podIndex++ {
    podPlan, ok := distResult.PerPodPlans[podIndex]
    if !ok {
        continue   // hit for every pod; POST to /api/v1/config/apply is never sent
    }
    ...
}
...
return unassigned, nil   // no error — reconciler logs a false "success"

Prior to c6e4606, DistributeTargets was called once per podIndex and always
returned a valid *ApplyPlan with Outputs/Inputs copied in regardless of target
count — only Targets itself was conditionally empty. The refactor to a
capacity-aware, map-returning API changed that guarantee as a side effect, and no
test in distribute_test.go or placement_blrh_test.go covers the zero-target case
(every test seeds at least one target).

Suggested fix

In DistributeTargets, build result[podIndex] for every podIndex in
[0, numPods) up front (with Outputs/Inputs/Subscriptions/Processors
attached), and only use newAssignment to populate Targets per pod — mirroring
the pre-c6e4606 behavior. Alternatively, applyConfigToPods could special-case
numTargets == 0 to still push the shared plan to every pod once.

Impact

Breaks the documented "Centralized Processing"/relay pattern (Kafka/NATS Input →
any Output, no gNMI targets) for any user upgrading past v0.2.0. No error surfaces
anywhere — the Cluster/Pipeline both report Ready, and the only symptom is silent
data loss downstream.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions