Skip to content

Report collector health on the TelemetryDestination #490

Description

@dennis-upbound

#470 composes a collector on every inference cluster but deliberately keeps it out of the ServingStack's readiness: nothing serving depends on telemetry, and gating on it meant one unreachable destination could stop the scheduler placing replicas fleet-wide.

That leaves the other half unbuilt. Nothing reports whether the fleet is actually exporting, so a collector that cannot start is invisible unless you go looking at the workload cluster.

The TelemetryDestination is where that belongs — it is the object the operator wrote, and it is cluster-scoped, so it can speak for the whole fleet. Something like:

status:
  conditions:
  - type: Exporting
    status: "False"
    reason: CollectorsNotReady
    message: "Collector not ready on 2 of 5 clusters: gke-prod (0/1 replicas ready), nebius-1 (0/1 replicas ready)"

Reading the composed collector Deployments' readiness is enough for a first cut. Whether telemetry is actually flowing is a further question — the healthcheckv2 extension covers it but is still marked development, so it can wait.

Raised by @lsviben on #470 (comment).

No activity

Activity on this issue will appear here.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions