Report Cluster Ids in Kafka Metrics

[kanderson250]

Resolves #2984

This fulfills a spec change request to add per-cluster produce/consume metrics to the agent.

Details

According to the spec change, we need to add optional capture of per-cluster, per-topic metrics for Kafka clients. These are meant to exist as a supplement to the existing per-node, per-topic client metrics and will be used by the backend to build entity relationships.

• Existing Node Topic Metric: MessageBroker/Kafka/Node/{server_address}/{Produce | Consume}/{topic_name}
• New Cluster Topic Metric: MessageBroker/Kafka/Cluster/{cluster_id}/{Produce | Consume}/{topic_name}

What's changed

The cluster topic metric has been added to all eight kafka clients metrics modules that currently report the analogous Node Topic Metric.

The implementation is slightly different between kafka-clients-metrics and kafka-clients-node-metrics, but it is exactly the same across different versions of the same module (I had to backport it to all versions due to current adoption rates for earlier modules being high). It follows the implementation of the analagous NodeTopicRegistry as closely as possible.

For all versions, the bulk of the logic for fetching the cluster id and building the metric names lives in ClusterTopicRegistry.

Important requirements

These changes MUST:

• Gate the capture of the metric behind a configuration kafka.metrics.cluster.metrics.enabled which defaults to false.
• Not report anything if a clusterId is unavailable.
• After successful fetch of the clusterId, continue to reuse it for the lifetime of the client.
• Capture the cluster metric in addition to (not instead of) the node topic metric.
• Report the cluster topic metric at the same frequency as the node topic metric (when available).

Background (why this change was necessary)

Prior to this change, entity relationships were based on the combination of server address + topic from the Node Topic Metric. However, for self-hosted Kafka instances utilizing VPCs, it is possible for separate clusters to report the same broker address AND same topic name, in which case their identities will be conflated.

This change allows a new relationship to be formed based on clusterId + topic. Since cluster ids are always unique to the cluster, this prevents NR from mixing up identities.

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 70.93%. Comparing base (11af35c) to head (a96b22d).
⚠️ Report is 45 commits behind head on main.

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #3054      +/-   ##
============================================
+ Coverage     70.92%   70.93%   +0.01%     
- Complexity    10681    10683       +2     
============================================
  Files           871      871              
  Lines         43006    43006              
  Branches       6503     6503              
============================================
+ Hits          30501    30508       +7     
+ Misses         9557     9549       -8     
- Partials       2948     2949       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.