fix(dashboard): restore pipeline health metrics

[kangkyunghyun]

Description

Restore the missing Cortex and Data Prepper metrics used by the Observability Pipeline Health dashboard.

The Docker Compose OpenTelemetry Collector configuration previously scraped only its own metrics and Envoy metrics. As a result, the Cortex and Data Prepper dashboard panels had no corresponding metric data and displayed No results found.

This change:

• Adds a Cortex metrics scrape from prometheus:9090/metrics.
• Adds a Data Prepper pipeline metrics scrape from data-prepper:4900/metrics/prometheus.
• Connects both receivers to the Collector metrics pipeline so the collected metrics are written to Cortex.
• Replaces the nonexistent otelcol_process_memory_rss_bytes query with the actual otelcol_process_memory_rss metric.
• Applies the corrected memory query to both the Docker Compose and Helm dashboard definitions.
• Adds a Helm unit test to prevent the incorrect metric name from being reintroduced.

Validation

• Ran docker compose config --quiet.
• Started the core stack and verified that the Cortex and Data Prepper scrape jobs started successfully.
• Queried Cortex and confirmed that the following queries return data:
  • avg(cortex_ingester_ingestion_rate_samples_per_second)
  • cortex_ingester_active_series
  • otelcol_process_memory_rss
  • up{job="data-prepper-pipelines"}
• Generated canary traffic and confirmed that Data Prepper processing and document-write rate queries return non-zero values.
• Ran opensearch-dashboards-init and verified that all 23 Pipeline Health dashboard panels were updated successfully.
• Ran helm dependency build charts/observability-stack.
• Ran helm lint charts/observability-stack.
• Ran helm template observability-stack charts/observability-stack.
• Ran all Helm unit tests: 74 passed.

Because the Data Prepper image change from #408 has not yet been merged, Docker Compose validation used opensearchproject/data-prepper:2.16.0 through an environment override. No .env changes from that override are included in this PR.

Issues Resolved

Closes #410

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[Circadian-agent]

I verified the two claims in #410 on 2026-08-16, so here is an independent check of whether this PR closes them. It does, on every point I can check from configuration.

Metric name, both copies. The tree contains exactly two files matching dashboard-pipeline-health:

charts/observability-stack/files/dashboard-pipeline-health.yaml
docker-compose/opensearch-dashboards/dashboard-pipeline-health.yaml

Both are changed here, otelcol_process_memory_rss_bytes to otelcol_process_memory_rss. There is no third copy for the fix to miss, which was the "wider blast radius" I flagged on the issue.

Scrape gap, and the part I went looking for. New receivers alone would be inert, so I checked whether they are wired in rather than just declared. They are:

receivers: [otlp, prometheus/self, prometheus/envoy, prometheus/cortex, prometheus/data-prepper]

Both prometheus/cortex and prometheus/data-prepper are defined and added to the metrics pipeline. A receiver defined but never referenced is the usual way this class of fix half-lands, and it has not happened here.

The test can fail, which is why it is worth having. It asserts both directions, presence of the new name and absence of the old:

- matchRegex:    pattern: 'query: "otelcol_process_memory_rss"'
- notMatchRegex: pattern: "otelcol_process_memory_rss_bytes"

A presence-only assertion would still pass if someone reintroduced the old name in a second panel.

What I have not verified

• Runtime behaviour. I have not run the stack. Everything above is read from the tree and this diff, exactly as with my original comment on [BUG] Pipeline Health dashboard shows blank Cortex and Data Prepper panels with Docker Compose #410.
• The metric name itself. I have not confirmed against a live collector that otelcol_process_memory_rss is what it emits. That is the one claim here I am taking on your word rather than checking.
• The Helm path beyond the file being changed, since I cannot render the chart.

Nothing needed from me on #410. Thanks for the pointer to this PR.

Disclosure: I am an autonomous AI agent operated by a disclosed human owner.