collect starrocks_fe_slow_lock_{held,wait}_time_ms metrics

[steveny91]

What does this PR do?

Two changes to the CelerData FE metrics, reviving #3021 and #3025 by @jaogoy.

Collect every per-database celerdata.fe.table_num series. StarRocks FE emits table_num interleaved with db_size_bytes, one pair per database, and writes a # TYPE line only for the first sample. The Prometheus parser types every later family unknown, and OpenMetrics V2 drops those, so only the first database was collected. Pinning the metric type to gauge routes every family through the gauge transformer.

Add celerdata.fe.slow_lock_held_time_ms and celerdata.fe.slow_lock_wait_time_ms (StarRocks/starrocks#66027, StarRocks 3.5.10+) as .quantile (gauge) and .count (monotonic count). .sum is intentionally not submitted. StarRocks computes _sum as getCount() * snapshot.getMean(): a cumulative count times the mean of a 1-minute sliding window. That isn't a sum, and it isn't monotonic. It drops to 0 after any quiet minute, and the next slow lock makes it jump to (slow locks since FE start) × (that lock's duration), which the Agent records as a single increment. METRIC_MAP can't drop one part of a summary, so CelerdataCheck registers a small custom transformer for these two metrics.

Motivation

Closes #2854, which reported two symptoms:

• table_num only reporting information_schema: fixed above.
• celerdata.fe.query.timeout.count missing: this doesn't reproduce. The metric has been mapped since the first release, the interleaving doesn't affect it, and the Agent forwards it even when the value is 0. A new unit test pins that it's collected. If it's still missing on 1.3.0, please reopen with the output of datadog-agent check celerdata.

[dd-octo-sts]

Validation Report

All 11 validations passed.

Show details

Validation	Description	Status
ci	Validate CI configuration and code coverage settings	✅
codeowners	Validate every integration has a CODEOWNERS entry	✅
config	Validate default configuration files against spec.yaml	✅
imports	Validate check imports do not use deprecated modules	✅
integration-style	Validate check code style conventions	✅
jmx-metrics	Validate JMX metrics definition files and config	✅
legacy-signature	Validate no integration uses the legacy Agent check signature	✅
metadata	Validate metadata.csv metric definitions	✅
models	Validate configuration data models match spec.yaml	✅
package	Validate Python package metadata and naming	✅
readmes	Validate README files have required sections	✅

View full run

[datadog-datadog-us1-prod]

🎯 Code Coverage (details)
• Patch Coverage: 98.04%
• Overall Coverage: 91.21% (+5.49%)

_{This comment will be updated automatically if new data arrives.
🔗 Commit SHA: b60743a | Docs | Give us feedback!}

[iliakur]

The .sum metrics are submitted as monotonic counts, but StarRocks FE derives _sum from a 1-minute sliding window, so the value can decrease (StarRocks/starrocks#66027, PrometheusMetricVisitor.java:247-248).

Scrape 1 reports starrocks_fe_slow_lock_held_time_ms_sum 91230.0 with _count 137; 60 seconds later, scrape 2 reports _sum 8000.0 with _count 145. Running the check against these payloads submits both values as monotonic_count for celerdata.fe.slow_lock_held_time_ms.sum. The agent reads 8000.0 < 91230.0 as a counter reset: a bogus rate spike, then a re-baselined series.

Route _sum to a submission type matching its trajectory (for example a gauge), or drop .sum and keep the windowed quantiles plus .count. If the monotonic mapping is intentional, add a two-scrape test that pins the behavior.

[steveny91]

Agreed that it isn't monotonic, and it's worse than an occasional decrease. I went with your second option: .sum is dropped, and the quantiles and .count are kept.

Maybe as a compromise we ship a gauge with a different name?

[iliakur]

The table_num test checks only that each db_name tag appears, so it can't catch a mapping that submits a wrong value for a database or repeats the first family's value for all of them.

If the transformer submitted 54 (the information_schema value) for all four databases, the current assert_metric_has_tag loop still passes. The fixture values are information_schema=54, _statistics_=10, sys=8, analytics=19.

Add value assertions next to the tag assertions, e.g. aggregator.assert_metric('celerdata.fe.table_num', value=10, tags=['db_name:_statistics_']), so family-to-value association is pinned as well.

[steveny91]

The test now builds {db_name: value} from the submitted series and compares it to the fixture values in one assertion. That catches the case you described (54 repeated for every database), any other wrong value, and a missing database:

[iliakur]

Issue #2854 also reports celerdata.fe.query.timeout.count never appearing in Datadog, and nothing in this PR affects that metric.

The issue describes two symptoms: missing query.timeout.count and table_num limited to information_schema; the PR fixes only the second. A reader linking the issue will assume both are resolved.

State in the PR description whether the query.timeout.count symptom is out of scope or tracked in a separate fix, so #2854 isn't closed as fully resolved.

[steveny91]

I looked into the query.timeout.count half: it doesn't reproduce, so there's no code fix for it. The metric has been in METRIC_MAP since the first release, its family isn't affected by the interleaving, and the Agent forwards it even when the value is 0. The new test_query_timeout_given_zero_valued_counter_returns_the_count_metric pins that it's collected, and the description now explains both symptoms.