[dbshah12]

Jira

DLPX-100030 — part of epic APIGW-12274

Design

Telegraf → Fluentd Metric Forwarding (Backward-Compat for DCT-Managed Engines) — kept up to date, source of truth for the full field matrix and the review-driven decisions.

Companion app-gate PR: delphix/dlpx-app-gate#5043 (fluentd HTTP source + record_transformer that nests data: {} and the Java passthrough)

Background

Adding a DE to DCT turns off Java analytics on the engine so DCT's own Grafana dashboards aren't fed from two collectors at once. One side effect of that analyticsManager.stop() is that pushInsightMetric — the long-running path that pushed six KPI metrics into Fluentd, from where customers' Splunk / Datadog / Dynatrace integrations picked them up — also stops. Those customers see nothing in their monitoring tools, with no obvious signal that anything changed.

Problem

Telegraf is already running on every engine and already collecting the raw inputs we need. What was missing: an HTTP output to Fluentd, a transform that reshapes the raw Telegraf metrics into the exact event shape customer dashboards were built against, and a way to flip the whole thing on and off from the setup screen.

Solution

The PR adds three files and modifies two.

telegraf/telegraf.outputs.fluentd — HTTP output pointed at localhost:9880. data_format = "splunkmetric" with splunkmetric_hec_routing = true gives us the HEC envelope that app-gate's record_transformer expects. use_batch_format = false is the subtle but important bit: the batched splunkmetric serializer concatenates JSON objects without a delimiter, and Fluentd's @type http JSON parser silently truncates everything after the first object — one POST per metric avoids that landmine entirely. 60s flush, 10s timeout, 2048 metric buffer so a short Fluentd blip doesn't lose data.

telegraf/telegraf.processors.fluentd — Starlark processor. Explicit order = 4 so it runs after the base-config storage_io processors (otherwise the name=total row from estat could land in our accumulators alongside the read + write rows and double-count NFS/iSCSI). Maps six Telegraf measurements to the Java metric names and matches Java's push semantics:

Java metric_name	Source	Semantics
system.cpu.util.pct	inputs.cpu (cpu-total)	gauge; last 100 - usage_idle per 60 s
system.net.total.bytes	inputs.net	sum of per-interface deltas of bytes_recv + bytes_sent over 60 s
system.disk.ops.count	inputs.diskio	sum of per-device deltas of reads + writes over 60 s
system.nfs.ops.count	inputs.execd (estat nfs -jm 10)	sum of iops(/s) × 10 across read + write rows over 60 s
system.iscsi.ops.count	inputs.execd (estat iscsi -jm 10)	same shape as nfs
system.tcp.total.bytes	inputs.execd (connstat-stats.sh)	sum of per-endpoint deltas of inbytes + outbytes over 60 s

Each compat metric carries a data.* breakdown that gets nested into a Java-shaped data: {} object by the record_transformer in app-gate #5043: 7 fields for CPU, 4 for net, 6 for disk, 6 each for NFS/iSCSI, 3 for TCP — 35 total. systemUniqueId / systemVersion are injected per compat metric (not Telegraf-global, so InfluxDB cardinality is unaffected), with values substituted at config-regen time from env vars the service script exports.

telegraf/perf_fluentd — enable | disable | status CLI installed to /usr/bin/. Manages /etc/telegraf/FLUENTD_ENABLED and restarts delphix-telegraf. Idempotent (no-op when the flag is already in the desired state so we don't bounce the service for no reason), dies loudly on touch failure. Off by default; app-gate wires perf_fluentd enable into the 3rd-party integration lifecycle in #5043.

telegraf/delphix-telegraf-service (modified) — exports DELPHIX_SYSTEM_UUID and DELPHIX_SYSTEM_VERSION from get-system-uuid / get-appliance-version, validates them against a safe char class before envsubst inlines them into the Starlark triple-quoted TOML string (bad chars fall back to "unknown" with a syslog warning rather than failing to boot). Includes the fluentd fragments when the flag is set; the storage_io inputs are now included when either InfluxDB or fluentd is enabled (previously only InfluxDB). Missing-file warnings fire for both outputs symmetrically. Discard fallback only when neither output is configured.

debian/rules (modified) — installs perf_fluentd. The two config fragments are picked up by the existing telegraf/telegraf* glob.

Backward compatibility

The full field matrix is in the design doc §6. In short: metric_name, _value, _time, event: "metric", type: "g", systemUniqueId, systemVersion, 1-event/minute cadence, and the nested data: {} breakdown all match Java's shape. Dataset and capacity events (system.dataset.total.bytes, system.capacity.consumer.size, system.capacity.source.size) flow through unchanged via Java's untouched logMetric / pushInsightMetrics paths — only the KPI pushInsightMetric singular was removed in app-gate.

What customers do lose: TCP per-connection dimensions (data.service, data.localAddress, data.remoteAddress, window sizes, RTT) because the aggregated system.tcp.total.bytes can't preserve them; and performanceMetricsResolution = SECOND (hardcoded 60s bucket). Customer migration cost: Splunk HEC users do nothing (template change ships in #5043); Datadog (×2) and Dynatrace (×1) customers add a 5-line <source @type http port 9880> stanza to their custom Fluentd template.

Review-driven changes

Three Copilot findings fixed in this PR, four more accepted and documented as follow-ups in the design doc §6.

Fixed in the code:

• Processor ordering (order = 4 on the Starlark block) — Copilot #4409213790
• TCP data.* cut from 8 fields to 3 — the five I dropped (send/congestion/receive window sizes, RTT, connections) were last-write-wins across endpoints and had no meaningful value at the aggregated level
• envsubst safe-char guard on UUID / version in the service script

Accepted and documented (not blocking):

• Idle source last-bucket flush — bucket emission is sample-driven, so a minute of zero activity leaves the final bucket buffered. Impact is a short gap at the start of idle periods, not wrong values. Copilot #4409213817
• connstat-stats.sh endpoint totals are non-monotonic (socket close shrinks the aggregate), so bytes from a closing session are undercounted. Rare on Delphix workloads; follow-up to make connstat emit true cumulative counters. Copilot #4409213874
• iops(/s) * 10 loses precision below one op per second (bcchelper does integer division). _value and data.* share the same source so they stay internally consistent. Copilot #4409213850
• No shell / Starlark test harness in the repo today; manual deploy-and-capture is how this PR was verified. Follow-up to add a debian/tests/ autopkgtest or Actions job that generates the config and runs telegraf --test against fixture input.

Testing

Deployed to ev-dev.dlpxdc.co and dbs-test.dlpxdc.co and captured end-to-end:

• perf_fluentd enable|disable|status behaves correctly across all state transitions; idempotent reruns are no-ops.
• Service regenerates telegraf.conf correctly for every flag combination; envsubst substitutes the real UUID/version into the Starlark source; missing-file warnings fire as expected.
• Captured raw POSTs to :9880 and confirmed each is one JSON object with the full HEC envelope + 35 data.* fields populated.
• End-to-end through Fluentd into Splunk at qa-splunk.ops.delphix.com, index=suyash_metrics: all six KPI events arrive with correct metric_name, _value, nested data: {}, systemUniqueId, systemVersion. Dataset and capacity events pass through from Java with their flat schema preserved. No null events once the record_transformer fallback landed in #5043.

Live samples for each metric_name are in the design doc §10.