feat: add apply dedup and flush delivery telemetry by vahidlazio · Pull Request #575 · spotify/confidence-resolver

vahidlazio · 2026-09-01T14:14:51Z

Summary

Add ApplyDedupTelemetry (proto field 10) tracking dedup counters: applies_total, applies_deduped, applies_not_cached, sweeps, map_size, map_capacity
Add flush_succeeded/flush_failed (proto fields 11-12) tracking WriteFlagLogs batch delivery outcomes across all providers
Host SDKs (JS, Java, Go, Python) track delivery locally and merge into TelemetryData before next send — zero WASM overhead, no resolve contention
Cloudflare resolver tracks delivery in queue consumer

Apply Dedup Metrics

Metric	Type	Description
`applies_total`	delta counter	Total flag applies entering dedup filter
`applies_deduped`	delta counter	Duplicates filtered out
`applies_not_cached`	delta counter	Passed through but not cached (map full)
`sweeps`	delta counter	Sweep operations executed
`map_size`	gauge	Current dedup map entries
`map_capacity`	gauge	Max allowed entries

Flush Delivery Metrics

Metric	Type	Description
`flush_succeeded`	delta counter	Successful WriteFlagLogs batch deliveries
`flush_failed`	delta counter	Failed WriteFlagLogs batch deliveries

Compatibility

Proto fields are additive — old consumers ignore them. Follow-up PRs needed:

epx-flags-admin: convertApplyDedup + convertFlush in VictoriaMetricsClient
epx-flags-resolver: same (has its own VictoriaMetricsClient)
epx-insights: GraphType entries + PromQL queries for dashboards

Test plan

447 Rust unit tests pass (7 new tests for dedup counters, proto delta, Prometheus rendering, accumulate_delta, flush counters)
Clippy clean on confidence_resolver + confidence-cloudflare-resolver
JS provider build
Java provider build
Go provider build
Python provider proto regeneration
Integration test with dedup enabled

🤖 Generated with Claude Code

Add telemetry for the apply deduplication feature and WriteFlagLogs batch delivery outcomes across all providers. Apply dedup telemetry (ApplyDedupTelemetry, field 10 on TelemetryData): - applies_total, applies_deduped, applies_not_cached, sweeps (delta counters) - map_size, map_capacity (gauges) - Tracked via inline u64 counters on ApplyDedup — zero overhead - Wired into WASM guest (bounded_flush_logs + prometheus_snapshot) - Wired into Cloudflare resolver (per-request delta) Flush delivery telemetry (fields 11-12 on TelemetryData): - flush_succeeded, flush_failed (delta counters) - Host SDKs track delivery outcomes locally and merge into TelemetryData before the next send — zero WASM calls, no resolve-path contention - JS: addFlushDeliveryTelemetry() decode/set/encode - Java: drainFlushCounters() on WasmFlagLogger, merged in ProviderTelemetryResolver - Go: atomic counters, swap-and-inject in Write() - Python: counters merged in _write_logs() - Cloudflare: tracked in queue consumer via update_prometheus_kv() Proto fields are additive — old consumers ignore them. Follow-up PRs needed in epx-flags-admin and epx-flags-resolver (VictoriaMetricsClient converters) and epx-insights (dashboard GraphType entries) to display the metrics. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add events_published, event_batches_succeeded, event_batches_failed (proto fields 13-15 on TelemetryData) to track event pipeline delivery outcomes. Piggybacked on the flag logs TelemetryData channel since the events pipeline has no telemetry path of its own. Host SDKs track event delivery locally and merge counters into TelemetryData during the next flag logs flush — same zero-overhead pattern as flush_succeeded/flush_failed. - JS: counters in addFlushDeliveryTelemetry alongside flush counters - Java: drainEventCounters() on provider, merged in ProviderTelemetryResolver - Go: RecordEventBatch() on FlagLogger interface, drained in Write() - Python: _event_telemetry_* counters, drained in _write_logs() - Cloudflare: update_events_kv() in queue consumer after delivery Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…on failure - Remove duplicate map_size Prometheus block in write_apply_dedup (would cause Prometheus/OpenMetrics parser rejection) - Cloudflare: only increment events_published on successful delivery - Java: use separate telemetryFlushSucceeded/telemetryFlushFailed counters instead of reusing failures (which races with the periodic log window) - Add OpenMetrics parser test with all new telemetry types - Add accumulate_delta test for event counters Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Bugs fixed: - Remove duplicate map_size Prometheus block (would break scrapers) - Cloudflare: only count events_published on successful delivery - Java: separate telemetryFlushSucceeded/Failed counters to avoid racing with the existing periodic log window - Go: add RecordEventBatch to test mocks (CapturingFlagLogger, trackingFlagLogger, MockFlagLogger) — adding the method to the FlagLogger interface broke all existing tests Tests added: - OpenMetrics parser test with all new telemetry types - Event counter accumulate_delta test Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Add the ApplyDedupTelemetry message to the shared provider proto so host SDKs can read dedup counters (applies_total, applies_deduped, applies_not_cached, sweeps, map_size, map_capacity) from TelemetryData. Verified under concurrent load test (2000 resolves, 2000 events): - applies_total correctly counts only matching resolves entering dedup - applies_deduped shows 97.4% dedup ratio for repeated same-user resolves - map_size/map_capacity gauges report correct values - Zero applies_not_cached (map well within capacity) - No panics or data races under concurrent load Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The Go provider's embedded WASM must be built in Docker for deterministic output. The locally-built binary doesn't match the CI Docker build, failing the WASM sync check. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The previously committed Python proto was missing apply_dedup (field 10), events_published (field 13), event_batches_succeeded (field 14), and event_batches_failed (field 15). The provider code references these fields and would crash at runtime with AttributeError. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Docker-built WASM reflecting apply_dedup telemetry changes in confidence-resolver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…itTelemetry TypeScript strict mode requires all non-optional fields. The existing TelemetryData literal in addProviderInitTelemetry was missing the new flush and event counter fields. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

ts-proto drops unknown fields during decode→encode. The provider proto was missing resolveLatency, resolveRate, stateAge, memoryBytes — fields set by the WASM guest that must survive when hosts re-encode TelemetryData (in addFlushDeliveryTelemetry, addProviderInitTelemetry). Without these fields, JS resolve-logging tests fail because resolve rate and latency telemetry are stripped during re-encoding. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Align with epx-flags-resolver PR #1734: restructure flush and events telemetry from flat fields (11-15) to nested messages (11-12). TelemetryData.flush = FlushTelemetry { succeeded, failed } TelemetryData.events = EventsTelemetry { published, batches_succeeded, batches_failed } Updated across all providers: JS, Java, Go, Python, Cloudflare, and core Rust telemetry. WASM rebuilt in Docker for Go sync check. Also adds WASM-produced fields (resolve_latency, resolve_rate, state_age, memory_bytes) to provider proto so decode→encode round-trips in ts-proto preserve them. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

On failed WriteFlagLogs send, the drained flush/events counters were lost — causing N consecutive failures to report flush_failed=1 instead of flush_failed=N. Fix: on failure, add the drained values back to the atomics so the next successful send carries the full accumulated count. Includes Go test proving 3 consecutive failures → flush_failed=3. Providers fixed: Go, JS, Java, Python. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace fragile save-before-drain + restore-from-saved pattern with restoreDrainedCounters() that reads values directly from the encoded request on failure. The drain stays in addFlushDeliveryTelemetry, the restore reads from the same proto — no coupling through captured locals. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace saved-before-drain locals with reading values from the request proto on failure, matching the JS/Go/Java pattern. Event counter drain also moved inside the rewrite block under _event_stats_lock. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

#1 Python flush counter race: protect _flush_succeeded/_flush_failed with _event_stats_lock (read, drain, restore all under lock). #2 Cloudflare cross-queue KV race: merge update_prometheus_kv and update_events_kv into single update_kv_snapshot — one read-modify- write per call, no cross-queue clobber. #3 Java silent exception: add logger.warn in empty-destinations catch. #4 Go drain/restore duplication: extract TelemetryCounters struct with DrainAndStamp, RestoreOnFailure, RecordEventBatch. Both grpc.go and multi.go embed it — ~47 lines of duplication removed. #5 JS double decode/encode: merge addProviderInitTelemetry and addFlushDeliveryTelemetry into single enrichTelemetry method — one decode/encode pass on first flush. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

nicklasl · 2026-09-04T11:12:11Z

@@ -704,11 +722,6 @@ async fn consume_flag_logs(

        let req = flag_logger::aggregate_batch(logs);


AI claims that the aggregate_batch method:

merges only latency, resolve rates, state age, memory, and resolver version. Consequently, apply_dedup, provider flush, provider events, and incidentally provider_init_rate disappear before update_kv_snapshot or downstream delivery sees them.

nicklasl · 2026-09-04T11:17:28Z

      const isLast = i === destinations.length - 1;
      try {
        const ok = await this.sendFlagLogsToDestination(encodedWriteFlagLogRequest, destinations[i], signal);
        if (ok) return;


I think sendFlagLogsToDestination() will return true even if the last retry fails which will mean that sendFlagLogs returns without throwing and therefore we increment and count that as a successful flush on L526.

This sounds like a mis-reporting.

nicklasl · 2026-09-04T11:25:32Z

+			p.flagLogger.RecordEventBatch(eventCount, false)
+		}
+	} else if p.flagLogger != nil {
+		p.flagLogger.RecordEventBatch(eventCount, true)


are we sure all events were accepted at this point? don't we need to look at the response?

nicklasl · 2026-09-04T11:26:30Z

@@ -784,6 +795,7 @@ private void sendEvents(FlushEventsResponse batch) {
      }


Here we report successful publish even though we potentially got errors back

nicklasl · 2026-09-04T11:27:32Z

+      this.eventBatchesSucceeded++;
+      this.eventsPublished += batch.events?.length ?? 0;
      const { errors } = PublishEventsResponse.decode(new Uint8Array(await response.arrayBuffer()));
      for (const error of errors) {
        logger.error(
          `Failed to publish event at index ${error.index}: ${EventError_Reason[error.reason]} ${error.message}`,
        );


same with other providers: we claim in telemetry that all events were published but some could have been rejected.

nicklasl

I think it makes sense to have a counter for rejected events.

nicklasl · 2026-09-04T11:30:03Z

 type FlagLogger interface {
 	Write(request *resolverv1.WriteFlagLogsRequest)
 	Shutdown()
+	RecordEventBatch(eventCount int, succeeded bool)


This is a public API change. Is there a way to keep it internal?

nicklasl · 2026-09-04T11:32:24Z

+  message ApplyDedupTelemetry {
+    uint32 applies_total = 1;
+    uint32 applies_deduped = 2;
+    uint32 applies_not_cached = 3;


could we name this a bit differently?
We increment this when the apply buffer "overflow", right? so maybe include something about that in the name?

- JS: throw when all flag log destinations fail with non-OK response - Go/Java/JS: subtract per-event errors from events_published count - Go: remove RecordEventBatch from public FlagLogger interface - Proto: rename applies_not_cached to unique_applies Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vahidlazio and others added 10 commits September 1, 2026 16:14

fix: gofmt Go provider files

977fbc8

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: sync WASM module for Go provider

8fe91b9

Docker-built WASM reflecting apply_dedup telemetry changes in confidence-resolver. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: prettier format ConfidenceServerProviderLocal.ts

04a1b32

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

vahidlazio marked this pull request as ready for review September 2, 2026 13:25

vahidlazio and others added 10 commits September 2, 2026 15:29

fix(java): fmt ProviderTelemetryResolver.java

6490dc8

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(java): use logger.debug for edge delivery failure, not warn

50380d8

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix(python): ruff format provider.py

7dd79a0

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

nicklasl reviewed Sep 4, 2026

View reviewed changes

vahidlazio and others added 2 commits September 4, 2026 16:09

fix: rename unique_applies to overflow in ApplyDedupTelemetry

ab8d906

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: rename proto field overflow to apply_dedup_overflow

6ec9675

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Navigation Menu

Uh oh!

feat: add apply dedup and flush delivery telemetry - #575

feat: add apply dedup and flush delivery telemetry#575
vahidlazio wants to merge 23 commits into
mainfrom
vahidt/apply-dedup-telemetry

vahidlazio commented Sep 1, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl left a comment

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

nicklasl Sep 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		@@ -704,11 +722,6 @@ async fn consume_flag_logs(

		let req = flag_logger::aggregate_batch(logs);

		@@ -784,6 +795,7 @@ private void sendEvents(FlushEventsResponse batch) {
		}

Uh oh!

Conversation

vahidlazio commented Sep 1, 2026

Summary

Apply Dedup Metrics

Flush Delivery Metrics

Compatibility

Test plan

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl left a comment

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

nicklasl Sep 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants