Skip to content

Add opt-in Prometheus outbox queue lag metrics - #270

Merged
lesnik512 merged 2 commits into
mainfrom
outbox-lag-metric
Nov 3, 2026
Merged

lesnik512 merged 2 commits into
mainfrom
outbox-lag-metric

Conversation

@lesnik512

Copy link
Copy Markdown
Member

Summary

  • Enable queue lag sampling with PrometheusRecorder(queue_lag_interval_seconds=60). Sampling is disabled by default and follows the broker lifecycle.
  • Export the oldest claimable row age and last successful sample time per queue. Derive lease TTLs from registered subscribers.
  • Document an optional lease expiry index for queues with many active leases.

Benchmark

Postgres 17, one million rows per queue, 25 samples per case. Median / p95 query round-trip times in milliseconds.

Workload Existing indexes Optional lease index
1M pending 0.29 / 25.80 0.32 / 3.93
900k active leases, 100k pending 197.57 / 357.45 0.30 / 1.67
1M active leases 225.03 / 845.59 0.32 / 0.82

EXPLAIN ANALYZE showed scans over all active leases with the existing indexes in the lease-heavy cases. The optional index allowed an index-only scan bounded by the expiry cutoff. The benchmark used a static backlog, so it does not measure index maintenance cost under concurrent writes.

Verification

830 tests passed against Postgres. Ruff, type checking, and the strict docs build passed.

@github-actions

github-actions Bot commented Nov 3, 2026 •

Copy link
Copy Markdown

Benchmark gate

✅ gate passed

scenario msg/s delete/msg WALrec/msg WALB/msg fpi upd del dead_tup
consumer/w1/b10 1638 1.000 6.71 919 242 5000 5000 10000
consumer/w1/b100 1887 1.000 7.11 1154 243 5000 5000 10000
consumer/w2/b10 1725 1.000 6.72 924 242 5000 5000 10000
consumer/w2/b100 2284 1.000 7.11 1154 243 5000 5000 10000
consumer/w4/b10 2210 1.000 6.73 942 242 5000 5000 10000
consumer/w4/b100 1990 1.000 7.15 1188 263 5000 5000 10000
consumer/w1/b100/tfbs100 8201 0.010 6.11 1132 244 5000 5000 10000
consumer/w1/b100/interval1-10 1835 1.000 7.11 1154 243 5000 5000 10000
producer/w1/b100 4566 0.000 3.04 597 1 0 0 0
producer/w1/b100/ppt1 1099 0.000 4.04 635 0 0 0 0

Gated (fails the build): delete_calls + tuple counters (upd/del/ins) + the producer's insert_calls, exact; select_calls and other_calls within +2; wal_records within a 10% band; fetch_calls within 2x. msg/s, WAL bytes and total calls are informational (timing/FPI noise).

@lesnik512
lesnik512 merged commit 57c5708 into main Nov 3, 2026
15 checks passed
@lesnik512
lesnik512 deleted the outbox-lag-metric branch November 3, 2026 05:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant