Bound the dead-letter stream, circuit breaker, damped metrics and dashboard fixes - #1
Merged
Conversation
… dashboard fixes
Dead-letter / Redis OOM (scrpr 2026-08-27 incident):
- XADD MAXLEN cap on the dead-letter stream (dead_letter.max_entries)
- master-driven housekeeping every dead_letter.prune_interval seconds
(TTL trim, cap, stale consumer sweep), plus torque:prune [--deep]
- workers delete their own consumer on clean shutdown
- per-stream circuit breaker with configurable window, threshold,
cooldown and half-open probes, reusing the per-queue pause path
- version-keyed data upgrade on master start ({prefix}version) that
sweeps orphaned job streams, stale index members and legacy keys
Metrics:
- minute / hour / day rollups (processed,failed) cluster-wide, per stream
and per job class; gauge tier for latency, concurrency, memory, depth
- damped 5-minute throughput gauge with exact last-hour count and
auto-fitting scale (dashboard.gauge_max)
- Jobs page: throughput, avg and peak runtime, fail rate per job class
- Queues page: processed and failed today, jobs/min, breaker badge
Dashboard fixes:
- refresh popover and theme toggle no longer reset on every poll tick
(Livewire morph vs x-show); tables scroll inside cards; grids stack
below 1180px
- inspector no longer lists Torque bookkeeping keys as queues
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On 2026-08-27 a scrpr.nl instance lost its KvK upstream. Every job failed permanently, the dead-letter stream grew to 3.9M entries (22 GB) and Redis went OOM. The stream had no cap and nothing ever called
trim(). Meanwhile the dashboard's refresh popover was stuck open and the throughput gauge swung between 0 and 90K/min on bursty workloads.What
Dead-letter / OOM
XADD MAXLEN ~cap (dead_letter.max_entries, default 100k)dead_letter.prune_intervalseconds;torque:prune [--deep] [--dry-run]XGROUP DELCONSUMERthemselves on clean shutdowncircuit_breaker.*, per-stream override or opt-out), reusing the per-queue pause path;torque:pause continue/queue:resumeforce-close it{prefix}version) that sweeps orphaned job streams, stale index members and legacy keys onceMetrics
Dashboard fixes
wire:ignore, Alpine-driven)Verified
vendor/bin/pest: 485 passed, 0 failed (8 environment skips) against Redis 8vendor/bin/pintcleanDocs: CHANGELOG 0.16.0, README, new UPGRADE.md.