Repository navigation
/metrics: the dispatcher's bb_* counters, one HELP/TYPE family each (stacked on #952) - #971
Draft
MattJackson wants to merge 2 commits into
Draft
MattJackson wants to merge 2 commits into
MattJackson wants to merge 2 commits into
Conversation
…ted as bb_plugin_quarantined_total THE DESIGN A.8 (BUSBAR-1.6.0.md l.3689-3690): "on a breach of the threshold, N times, the instance is quarantined: new calls get FAULT (hooks go to on_error), plus a diagnostic and bb_plugin_quarantined_total". predev faulted the instance on the first breach with no count, no diagnostic and no counter (dispatch/watchdog.rs:4-13). - DispatchConfig::quarantine_after (N, clamped >= 1; DEFAULT_QUARANTINE_AFTER = 1 keeps the first-breach quarantine every caller has today). - every breach (a worker's hung crossing, or a hung ticket-less call, counted ONCE however many scans see it) is counted on the instance and warned with the plugin's name; below N the worker is replaced and the op is FAULT but the instance keeps serving; the Nth quarantines it: FAULT without a crossing, its connector streams purged, a diagnostic, and DispatchStats::quarantined (QUARANTINED_TOTAL = "bb_plugin_quarantined_total"). - the timed-probe exit (R2) is the hook door's existing trial (hook_door.rs, accepted difference Q133, owner-signed 2026-10-07); unchanged. RED: red_an_instance_is_quarantined_on_its_nth_breach_and_not_before (N = 2: one hung call is one breach across many scans, the instance serves after it, the second breach quarantines and counts); red_a_hung_ticketless_call_faults_its_instance now also asserts the counter.
…family each Coordinator ruling 2026-10-14: every bb_* counter DispatchStats holds is exported through the existing metrics path. DispatchStats::families() is the one list (bb_deadline_without_wake_total, THE DESIGN A.4.3; bb_plugin_quarantined_total, A.8). The composition root hands the kernel a reader through RootInstall::host_counters; the kernel's whole-snapshot read describes each family and sets its cumulative value before the scrape sink renders it. Additive 1.6.0 surface: docs/observability.md rows and a CHANGELOG Added line. RED: the_dispatchers_bb_counters_are_scraped_with_help_and_type (the real binary's /metrics).
This was referenced Oct 14, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
CONTRACT-ABI lane: DispatchStats Prometheus exposition. STACKED on #952 (it needs
bb_plugin_quarantined_total), so it stays DRAFT until #952 lands (queue rule 19).Coordinator ruling 2026-10-14: every bb_* counter DispatchStats holds must be exported through the existing metrics path with the HELP/TYPE lines (one family per counter), with a RED that scrapes it; new 1.6.0 surface (additive). The counters are named in THE DESIGN A.4.3 (
bb_deadline_without_wake_total) and A.8,BUSBAR-1.6.0.mdl.3690 (bb_plugin_quarantined_total). Before this PR neither was exported.plugin-loaderdispatch/worker.rs:DispatchStats::families()is the one list of thebb_*counters the stats hold, each as (name, HELP, value), withDEADLINE_WITHOUT_WAKE_TOTALbeside dispatch watchdog: quarantine on the Nth breach, counted as bb_plugin_quarantined_total (A.8) #952'sQUARANTINED_TOTAL. TheQUARANTINED_TOTALdoc now says it is the process's sum, which is what the stats hold, not "labelled by plugin". The otherDispatchStatsfields (stale_wakes,replacements,write_behind_late,live_reapers,driver_kept_high) have nobb_*name in the spec and stay internal.busbar-kernelpreflight.rs:RootInstall::host_counters: Option<HostCounters>(fn() -> Vec<(&'static str, &'static str, u64)>). This is a by-name field on the existing root install, not a new static seam, and the kernel names neither the loader nor the dispatcher.snapshot::refresh_host_counters/emit_host_countersdescribe each family and set its cumulative value (Counter::absolute).export/scrape.rswhole()calls it next torefresh_scrape_gaugesbefore the scrape sink renders. The test-buildSTAND_INinstallsNone, so kernel scrapes are unchanged.busbarroot:root::dispatch::host_countersreadsDISPATCHER.get(), so a scrape never builds a dispatcher, andlinked.rsinstalls it.docs/observability.md("New in 1.6.0") and a CHANGELOG### Addedline. Nothing 1.5.5 exposed moves: thebusbar_*families and their order are untouched. Each new family sits at 0 on a healthy node and does not move inside an oracle cell. No accepted-differences entry is added because an entry needs an ownersignoff, which an agent cannot write. If the oracle's metrics diff shows the two new families, that entry goes to the owner through the coordinator.RED:
crates/busbar/tests/metrics_scrape_boot_window.rs::the_dispatchers_bb_counters_are_scraped_with_help_and_typeboots the real binary withmodule: prometheus, scrapes/metrics, and requires for each family the exact# HELPline,# TYPE <name> counterand an integer sample. On predev (and on #952 alone) neither family is in the exposition. The sibling boot-window test's closed HELP set is taken from the same process, so it gains the two families on both sides.Shared files (rule 20, disjoint hunks):
crates/busbar-kernel/src/preflight.rsis also edited by #805/#863/#864/#865 (their hunks are at l.96-110 and l.777+; this PR's are at theRootInstalltail andSTAND_IN).crates/busbar/src/root/linked.rsis also edited by #864/#865 (theirregister_storeshunk is thelink_secretsline; this PR adds one field line afterexport_axis).crates/busbar/src/root/dispatch.rsis also edited by #913 (its hunk is inRootAuthAxisat l.112; this PR's is abovedispatcher()).No dependency change.