Skip to content

/metrics: the dispatcher's bb_* counters, one HELP/TYPE family each (stacked on #952) - #971

Draft
MattJackson wants to merge 2 commits into
predevfrom
lane-cabi-dispatch-metrics
Draft

MattJackson wants to merge 2 commits into
predevfrom
lane-cabi-dispatch-metrics

Conversation

@MattJackson

Copy link
Copy Markdown
Collaborator

CONTRACT-ABI lane: DispatchStats Prometheus exposition. STACKED on #952 (it needs bb_plugin_quarantined_total), so it stays DRAFT until #952 lands (queue rule 19).

Coordinator ruling 2026-10-14: every bb_* counter DispatchStats holds must be exported through the existing metrics path with the HELP/TYPE lines (one family per counter), with a RED that scrapes it; new 1.6.0 surface (additive). The counters are named in THE DESIGN A.4.3 (bb_deadline_without_wake_total) and A.8, BUSBAR-1.6.0.md l.3690 (bb_plugin_quarantined_total). Before this PR neither was exported.

  • plugin-loader dispatch/worker.rs: DispatchStats::families() is the one list of the bb_* counters the stats hold, each as (name, HELP, value), with DEADLINE_WITHOUT_WAKE_TOTAL beside dispatch watchdog: quarantine on the Nth breach, counted as bb_plugin_quarantined_total (A.8) #952's QUARANTINED_TOTAL. The QUARANTINED_TOTAL doc now says it is the process's sum, which is what the stats hold, not "labelled by plugin". The other DispatchStats fields (stale_wakes, replacements, write_behind_late, live_reapers, driver_kept_high) have no bb_* name in the spec and stay internal.
  • busbar-kernel preflight.rs: RootInstall::host_counters: Option<HostCounters> (fn() -> Vec<(&'static str, &'static str, u64)>). This is a by-name field on the existing root install, not a new static seam, and the kernel names neither the loader nor the dispatcher. snapshot::refresh_host_counters / emit_host_counters describe each family and set its cumulative value (Counter::absolute). export/scrape.rs whole() calls it next to refresh_scrape_gauges before the scrape sink renders. The test-build STAND_IN installs None, so kernel scrapes are unchanged.
  • busbar root: root::dispatch::host_counters reads DISPATCHER.get(), so a scrape never builds a dispatcher, and linked.rs installs it.
  • Additive surface: two rows in docs/observability.md ("New in 1.6.0") and a CHANGELOG ### Added line. Nothing 1.5.5 exposed moves: the busbar_* families and their order are untouched. Each new family sits at 0 on a healthy node and does not move inside an oracle cell. No accepted-differences entry is added because an entry needs an owner signoff, which an agent cannot write. If the oracle's metrics diff shows the two new families, that entry goes to the owner through the coordinator.

RED: crates/busbar/tests/metrics_scrape_boot_window.rs::the_dispatchers_bb_counters_are_scraped_with_help_and_type boots the real binary with module: prometheus, scrapes /metrics, and requires for each family the exact # HELP line, # TYPE <name> counter and an integer sample. On predev (and on #952 alone) neither family is in the exposition. The sibling boot-window test's closed HELP set is taken from the same process, so it gains the two families on both sides.

Shared files (rule 20, disjoint hunks): crates/busbar-kernel/src/preflight.rs is also edited by #805/#863/#864/#865 (their hunks are at l.96-110 and l.777+; this PR's are at the RootInstall tail and STAND_IN). crates/busbar/src/root/linked.rs is also edited by #864/#865 (their register_stores hunk is the link_secrets line; this PR adds one field line after export_axis). crates/busbar/src/root/dispatch.rs is also edited by #913 (its hunk is in RootAuthAxis at l.112; this PR's is above dispatcher()).

No dependency change.

…ted as bb_plugin_quarantined_total

THE DESIGN A.8 (BUSBAR-1.6.0.md l.3689-3690): "on a breach of the threshold, N times, the instance
is quarantined: new calls get FAULT (hooks go to on_error), plus a diagnostic and
bb_plugin_quarantined_total". predev faulted the instance on the first breach with no count, no
diagnostic and no counter (dispatch/watchdog.rs:4-13).

- DispatchConfig::quarantine_after (N, clamped >= 1; DEFAULT_QUARANTINE_AFTER = 1 keeps the
  first-breach quarantine every caller has today).
- every breach (a worker's hung crossing, or a hung ticket-less call, counted ONCE however many
  scans see it) is counted on the instance and warned with the plugin's name; below N the worker is
  replaced and the op is FAULT but the instance keeps serving; the Nth quarantines it: FAULT without
  a crossing, its connector streams purged, a diagnostic, and DispatchStats::quarantined
  (QUARANTINED_TOTAL = "bb_plugin_quarantined_total").
- the timed-probe exit (R2) is the hook door's existing trial (hook_door.rs, accepted difference
  Q133, owner-signed 2026-10-07); unchanged.

RED: red_an_instance_is_quarantined_on_its_nth_breach_and_not_before (N = 2: one hung call is one
breach across many scans, the instance serves after it, the second breach quarantines and counts);
red_a_hung_ticketless_call_faults_its_instance now also asserts the counter.
…family each

Coordinator ruling 2026-10-14: every bb_* counter DispatchStats holds is exported through the
existing metrics path. DispatchStats::families() is the one list (bb_deadline_without_wake_total,
THE DESIGN A.4.3; bb_plugin_quarantined_total, A.8). The composition root hands the kernel a
reader through RootInstall::host_counters; the kernel's whole-snapshot read describes each
family and sets its cumulative value before the scrape sink renders it. Additive 1.6.0 surface:
docs/observability.md rows and a CHANGELOG Added line.

RED: the_dispatchers_bb_counters_are_scraped_with_help_and_type (the real binary's /metrics).

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant