Skip to content

Health samples: send them apart, at most one a window - #162

Merged
fxck merged 5 commits into
mainfrom
fix/health-frames
Oct 9, 2026
Merged

fxck merged 5 commits into
mainfrom
fix/health-frames

Conversation

@fxck

@fxck fxck commented Oct 9, 2026

Copy link
Copy Markdown
Member

Health samples from a strained Mate flooded the client's HQ structure socket. In run 1 two projects each sent a 3.7 KB attention frame every 2 s. In run 3 Milo, at its memory limit, sent up to 33 frames a second at about 5 KB each (11.7 KB/s on average). There were three causes:

  • Mate (apps/server/src/zerops/mateResourceHealth.ts): while strained, the changesWith key was the whole sample. Every changed number got published, and the reader woke on every memory event.
    • It now publishes again only when the warning changes (status, severity, resources, unavailable, caps) or when a number the warning quotes changes. Only the CPU warning quotes numbers (cpuWarningQuote in contracts, shared with the copy).
    • It sends at most one sample per 2 s window, the latest (latestAtMostEvery).
  • HQ relay (apps/hq/src/hqScopes.ts, attention branch of load): every sample resent the Mate's whole record, overview included. An attention scope can now ask for health: "apart". HQ then keeps the record and health as two values in that scope: a sample changes only health, and the record is resent only when it changes.
  • HQ link (apps/hq/src/link.ts): HQ relays at most one sample per link per 2 s window, the latest. This also protects readers from Mates that predate this change.

Version tolerance

  • The scope's key (hqScopeKey) ignores the new field, so a client reads either HQ's answer.
  • An HQ from before passes the field by (excess key) and sends health inside the record. The client's mateHealth family reads it from either shape.
  • A client from before does not ask, so it gets the whole record as before. HQ keeps the two shapes in separate journals.

Deploy order: HQ first (pack-core, deploy), then the hosted client, then a Mate release. Any order is safe; the savings arrive with each part.

Measured. Test world: a running Core, a Mate link and two reader sockets of one person, with a recorded I/O-strained sample sent at 33/s for 20 s, scaled to a minute.

Setup Bytes per minute Frames per minute
Before (whole record, no cap) 5.48 MB 1,980
Health apart, no cap 2.47 MB 1,980
After (health apart + cap) 45 KB ~30
Old client on new HQ (whole record + cap) 99.5 KB ~30

The Mate rule, replayed on run 1's recorded series: the I/O-strained project sent 274 samples in 9.1 min and would now send 1. The other project would send 4 instead of 103.

Tests (new sentences; no title changed):

  • server: "a Mate publishes a sample again only when its warning, or a number the warning quotes, changes" (table), and "publishes at most one sample a window however often the kernel wakes it, and that the latest". In "a tick during an event read…" only the setup changed: the tick's window is now strained, so it is published.
  • hq link.test.ts: "sends each health sample to a reader that takes health apart as that sample alone, never the Mate's unchanged record again" (an older reader beside it still gets the whole record), and "relays at most one health sample a window from a Mate that sends many, the latest".
  • client-runtime hq.test.ts: "asks HQ for a Mate's health apart and reads it from that value, or from the whole record of an HQ that sends it inside". Three tests that pin the subscription now expect health: "apart".

Gates

  • Passing:
    • suites: hq 1185, client-runtime 9077, server src/zerops 2759;
    • typechecks: contracts, shared, client-runtime, server, hq, web, mobile.
  • gate-changed fails only on the scenario "history does not paint through the composer footer". It also fails intermittently on the base code, with both variants failing alone.

🤖 Generated with Claude Code

fxck added 5 commits October 9, 2026 21:07
…r a quoted number changes

A Mate under strain published every 2 s sample whole, so an I/O stall sent
fresh CPU windows, memory and disk numbers nobody reads, and HQ relayed each.
The warning quotes numbers only for CPU (the measured window and its top
process, to two decimals), so the Mate now publishes again when the warning
changes or one of those quoted numbers does. cpuWarningQuote is the one place
the copy and the publish rule read the quote from.
… asks

HQ relayed every health sample inside the Mate's whole attention record, so a
strained Mate's 2 s samples resent its overview each time: 3.7 KB a sample on
the structure socket in the 2026-10-09 recording. An attention scope may now
ask for health apart; HQ then keeps the record and the health as two values of
the scope, so a sample changes only the health value and the record goes out
when it changes. The two shapes are separate journals of one scope key. A
reader that does not ask, and an HQ from before (which passes the key by),
keep the record whole, health inside.
…Q's attention scope

The relayed Mate subscribes with health apart and reads health from its own
value; from an HQ that does not take it apart it still reads health inside the
record. The record's families pass the health value by.
In the 2026-10-09 run 3 Milo sat at its memory limit and the kernel woke the
health reader on every memory event, so samples left at up to 33 a second.
The Mate now publishes the first sample at once and then at most one per 2 s
window, the latest, and only one that says something new.
…atest

An older Mate at its memory limit sends a sample on every kernel wake (up to 33
a second in the 2026-10-09 run 3), and HQ relayed each to every reader. Each
link now relays the first sample at once and then at most one per 2 s, the
latest; its reports run one at a time, the relay beside the frames it reads.
@fxck
fxck merged commit 98d98f1 into main Oct 9, 2026
19 checks passed
fxck added a commit that referenced this pull request Oct 9, 2026
- #162 Health samples: send them apart, at most one a window
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant