Skip to content

hr-benchmarking: add a continuous observer for one measured unit - #509

Merged
bompus merged 7 commits into
mainfrom
feat/unit-observer
Oct 31, 2026
Merged

bompus merged 7 commits into
mainfrom
feat/unit-observer

Conversation

@bompus

@bompus bompus commented Oct 31, 2026

Copy link
Copy Markdown
Owner

Adds skills/hr-benchmarking/scripts/unit-observer.py, a continuous observer for one measured unit that runs inside one cgroup, so a timed run can show that the host and the unit's own cgroup stayed within the admission limits throughout, not only at its boundaries.

What it checks, once a second plus one last sample at the stop: available memory (unrounded); host and workload memory and I/O full PSI per observation interval (unreadable PSI rejects); unbroken coverage on the monotonic clock; competing CPU as host busy cores minus the workload cgroup's own CPU over the whole run (--host-only counts the unit's own CPU); and, comparing every sample with the first, growth of memory.swap.current or memory.zswap.current, any change of zswpin, zswpout, zswpwb or of memory.events high, max, oom or oom_kill, a replaced directory, backward counters, unreadable required counters and a changed set of optional counters. Host swap-out and zswap activity is recorded per interval and rejects only with --storage-phase. It records its own cost against budgets and writes every sample to <output>.samples.jsonl. A clean summary does not qualify a run by itself; the protocol still declares observers, budgets, stop rules and retries.

Proof:

  • Nine tests against a fake --proc-root and a fake cgroup directory: quiet clean, final sample at SIGTERM, OOM, swap, zswap and memory.stat changes, an excursion that rises and returns, missing or vanishing counters, pressure over the limit, a replaced directory, a backward counter, unreadable host I/O pressure (summary still written), competing CPU with and without a workload to subtract, --host-only, and --storage-phase. Weakening two checks (final-only events, unsubtracted host CPU) makes the corresponding tests fail.
  • Real cgroups: a workload burning about 5 cores in its own scope came out clean with 3.5 competing cores; seven foreign loops outside the scope were rejected at 4.8 competing cores; observer cost was 0.003 cores and 19 MiB.
  • bun run check passes.

Review: an independent review of the first version (GPT-6.1 Sol, xhigh) returned BLOCK with nine findings; all are fixed in the second commit. A second round could not run: the Codex quota is exhausted until Nov 2 and Muse's until Nov 2, so the rewrite has had no second local review. The private reviewer covers the head.

@bompus bompus added the author:anthropic Pull request written by a anthropic model label Oct 31, 2026
@bompus-ai-review

Copy link
Copy Markdown

Bompus AI Review could not run on 736ec70: no reviewer seat finished the review (GPT-6.1 Sol: usage limit reached; Muse Wonder Full: usage limit reached; SWE-2 Max: failed (exit 3)).

Nothing was reviewed. Pushing a new commit or re-running the review tries the seats again.

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (SWE-2 Max, max reasoning): 1 finding(s).

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (SWE-2 Max, max reasoning): 1 finding(s).

@bompus-ai-review

Copy link
Copy Markdown

Bompus AI Review could not run on c1dd437: no reviewer seat finished the review (GPT-6.1 Sol: usage limit reached; Muse Wonder Full: usage limit reached; SWE-2 Max: failed (exit 3)).

Nothing was reviewed. Pushing a new commit or re-running the review tries the seats again.

@bompus bompus added author:anthropic Pull request written by a anthropic model and removed author:anthropic Pull request written by a anthropic model labels Oct 31, 2026

@bompus-ai-review bompus-ai-review Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bompus AI Review (SWE-2 Max, max reasoning): 0 finding(s).

Review 3: only high findings and above are posted from here on.

@bompus bompus closed this Oct 31, 2026
@bompus
bompus merged commit c1dd437 into main Oct 31, 2026
2 checks passed
@bompus
bompus deleted the feat/unit-observer branch October 31, 2026 10:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

author:anthropic Pull request written by a anthropic model

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant