Repository navigation
hr-benchmarking: add a continuous observer for one measured unit - #509
Merged
Merged
Conversation
… in host-only runs, record cost and samples
|
Bompus AI Review could not run on 736ec70: no reviewer seat finished the review (GPT-6.1 Sol: usage limit reached; Muse Wonder Full: usage limit reached; SWE-2 Max: failed (exit 3)). Nothing was reviewed. Pushing a new commit or re-running the review tries the seats again. |
…o the inode cannot be reused
|
Bompus AI Review could not run on c1dd437: no reviewer seat finished the review (GPT-6.1 Sol: usage limit reached; Muse Wonder Full: usage limit reached; SWE-2 Max: failed (exit 3)). Nothing was reviewed. Pushing a new commit or re-running the review tries the seats again. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds
skills/hr-benchmarking/scripts/unit-observer.py, a continuous observer for one measured unit that runs inside one cgroup, so a timed run can show that the host and the unit's own cgroup stayed within the admission limits throughout, not only at its boundaries.What it checks, once a second plus one last sample at the stop: available memory (unrounded); host and workload memory and I/O full PSI per observation interval (unreadable PSI rejects); unbroken coverage on the monotonic clock; competing CPU as host busy cores minus the workload cgroup's own CPU over the whole run (
--host-onlycounts the unit's own CPU); and, comparing every sample with the first, growth ofmemory.swap.currentormemory.zswap.current, any change ofzswpin,zswpout,zswpwbor ofmemory.eventshigh, max, oom or oom_kill, a replaced directory, backward counters, unreadable required counters and a changed set of optional counters. Host swap-out and zswap activity is recorded per interval and rejects only with--storage-phase. It records its own cost against budgets and writes every sample to<output>.samples.jsonl. A clean summary does not qualify a run by itself; the protocol still declares observers, budgets, stop rules and retries.Proof:
--proc-rootand a fake cgroup directory: quiet clean, final sample at SIGTERM, OOM, swap, zswap andmemory.statchanges, an excursion that rises and returns, missing or vanishing counters, pressure over the limit, a replaced directory, a backward counter, unreadable host I/O pressure (summary still written), competing CPU with and without a workload to subtract,--host-only, and--storage-phase. Weakening two checks (final-only events, unsubtracted host CPU) makes the corresponding tests fail.bun run checkpasses.Review: an independent review of the first version (GPT-6.1 Sol, xhigh) returned BLOCK with nine findings; all are fixed in the second commit. A second round could not run: the Codex quota is exhausted until Nov 2 and Muse's until Nov 2, so the rewrite has had no second local review. The private reviewer covers the head.