fix(worker): discard impossible cgroup v2 cpu samples

[longcw]

Problem: On some hypervisors the root cgroup usage_usec counter returns a torn per-cpu sum, and CGroupV2CPUMonitor.cpu_percent turns that one read into a 1.0 or a large negative sample. Four such samples in the five-slot load average mark an idle worker FULL for 2.5 s, and a call that lands in that window gets no server.

Fix: The monitor measures elapsed time around the two reads and discards a delta that is negative or above elapsed x host cpus, since no host can produce either. A discarded sample returns the last good value instead of a fresh number, so the average is unchanged on idle and busy hosts alike.

Fixes #7102. Supersedes #7103.

Context for reviewing and coding agents

How to see it

tests/test_cpu_monitor.py replays the six raw (start, end) counter pairs from the issue log through a scripted _read_cpu_usage, a no-op time.sleep and a scripted time.monotonic on an 8-cpu host. test_reporter_pattern_stays_below_threshold pushes the reporter's sample sequence through utils.MovingAverage(5) and asserts the average stays at the idle fraction on every step; on main the same sequence averages 0.80, the value in the report.

Blast radius

The only caller is _DefaultLoadCalc in worker.py:98, which feeds cpu_percent(interval=0.5) into utils.MovingAverage(5) at worker.py:88. cpu_count, _read_cpu_max, _read_cpu_usage, CGroupV1CPUMonitor and DefaultCPUMonitor are unchanged, so the num_idle_processes defaults at worker.py:208 and worker.py:299 keep their values.

Why the ceiling uses host cpus and not the quota

cpu.max is an average limit and a container can burst above it for one interval, so a quota-based ceiling would discard real samples. psutil.cpu_count() is the physical bound; the quota still divides the accepted sample, so a burst clamps to 1.0 rather than being thrown away.

Alternatives rejected

Clamping to [0, 1] removes the negative case only; an inflated delta still clamps to 1.0 and four of them still average to 0.80. Falling back to psutil.cpu_percent is not safer, because /proc/stat is summed by the kernel from the same per-cpu kcpustat counters as the root cgroup's cpu.stat. Returning 0.0 on a discard, as #7103 does, is the same defect mirrored: on a saturated worker each glitch drops the average by 0.2 and the worker keeps accepting jobs. #7103 also reads the process's own cgroup and walks cpu.max up the ancestor chain, which changes what load means for a systemd-managed worker and belongs in its own discussion.

Where the numbers come from

The reporter's log shows deltas of +5.9e8 and -2.7e9 microseconds over 0.5 s on 8 cpus, against a physical ceiling of 4e6. Four 1.0 samples in a five-slot average give 0.8011 (4.0054 / 5), which matches the load value in the reported log line.

[devin-ai-integration]

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🟡 Pre-sample pauses suppress CPU load

Scheduler pauses before the first counter read inflate elapsed without adding measured CPU time. A busy worker can report artificially low load and accept excess work.

Suggested change

	start = time.monotonic()
	cpu_usage_start = self._read_cpu_usage()
	cpu_usage_start = self._read_cpu_usage()
	start = time.monotonic()

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 First corrupt sample reports idle

When the initial counter delta is rejected, _last_cpu_percent remains zero. A saturated worker can advertise idle load until a valid sample arrives.

Prompt for agents

CGroupV2CPUMonitor initializes _last_cpu_percent to 0.0, but cpu_percent returns that value whenever an impossible delta is discarded. On startup there is no valid previous sample, so a busy worker is treated as idle for one or more rejected samples. Track whether a valid sample exists and handle the no-history case conservatively, such as retrying the counter read or returning a load value that cannot admit excess work. Preserve the existing last-good-value behavior after a valid sample has been recorded, and add tests for first-sample positive and negative torn reads on a saturated worker.

---

Was this helpful? React with 👍 or 👎 to provide feedback.