Skip to content

hermes: bound the checkpoint manager's pre-snapshot tree walk - #180

Merged
AlienWalker1995 merged 1 commit into
mainfrom
fix/hermes-checkpoint-scan-bounds
Aug 19, 2026
Merged

hermes: bound the checkpoint manager's pre-snapshot tree walk#180
AlienWalker1995 merged 1 commit into
mainfrom
fix/hermes-checkpoint-scan-bounds

Conversation

@AlienWalker1995

Copy link
Copy Markdown
Owner

Root cause (audited 2026-08-17)

Hermes sessions were stalling ~530–630s on every terminal call containing a > redirect (2>/dev/null counts, via _REDIRECT_OVERWRITE). The destructive-command heuristic triggers ensure_checkpoint()_dir_file_count(), an unbounded rglob of the process cwd. /workspace/data is a Docker Desktop 9p bind (~50–95 stats/s), so each walk toward the 50k-file cap took ~9 minutes — and new_turn() clears the dedup set, so the walk repeated every turn. The scan always ended in ">50k files, checkpoint skipped" logged at DEBUG only: no checkpoint was ever produced, pure invisible cost.

Evidence: session 20260817_152304_ed681a spent ~3h on a trivial config task (~20 stalls × ~530s); py-spy caught the live process inside _dir_file_count → rglob; the 2026-08-16 gateway/cron sessions show identical ~600s stalls all day; a 30s live sample measured 48 files/s on /workspace (50k cap ⇒ 9–17 min).

Fix

Tracked build-time patch, same mechanism as pressure-user-condense.diff (git apply --check fails the build loudly on a pin bump):

  • Wall-clock bound on the walk: HERMES_CHECKPOINT_SCAN_SECONDS (default 10s) — a tree that can't even be counted quickly is treated as oversized
  • Oversized verdict is remembered for the session (_oversized_dirs, deliberately not cleared by new_turn())
  • The skip is logged once at INFO instead of silently at DEBUG

Also pins LF endings for services/hermes/*.diff so a Windows checkout can't corrupt hunks applied inside the Linux build.

Validation

  • git apply --recount --check passes against the pinned checkout (3c27eb62, v0.20.0) — same command the Dockerfile runs
  • Patched file passes py_compile under the image's Python 3.11
  • Deployment mitigation already applied live: checkpoints.enabled: false (the feature has never successfully snapshotted this tree); this patch bounds the damage if it's ever re-enabled

Upstream NousResearch/hermes-agent is still unbounded as of v0.20.3 — re-check on pin bumps.

🤖 Generated with Claude Code

With checkpoints.enabled, any terminal command containing a bare `>`
redirect (`2>/dev/null` qualifies) is classified destructive and triggers
_dir_file_count(), an unbounded rglob of the process cwd. /workspace/data
is a 9p bind, so each walk toward the 50k-file cap took ~9 minutes at
~50-95 stats/s, and new_turn() clears the dedup set so the doomed walk
repeated every single turn. Session 20260817_152304_ed681a burned ~3h on
~20 such stalls doing a trivial config task (py-spy caught the live
process inside _dir_file_count); the 2026-08-16 cron/gateway sessions
show the same ~600s stalls all day. The scan always ended in ">50k
files, checkpoint skipped" at DEBUG, so no checkpoint was ever taken —
pure cost, invisible in logs.

Tracked build-time patch (same mechanism as pressure-user-condense.diff,
git apply --check fails the build loudly on a pin bump):

- wall-clock bound on the walk: HERMES_CHECKPOINT_SCAN_SECONDS (default
  10s); a tree that can't be counted quickly is treated as oversized
- the oversized verdict is remembered for the session instead of being
  re-derived every turn
- the skip is logged once at INFO so this failure mode is visible

Also pins LF endings for services/hermes/*.diff so a Windows checkout
can't corrupt the hunks the Linux image build applies.

Deployment note: checkpoints.enabled has been set to false in the live
config (the feature has never successfully snapshotted this tree); this
patch bounds the damage if it is ever re-enabled. Upstream hermes-agent
v0.20.3 is still unbounded.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@AlienWalker1995
AlienWalker1995 merged commit 2dfd192 into main Aug 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant