hermes: bound the checkpoint manager's pre-snapshot tree walk - #180
Merged
Conversation
With checkpoints.enabled, any terminal command containing a bare `>` redirect (`2>/dev/null` qualifies) is classified destructive and triggers _dir_file_count(), an unbounded rglob of the process cwd. /workspace/data is a 9p bind, so each walk toward the 50k-file cap took ~9 minutes at ~50-95 stats/s, and new_turn() clears the dedup set so the doomed walk repeated every single turn. Session 20260817_152304_ed681a burned ~3h on ~20 such stalls doing a trivial config task (py-spy caught the live process inside _dir_file_count); the 2026-08-16 cron/gateway sessions show the same ~600s stalls all day. The scan always ended in ">50k files, checkpoint skipped" at DEBUG, so no checkpoint was ever taken — pure cost, invisible in logs. Tracked build-time patch (same mechanism as pressure-user-condense.diff, git apply --check fails the build loudly on a pin bump): - wall-clock bound on the walk: HERMES_CHECKPOINT_SCAN_SECONDS (default 10s); a tree that can't be counted quickly is treated as oversized - the oversized verdict is remembered for the session instead of being re-derived every turn - the skip is logged once at INFO so this failure mode is visible Also pins LF endings for services/hermes/*.diff so a Windows checkout can't corrupt the hunks the Linux image build applies. Deployment note: checkpoints.enabled has been set to false in the live config (the feature has never successfully snapshotted this tree); this patch bounds the damage if it is ever re-enabled. Upstream hermes-agent v0.20.3 is still unbounded. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause (audited 2026-08-17)
Hermes sessions were stalling ~530–630s on every terminal call containing a
>redirect (2>/dev/nullcounts, via_REDIRECT_OVERWRITE). The destructive-command heuristic triggersensure_checkpoint()→_dir_file_count(), an unboundedrglobof the process cwd./workspace/datais a Docker Desktop 9p bind (~50–95 stats/s), so each walk toward the 50k-file cap took ~9 minutes — andnew_turn()clears the dedup set, so the walk repeated every turn. The scan always ended in ">50k files, checkpoint skipped" logged at DEBUG only: no checkpoint was ever produced, pure invisible cost.Evidence: session
20260817_152304_ed681aspent ~3h on a trivial config task (~20 stalls × ~530s); py-spy caught the live process inside_dir_file_count → rglob; the 2026-08-16 gateway/cron sessions show identical ~600s stalls all day; a 30s live sample measured 48 files/s on/workspace(50k cap ⇒ 9–17 min).Fix
Tracked build-time patch, same mechanism as
pressure-user-condense.diff(git apply --checkfails the build loudly on a pin bump):HERMES_CHECKPOINT_SCAN_SECONDS(default 10s) — a tree that can't even be counted quickly is treated as oversized_oversized_dirs, deliberately not cleared bynew_turn())Also pins LF endings for
services/hermes/*.diffso a Windows checkout can't corrupt hunks applied inside the Linux build.Validation
git apply --recount --checkpasses against the pinned checkout (3c27eb62, v0.20.0) — same command the Dockerfile runspy_compileunder the image's Python 3.11checkpoints.enabled: false(the feature has never successfully snapshotted this tree); this patch bounds the damage if it's ever re-enabledUpstream
NousResearch/hermes-agentis still unbounded as of v0.20.3 — re-check on pin bumps.🤖 Generated with Claude Code