Skip to content

Commit 8011e72

Browse files
committed
fix(compaction): latch post-compact re-arm on a wide resume gap
1 parent ab31dae commit 8011e72

5 files changed

Lines changed: 143 additions & 66 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -189,12 +189,12 @@ The agent maintains an optional **`manage_tasks`** list (create/update via the h
189189

190190
When a cycle's input tokens cross a threshold, the director compacts the inference-facing history (the full run is always retained in the context store). The threshold is **model-aware** — roughly 60% of the active model's real context window — so small-window models compact early enough to avoid provider context-overflow while large-window models do not compact prematurely. The compacted prefix is **append-only across passes**: the existing compacted user turn stays byte-identical; new folds become later summary turns with a harness-inserted assistant spacer (identified by reserved `model: "harness"`, plus a visible sentinel; persisted `[compaction]` tokens without a producer still freeze) between them so the prompt head can remain in the provider KV cache. Model-emitted copies of the spacer are not frozen. Spacer-only model replies are incomplete: ChatDirector nudges, then falls through loop-protection, workflow-idle, and open-task rails rather than empty-settling with work still open. The governor covers five cases:
191191

192-
- **Threshold at a tool pause** — Once over threshold, the follow-up `infer` after a tool batch is swapped for a `compact` cycle, and inference resumes via a host continuation message. After a compact that remains over the high watermark, the governor uses **growth hysteresis** (wait for usage to grow by ~10% of the window) instead of re-arming on every cycle; dropping under 60% is not required.
192+
- **Threshold at a tool pause** — Once over threshold, the follow-up `infer` after a tool batch is swapped for a `compact` cycle, and inference resumes via a host continuation message. After a compact that remains over the high watermark, the governor latches: it does not re-arm on growth alone until usage climbs a wide resume gap (~20% of the window, a full warning band) past the post-compact measurement, or until a summarizing fold gets usage back at or under the threshold. A consecutive-compact cap holds across tool-call occupancy as a second rail.
193193
- **Idle (end-of-turn)** — An interactive turn can end with a reply and then sit idle with no tool batch to intercept; the governor requests a continuation at that pause and compacts when it arrives. An operator message that races the continuation still compacts first, then re-enters inference to answer it.
194194
- **Idle recompress past the provider TTL** — Cache expiry does not compact. Stored turns stay. When settings set `anthropicCachePrompt`, and the saved Anthropic-protocol cache write is at least 5 minutes old, the pre-inference context transform stubs old tool-result bodies on the outgoing prompt only. `turns.jsonl` is not rewritten. The setting defaults off.
195195

196196
- **Operator `/compact`** — A first-class slash command folds now without waiting for the occupancy governor. Optional trailing instructions go to the summarizer (empty uses the default structured fold) and are stored on the compact record so later auto-folds still see them. Director rebuild (`/model`, interrupt, resume) restores those instructions from the latest compact record's parameters. An in-flight turn uses the same pair-safe compact-then-continue hop as threshold compact; an idle compact shows the fold and does not start a new turn.
197-
- **Overflow recovery** — A `context_overflow` inference error would otherwise become a terminal error reply; the governor compacts and retries instead, bounded so a history the compactor cannot shrink does not loop forever. Overflow ignores hysteresis for the compact itself.
197+
- **Overflow recovery** — A `context_overflow` inference error would otherwise become a terminal error reply; the governor compacts and retries instead, bounded so a history the compactor cannot shrink does not loop forever. The overflow compact itself always fires, but the recovery counts against the same latch: only an under-threshold fold restores the budget, and the proactive path stays subject to the latched re-arm rule.
198198

199199
The compaction control flow is shaped by a reactor invariant: a `compact` action runs in its own cycle (it cannot be paired with `infer`), and **the reactor delivers no event after a compact cycle**. A director that simply emitted `compact` in place of the follow-up `infer` would leave the loop idle forever — the cause of an earlier stall. Instead the governor pairs `compact` with an emit action (`custom.compaction.continue`), and the host answers that emission by delivering a content-less inbound message (`buildCompactionContinuationMessage()`) through the serial operation queue, generation-guarded like every other deliver. A hop superseded because interrupt already bumped generation and enqueued a rebuild is re-queued onto the replacement agent rather than consumed on the outgoing liveAgent. That message adds no turn (`createInboundTurn` returns `null` for empty content) but re-enters the loop, where the director issues the follow-up `infer` against the freshly truncated history. Each emission is answered at most once (a replayed duplicate of an already-answered emission is ignored), and an unsolicited continuation with neither resume flag set answers `wait` rather than burning a billable inference. The legacy `requestContinuation` closure survives only on the sub-agent path, which delivers directly with no host emit hop; idle arming keeps the two channels exclusive (closure fires and the arming flag stays false, or no closure and the flag reports the arming) so a caller honoring both cannot double-deliver.
200200

‎src/agent/compaction.test.ts‎

Lines changed: 36 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -14,7 +14,7 @@ import {
1414
stickyExtraInstructionsFromRecords,
1515
} from "./compaction.js";
1616
import {
17-
compactionResumeDeltaFor,
17+
compactionWideResumeDeltaFor,
1818
compactionThresholdFor,
1919
} from "../provider/context-window.js";
2020
import {
@@ -147,7 +147,7 @@ function overflowError(): ReactorInboundEvent {
147147
}
148148

149149
const overThreshold = compactionThresholdFor("m") + 1;
150-
const resumeDelta = compactionResumeDeltaFor("m");
150+
const wideDelta = compactionWideResumeDeltaFor("m");
151151
const inferAction: ReactorAction[] = [{ type: "infer" }];
152152
const tenTurns = turnsOfLength(10, 1);
153153
const threeTurns = turnsOfLength(3, 1);
@@ -634,15 +634,15 @@ describe("compaction governor", () => {
634634
governor.interceptActions(toolDone(), inferAction, capabilities),
635635
).not.toBeNull();
636636

637-
// Post-compact snapshot is still over high; growth hysteresis must hold
638-
// the next arm until usage grows by resumeDelta.
637+
// Post-compact snapshot is still over high; the latch must hold the next
638+
// arm until usage climbs a wide resume gap past the snapshot.
639639
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
640640
expect(
641641
governor.interceptActions(toolDone(), inferAction, capabilities),
642642
).toBeNull();
643643
});
644644

645-
test("re-arms after usage grows by the resume delta past the last compact", () => {
645+
test("re-arms after usage grows by the wide resume gap past the last compact", () => {
646646
const governor = createCompactionGovernor(() => undefined);
647647
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
648648
expect(
@@ -655,7 +655,7 @@ describe("compaction governor", () => {
655655
).toBeNull();
656656

657657
governor.noteInferenceDone(
658-
inferenceDone(overThreshold + resumeDelta),
658+
inferenceDone(overThreshold + wideDelta),
659659
tenTurns,
660660
);
661661
const actions = governor.interceptActions(
@@ -667,7 +667,7 @@ describe("compaction governor", () => {
667667
expect(actions?.some((a) => a.type === "compact")).toBe(true);
668668
});
669669

670-
test("clears hysteresis once usage drops under the high watermark", () => {
670+
test("clears the latch once usage drops under the high watermark", () => {
671671
const governor = createCompactionGovernor(() => undefined);
672672
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
673673
expect(
@@ -695,7 +695,7 @@ describe("compaction governor", () => {
695695
expect(actions?.some((a) => a.type === "compact")).toBe(true);
696696
});
697697

698-
test("overflow still compact while hysteresis blocks the proactive path", () => {
698+
test("overflow still compacts while the latch blocks the proactive path", () => {
699699
const governor = createCompactionGovernor(() => undefined);
700700
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
701701
expect(
@@ -712,7 +712,7 @@ describe("compaction governor", () => {
712712
expect(actions?.some((a) => a.type === "compact")).toBe(true);
713713
});
714714

715-
test("consecutive threshold and idle compacts are bounded until occupancy", () => {
715+
test("consecutive threshold and idle compacts stay bounded across tool-call occupancy", () => {
716716
const governor = createCompactionGovernor(() => undefined);
717717
const echo = LEGACY_COMPACT_SPACER_TEXT;
718718
governor.noteInferenceDone(inferenceDone(overThreshold, echo), tenTurns);
@@ -722,42 +722,47 @@ describe("compaction governor", () => {
722722

723723
governor.noteInferenceDone(inferenceDone(overThreshold, echo), tenTurns);
724724
governor.noteInferenceDone(
725-
inferenceDone(overThreshold + resumeDelta, echo),
725+
inferenceDone(overThreshold + wideDelta, echo),
726726
tenTurns,
727727
);
728728
expect(
729729
governor.interceptActions(toolDone(), inferAction, capabilities),
730730
).not.toBeNull();
731731

732+
// Still over after two folds: the cap holds on both rails even as usage
733+
// keeps climbing past wide gaps ...
732734
governor.noteInferenceDone(
733-
inferenceDone(overThreshold + resumeDelta, echo),
735+
inferenceDone(overThreshold + wideDelta, echo),
734736
tenTurns,
735737
);
736738
governor.noteInferenceDone(
737-
inferenceDone(overThreshold + 2 * resumeDelta, echo),
739+
inferenceDone(overThreshold + 2 * wideDelta, echo),
738740
tenTurns,
739741
);
740742
expect(
741743
governor.interceptActions(toolDone(), inferAction, capabilities),
742744
).toBeNull();
743-
governor.noteIdleTurn(
744-
inferenceDone(overThreshold + 2 * resumeDelta, echo),
745-
[{ type: "reply", content: "done" }],
746-
);
745+
governor.noteIdleTurn(inferenceDone(overThreshold + 2 * wideDelta, echo), [
746+
{ type: "reply", content: "done" },
747+
]);
747748
expect(
748749
governor.interceptIdleContinuation(emptyMessage(), capabilities),
749750
).toBeNull();
750751

752+
// ... and tool-call occupancy does not reopen either rail.
751753
governor.noteInferenceDone(
752-
inferenceDone(overThreshold + 3 * resumeDelta, "real work"),
754+
inferenceDoneWithTools(overThreshold + 3 * wideDelta),
753755
tenTurns,
754756
);
755757
expect(
756758
governor.interceptActions(toolDone(), inferAction, capabilities),
757759
).toBeNull();
758760

761+
// Fold evidence restores the rails: under the watermark, then a fresh
762+
// crossing arms immediately with no gap required.
763+
governor.noteInferenceDone(inferenceDone(1000, "real work"), tenTurns);
759764
governor.noteInferenceDone(
760-
inferenceDoneWithTools(overThreshold + 4 * resumeDelta),
765+
inferenceDone(overThreshold, "real work"),
761766
tenTurns,
762767
);
763768
expect(
@@ -937,10 +942,9 @@ describe("compaction governor", () => {
937942
});
938943

939944
describe("post-compact above-threshold latch (CL-9006)", () => {
940-
// Wide resume gap: twice the old resume delta — a full warning band of
941-
// growth past the post-compact measurement before a still-over session may
942-
// fold again (danger-anchored; see COMPACTION_WIDE_RESUME_FRACTION).
943-
const wideDelta = resumeDelta * 2;
945+
// Half the wide gap: growth that used to re-arm under the old resume delta
946+
// must no longer re-arm on its own.
947+
const smallGrowth = Math.floor(wideDelta / 2);
944948

945949
test("resume-delta-scale growth while still over threshold does not re-arm", () => {
946950
const governor = createCompactionGovernor(() => undefined);
@@ -951,16 +955,16 @@ describe("post-compact above-threshold latch (CL-9006)", () => {
951955

952956
// Post-compact measurement stays over the high watermark: the latch sets.
953957
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
954-
// Growth by the old resume delta must NOT re-arm on its own.
958+
// Growth by half the wide gap must NOT re-arm on its own.
955959
governor.noteInferenceDone(
956-
inferenceDone(overThreshold + resumeDelta),
960+
inferenceDone(overThreshold + smallGrowth),
957961
tenTurns,
958962
);
959963
expect(
960964
governor.interceptActions(toolDone(), inferAction, capabilities),
961965
).toBeNull();
962966
// The idle path shares the same latch.
963-
governor.noteIdleTurn(inferenceDone(overThreshold + resumeDelta), [
967+
governor.noteIdleTurn(inferenceDone(overThreshold + smallGrowth), [
964968
{ type: "reply", content: "done" },
965969
]);
966970
expect(
@@ -1043,7 +1047,9 @@ describe("post-compact above-threshold latch (CL-9006)", () => {
10431047
expect(governor.resumeAfterCompact(emptyMessage())).toBe("infer");
10441048

10451049
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
1046-
expect(governor.interceptOverflow(overflowError(), capabilities)).toBeNull();
1050+
expect(
1051+
governor.interceptOverflow(overflowError(), capabilities),
1052+
).toBeNull();
10471053
});
10481054

10491055
test("an under-threshold fold restores the overflow budget", () => {
@@ -1074,7 +1080,7 @@ describe("post-compact above-threshold latch (CL-9006)", () => {
10741080
// re-arm the automatic path.
10751081
governor.noteInferenceDone(inferenceDone(overThreshold), tenTurns);
10761082
governor.noteInferenceDone(
1077-
inferenceDone(overThreshold + resumeDelta),
1083+
inferenceDone(overThreshold + smallGrowth),
10781084
tenTurns,
10791085
);
10801086
expect(
@@ -1205,10 +1211,10 @@ describe("cache expiry never folds (CL-8914)", () => {
12051211
).toBeNull();
12061212
});
12071213

1208-
test("the hysteresis gap does not fold on cache expiry", () => {
1214+
test("the latched gap does not fold on cache expiry", () => {
12091215
// After a threshold compact, a post-compact infer at the same usage
1210-
// clears `pending` via growth hysteresis — the exact re-entry where the
1211-
// removed TTL path used to fire.
1216+
// clears `pending` via the above-threshold latch — the exact re-entry
1217+
// where the removed TTL path used to fire.
12121218
let nowMs = 70_000_000;
12131219
const governor = createCompactionGovernor(
12141220
() => undefined,

‎src/agent/compaction.ts‎

Lines changed: 20 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,10 @@ import type {
77
ToolDefinition,
88
} from "@intx/types/runtime";
99
import {
10-
compactionResumeDeltaFor,
1110
compactionThresholdFor,
1211
contextTokensFromUsage,
12+
hasWideResumeGap,
13+
isAtOrUnderCompactThreshold,
1314
} from "../provider/context-window.js";
1415
import {
1516
COMPACTOR_KEEP_RECENT_TURNS,
@@ -32,10 +33,12 @@ const COMPACTOR_NAME = "pruning-compactor";
3233
const MIN_TURNS_TO_COMPACT = compactorNoOpFloor(COMPACTOR_KEEP_RECENT_TURNS);
3334
const MAX_OVERFLOW_RECOVERIES = 2;
3435
// Last-ditch bound on compact→infer→compact when the post-compact infer never
35-
// occupies the loop. Reset on tool-call occupancy or when a post-compact
36-
// measurement lands at or under the high watermark (that infer is not itself
37-
// a compact). Do not reset merely because assistant text ≠ spacer. Overflow
38-
// recoveries (above) reset on any successful inference.done instead.
36+
// gets under the high watermark. Counts consecutive threshold compacts with no
37+
// under-watermark relief between them — tool-call occupancy does not reset it,
38+
// or every few tool messages would re-enable the loop. Cleared only when a
39+
// measurement lands at or under the high watermark. Overflow recoveries
40+
// (above) share that same under-watermark reset instead of clearing on any
41+
// successful inference.done.
3942
const MAX_CONSECUTIVE_THRESHOLD_COMPACTS = 2;
4043

4144
// A compact action runs in its own reactor cycle, after which the reactor
@@ -163,10 +166,11 @@ export function createCompactionGovernor(
163166
let usingEstimate = false;
164167
let lastModel: string | undefined;
165168
let turnCount = 0;
166-
// Growth hysteresis after a compact that remained over the high watermark:
167-
// snapshot the post-compact infer's usage, then do not re-arm until usage
168-
// grows by resumeDelta. Cleared once usage drops back to or under high.
169-
// Overflow recovery ignores this and arms regardless.
169+
// Latch after a compact that remained over the high watermark: snapshot the
170+
// post-compact infer's usage, then do not re-arm on growth alone — only a
171+
// wide resume gap past the snapshot (hasWideResumeGap) or usage back at or
172+
// under the threshold clears it. Every compact sets this latch (threshold,
173+
// operator, and overflow alike); only fold evidence moves past it.
170174
let tokensAtLastCompact: number | undefined;
171175
let awaitingPostCompactMeasurement = false;
172176

@@ -192,10 +196,7 @@ export function createCompactionGovernor(
192196
const high = compactionThresholdFor(lastModel);
193197
if (contextTokens <= high) return false;
194198
if (tokensAtLastCompact !== undefined) {
195-
return (
196-
contextTokens >=
197-
tokensAtLastCompact + compactionResumeDeltaFor(lastModel)
198-
);
199+
return hasWideResumeGap(tokensAtLastCompact, contextTokens, lastModel);
199200
}
200201
return true;
201202
}
@@ -255,10 +256,6 @@ export function createCompactionGovernor(
255256
event: Extract<ReactorInboundEvent, { type: "inference.done" }>,
256257
turns: readonly ConversationTurn[],
257258
): void {
258-
overflowRecoveries = 0;
259-
if (event.turn.content.some((block) => block.type === "tool_call")) {
260-
consecutiveThresholdCompacts = 0;
261-
}
262259
syncFromTurns(turns);
263260
lastModel = event.source?.model;
264261
const reportedTokens = contextTokensFromUsage(event.usage);
@@ -270,9 +267,14 @@ export function createCompactionGovernor(
270267
tokensAtLastCompact = contextTokens;
271268
awaitingPostCompactMeasurement = false;
272269
}
273-
if (contextTokens <= compactionThresholdFor(lastModel)) {
270+
// Fold evidence: usage back at or under the threshold clears the latch
271+
// and restores both rails (consecutive threshold compacts, overflow
272+
// recoveries). Nothing else resets them — neither tool-call occupancy nor
273+
// a still-over measurement — or compact→infer→compact would loop forever.
274+
if (isAtOrUnderCompactThreshold(contextTokens, lastModel)) {
274275
tokensAtLastCompact = undefined;
275276
consecutiveThresholdCompacts = 0;
277+
overflowRecoveries = 0;
276278
}
277279
// Assign, don't OR: an under-threshold follow-up must disarm a sticky
278280
// pending left from an earlier over-threshold turn (e.g. after the

0 commit comments

Comments
 (0)