feat: autoCaptureContextTurns rolling context window for extraction

[gorkem2020]

Problem

In steady state (history-carrying sessions extract every turn) each extraction transcript contains only that call's own turns. The extractor never has the surrounding conversation, so it cannot resolve references ("yes exactly, that one", "make it the second option") and either drops the fact or stores a hollow one.

Change

A rolling pair window retains conversational context across extractions:

• The last N user turns, with their assistant replies interleaved in true order, stay in the transcript across extraction calls. Retained turns are context for the extractor; the existing watermark keeps them from re-becoming extraction sources.
• N = autoCaptureContextTurns (new knob): 0 (the default) disables retention entirely and preserves stock behavior byte-for-byte; 1 to 10 sets the window size, decoupled from the extractMinMessages warm-up gate.
• The window is bounded at set time by trimTurnsToUserCap, which never trims this call's own unextracted turns out of their transcript and never leaves an orphan assistant turn ahead of the window's first user turn.
• dedupePairWindow repairs double-included pairs: a below-threshold deferral keeps content alive on two paths (the rolling buffer and the watermark rollback or ingress re-queue), and the repair collapses duplicate exchanges at pair granularity while keeping legitimately repeated user messages whose replies differ.

Tests

test/pair-window-retention.test.mjs (full-pipeline harness over a mock LLM and embedding server): retention across successful extractions, window bounding, knob-0 wipe, absent-knob default. Unit coverage for trimTurnsToUserCap and dedupePairWindow in test/auto-capture-cleanup.test.mjs; manifest schema checks pin the 0 default. Registered in the test chain and CI manifest.

Stacked on #965; the diff includes its predecessors until they merge.

[gorkem2020]

Recomposed against current master after #988 merged: the branch now carries a single commit with only this family's delta (rolling pair window: trimTurnsToUserCap, dedupePairWindow, the autoCaptureContextTurns knob and pair-window wiring). The two earlier commits are gone because their content already landed on master via #942 and #965. Full suite green locally.

[rwmjhb]

The rolling-context direction is useful, and the focused suites, full suite, build, generated output, and current CI are green. On head 9409506, however, the new feature still has three behavioral blockers:

1. The default configuration never retains assistant replies. The auto-capture loop appends assistant turns to messageLoopTurns only when captureAssistant === true (index.ts:4229-4247), while that option defaults to false. autoCaptureContextTurns is documented as retaining user turns “with their assistant replies,” but under the normal configuration the rolling window contains only user turns, so it cannot provide the assistant-side context needed to resolve references such as “yes, that one.” The integration test feeds A1/A2/A3 but asserts only U1/U2/U3. Please collect assistant replies as context independently of whether assistant messages are eligible memory sources, keep captureAssistantEligible controlling extraction attribution, and assert the replies appear in later prompts with the default setting.

2. dedupePairWindow can delete a legitimate current user turn. After prior and current turns are concatenated, an older identical-text pair plus a newer reply-less user turn is treated as replay; the later current occurrence is discarded because grouping ignores message identity/origin. A user intentionally repeating the same text is therefore absent from that extraction. Preserve the current call's occurrence (or use message IDs/origin to distinguish replay) and add a prior-pair/current-repeat regression.

3. A remember flow overwrites the accumulated rolling window. When rememberPrependedTurns is non-empty, prior pairs are intentionally not prepended for that call, but autoCaptureRecentPairTurns.set(sessionKey, finalConversationTurns) still replaces the stored window with the remember-only transcript. This silently drops older retained pairs, especially when autoCaptureContextTurns exceeds the small referent window. Keep the accumulated window independent from the protected-prefix extraction transcript and add a remember-flow retention regression.

Non-blocking lifecycle cleanup: autoCaptureRecentPairTurns is not cleared by the terminal session-end sweep, and autoCaptureSessionIds is currently dead singleton state.

[gorkem2020]

All three blockers plus the lifecycle note addressed in 12f289f:

1. Assistant replies now ride the transcript and the rolling window as context under the default configuration. They are collected independently of captureAssistant (which keeps controlling source eligibility and attribution exactly as before: the prompt already restricts extraction to user blocks in that mode) and woven back in after their surviving anchor turns, so a selector-dropped user turn drops its reply with it. The integration regression asserts the retained pair's reply appears in the NEXT prompt with captureAssistant unset.

2. dedupePairWindow is origin-aware: a current-call user turn repeating a prior pair's text is kept as the newest turn instead of being deleted as replay (same-origin replay dedup unchanged). Regression covers the prior-pair/current-repeat case including ordering.

3. A remember flow no longer overwrites the accumulated window: the stored update always composes the prior window with the call's own turns, never with the remember-shaped transcript. Regression proves older retained pairs survive a remember call and reappear afterwards.

4. Lifecycle: both the terminal boundary and the session-end sweep clear the pair window now, and the dead autoCaptureSessionIds singleton state is removed.

All three regressions are red-proofed against the pre-fix sources; full local suite green.

[rwmjhb]

Thanks for addressing the previous three blockers and lifecycle cleanup. The new regressions confirm those exact cases, and the full suite, build/dist parity, and current CI are green. On head 12f289f, deeper review found two remaining source/isolation blockers plus related state-safety gaps:

1. Retained user turns are still ordinary extraction sources, not context-only. index.ts keeps current-only conversationText, but passes finalConversationTurns containing priorPairTurns into extractAndPersist. SmartExtractor prefers those turns when rendering the transcript, and every retained user is emitted as a normal <user_message>; the extraction prompt permits candidates from every such block. The watermark controls which texts initiate a run, not what the LLM may extract. Old facts can therefore be extracted repeatedly, consume the five-candidate limit, and crowd out the current turn. Please carry source eligibility per turn, render retained user/assistant blocks explicitly as context-only, and add an end-to-end assertion that retained-only facts cannot independently produce candidates.

2. The rolling window is keyed only by sessionKey, not by agent. Literal keys such as global can be shared across agents, and missing identity falls back to the shared string unknown. The adjacent remember window already uses rememberWindowKey(agentId, sessionKey) for this reason. With retention enabled, one agent's transcript can enter another agent's extraction and become durable memory in the receiving scope. Key this state by agent plus session, disable retention for unattributable keys, and add shared-global and missing-session isolation regressions.

3. Assistant context can consume the transcript budget before the current user source. Assistant replies are woven after their anchors, and ordinary runs keep the newest rendered tail. A trailing assistant block larger than extractMaxChars can consume the entire budget and omit the preceding current user turn. Reserve budget for current source turns first; truncate/drop retained or context-only blocks before them, with an oversized-reply integration test.

4. Terminal cleanup still has alias/race gaps. The early sweep uses only ctx.sessionKey, while the later flush resolves sessionId aliases; it can also return on an empty flush before the second deletion. An in-flight capture may repopulate the map after early cleanup. Resolve the same logical key, await in-flight runs, then delete before any no-work return; cover sessionId-only and in-flight reset cases.

Additional hardening worth covering in the same state machine: overlapping captures can overwrite newer windows, same-current-call repeated text can still be deleted, and non-adjacent identical exchanges evade pair deduplication.

[gorkem2020]

Round 2 addressed in 85e7e49:

1. Source isolation: retained window turns and captureAssistant-off replies now carry an explicit context-only marking end to end. They render as context_only_user_turn / context_only_assistant_turn blocks, and the extraction prompt's format contract states context blocks are never memory sources. Regression asserts the retained turn renders as a context block, the current turn stays a normal user_message source, and the prompt carries the non-source rule.

2. Agent isolation: the rolling window is keyed by agent plus session (same shape as the adjacent remember window), and retention is disabled entirely for unattributable ids. Regressions cover two agents sharing a literal "global" session key and the unattributable case.

3. Budget priority: the bounded transcript serves SOURCE turns first (preserving the protected-referent fair share among sources) and gives context blocks only the leftover, truncating or dropping them first. Regression: a trailing context reply larger than extractMaxChars can no longer evict the current user turn.

4. Teardown: the session_end sweep resolves the sessionId alias to the same logical key the flush uses, sweeps synchronously (a next capture never sees the ended session's windows), then re-sweeps after every in-flight run settles; a per-key epoch guard voids any straggler's store. The same guard covers the overlapping-captures overwrite from the hardening list.

On the two remaining hardening notes: same-call repeated text is deduplicated by design (identical text redelivered within one payload is host replay, the origin-aware rule only protects cross-boundary repeats), and non-adjacent identical exchanges are found by the full backward scan; happy to extend either if you see a concrete failing shape.

Full local suite green including the new regressions.

[rwmjhb]

Reviewed head eda0765. The focused tests, full suite, and current CI are green, and the previous isolation fixes move this in the right direction. Four correctness issues remain:

1. A repeated current-call user turn can be silently lost. In dedupePairWindow, the reply-less duplicate after a paired duplicate is retained only when the older pair came from the prior window. For a same-call sequence such as user deploy it (id 1), assistant reply, user deploy it (id 3), the helper returns only ids 1 and 2. finalConversationTurns therefore omits the newest source turn, while the watermark still advances through the eligible text count, so that occurrence is never offered to a later extraction. Please preserve the newest/current source occurrence and add this exact same-call regression.

2. autoCapturePairWindowEpoch grows without bound under the default disabled configuration. The contextTurns === 0 path creates or increments an entry for every agent/session, but pruning exists only inside the contextTurns > 0 store path and session teardown never deletes epoch entries. Avoid allocating this state while the feature is disabled, delete it during teardown, and keep epoch/window eviction coupled so pruning cannot reset the generation while a retained window survives.

3. Context-only source isolation is prompt-only and the grounding pass removes the distinction. SmartExtractor does not validate source provenance, while buildGroundingRejudgePrompt converts context_only_* tags back to ordinary speaker tags. A candidate grounded solely in retained context can therefore pass the second judge despite violating the extraction contract. Preserve enforceable provenance through validation or otherwise reject context-only-only candidates, with an end-to-end regression.

4. A null-anchored assistant context block can disable remember-referent budget protection. weaveContextOnlyAssistantTurns inserts that block at index 0 before the strict protected-prefix scan. Because the assistant is not in referentTurnSet, protectedPrefixTurns becomes 0 and an over-budget remember that flow can evict the actual referent. Compute protection independently of the woven context position (or preserve explicit protected indices) and cover the leading-assistant case.

Requesting changes on this head.

[gorkem2020]

Round 3 addressed on head 510c29f, all four accepted:

1. Same-call repeat: dedupePairWindow now keeps a reply-less duplicate whenever the repeat itself is a current-call turn with a distinct messageId, so a user genuinely repeating themselves in one call stays in their own extraction (a literal same-turn echo still collapses). Regression: the exact user/assistant/user sequence, asserting both source occurrences reach the transcript.

2. Epoch hygiene: the disabled default no longer allocates (the cleanup path only bumps an entry that already exists, the terminal-boundary bump is gated on the feature being enabled), session teardown now deletes epoch entries in the post-settle sweep pass once awaitSessionCaptureRuns resolves, and epoch eviction is coupled to window eviction (window-less entries evict first; a window and its epoch only ever leave together). debugAutoCaptureWindowStateSizes is exported so the regressions can assert map sizes directly: disabled captures leave zero entries, teardown drains to zero.

3. Rejudge provenance: buildGroundingRejudgePrompt no longer normalizes the context_only wrappers away; the judge sees the same tagged transcript as the extractor, and its doctrine now carries the context-is-never-a-source rule (a claim resting only on context-wrapped turns must be tagged constructed). The old test that pinned the normalize-away behavior was updated to pin the preserved-tags contract instead.

4. Protected prefix: both layers now count SOURCE turns only. The caller-side scan (extracted as countProtectedReferentPrefix) skips woven context turns, and buildBoundedTranscriptWithStats spends the protected count on source blocks, so a null-anchored assistant context block at position zero can no longer void the remember-referent guarantee. Regression covers the leading-assistant weave with an over-budget transcript.

Full suite and manifest verifier green; focused file 16/16.

[rwmjhb]

Re-reviewed head 510c29f. The four issues from the previous round are addressed:

• a distinct same-call repeated user turn now survives as a source occurrence, with the exact user/assistant/user regression;
• the disabled default no longer allocates epoch state, teardown removes entries after in-flight work settles, and window/epoch eviction is coupled;
• context-only tags and the non-source rule now survive into the grounding rejudge instead of being normalized away;
• protected referent budgeting counts source turns, so a leading woven assistant context block cannot void the remember prefix.

The focused tests, full suite, and current GitHub CI pass. A fresh TypeScript compile also matches the committed dist output byte-for-byte. Approving this head.

Non-blocking follow-ups: the epoch read and guarded store are currently in one synchronous block, so the guard cannot observe an intervening mutation and the concurrency machinery can likely be simplified; avoid making debugAutoCaptureWindowStateSizes permanent public API; and gate terminal epoch allocation for an explicit unknown agent. Context-only exclusion remains model-enforced rather than deterministically source-attributed, so that residual behavior should stay covered and be revisited if the extraction schema gains source IDs.