Skip to content

fix(test): identify pi's compaction summarization without sniffing its prompt wording - #47

Merged
hassiebp merged 1 commit into
mainfrom
fix/compaction-pi-086
Sep 25, 2026
Merged

hassiebp merged 1 commit into
mainfrom
fix/compaction-pi-086

Conversation

@milanagm

@milanagm milanagm commented Sep 24, 2026 •

Copy link
Copy Markdown
Contributor

Why

main has been red since 2026-09-20 in the Test (pi latest, …) matrix cell only — the 0.83.0, 0.84.1 and devDependency cells, Typecheck and the tarball job all pass (last green nightly: 09-19). Every open PR inherited the same two failures, so each one looked self-inflicted:

  • traces the compaction summarization call so the session total matches pi — 0 !== 1
  • carries the compaction summary after a compaction, not the messages it replaced

This is a stale test fixture, not a plugin defect. No src/ change: the plugin traced correctly throughout.

Root cause

The mock provider identified pi's compaction-summarization request by the <conversation> XML wrapper around the history:

if (JSON.stringify(messages).includes("<conversation>")) { /* answer with a summary */ }

pi 0.87.1 (earendil-works/pi#9908, split-turn summaries being refused by Claude Fable 5.1) rewrote the turn-prefix summarization prompt to markdown headings. The main history path still uses <conversation>, which is why 0.86.0 was unaffected and 0.87.1 is not:

pi 0.86.0  compaction.js:653  `<conversation>\n${conversationText}\n</conversation>\n\n${TURN_PREFIX_…}`
pi 0.87.1  compaction.js:751  `# Conversation\n${conversationText}\n\n# Instructions\n${TURN_PREFIX_…}`

The summarization call still happened — the mock just stopped recognizing it, fell through to the ordinary stage-based agent script, and answered a summarization request with a tool call. Compaction could then never complete, so nothing was traced. Measured on unmodified main, using tools.length === 0 as ground truth for "this is a summarization call":

pi real summarization calls old detector fires Compaction spans
0.85.1 req#3 yes 1 → green
0.87.1 req#2, req#4 no 0 → 0 !== 1

Note this is invisible in the CI log: the suite prints TAP output only, never provider request bodies.

What changed

  1. Identify summarization structurally, not by wording. A summarization call reaches the provider through completeSummarization, which bypasses the agent loop and therefore declares no tools; every agent-loop call offers the built-in tools. Verified stable across pi 0.83 → 0.87.
  2. Guard the mock's script against the post-compaction rewind. stage counts visible tool results, and a compaction replaces the ones it summarized — so the script rewound, re-served the read README.md it had just summarized away, grew the context back over the threshold, and pi compacted again without end (11k+ provider calls until the harness SIGKILL). The guard keys on the mock's own summary text, not pi's.
  3. Replace the hard-coded compaction count with one span per summarization call. pi ≤0.86 sizes the context from the usage the provider reported (~1678 tok here), pi 0.87 from its own estimate (~4058 tok), so the same sandbox legitimately compacts once or twice and no single threshold value can satisfy both. One-span-per-call is the invariant this plugin actually owes, and it still catches double-tracing.

Verification

Full suite green on every matrix version, including 0.86.0, which previously livelocked:

pi 0.83.0 0.84.3 (devDep) 0.85.1 0.86.0 0.87.1 (latest)
result 128/128 128/128 128/128 128/128 128/128

Typecheck clean. Also verified end-to-end by applying this patch to two open PRs and running against latest: #41 → 141/141, #46 → 143/143 — #46's 143 is exactly the total CI reported as "143 tests, 141 pass, 2 fail".

@hassiebp
hassiebp merged commit f9e6f49 into main Sep 25, 2026
13 checks passed
@hassiebp
hassiebp deleted the fix/compaction-pi-086 branch September 25, 2026 08:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants