Skip to content

fix(compact): count and shed reasoning_content - #79

Merged
zhanghanduo merged 3 commits into
mainfrom
fix/compaction-counts-reasoning
Oct 13, 2026
Merged

zhanghanduo merged 3 commits into
mainfrom
fix/compaction-counts-reasoning

Conversation

@zhanghanduo

@zhanghanduo zhanghanduo commented Oct 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

With thinking_format: reasoning_content and thinking_in_history: true, previous assistant turns resend their reasoning, but compaction previously ignored it. A llm-hub deepseek-flash TBS run estimated about 3.5k tokens for a 66–80k-token prompt, preventing the tiered compactor from recognizing sufficient relief.

This PR:

  • Counts reasoning_content in loop, state-aware middleware, and portable SummarizationMiddleware token estimates.
  • Adds strip_stale_reasoning(messages, keep_last) without mutating the source history.
  • Retains at least the latest assistant reasoning in Tier 1 (keep_tool_result=-1 still disables Tier 1 filtering). Tool-result compression candidates and Tier 2 input retain the latest assistant reasoning.
  • Updates the change fragment and documents the keep_tool_result=0 behavior.

Proxy validation

A real llm-hub deepseek-flash three-turn tool-call chain was replayed with reasoning on all turns, only the latest turn, and no turns. All three requests returned HTTP 200 and the correct answer (1296). Prompt tokens were 599, 487, and 479 respectively, confirming the field contributes to the prompt and older reasoning can be omitted for this proxy.

This does not establish compatibility with the self-hosted DSV4 checkpoint, which was unavailable, or answer-quality effects on long tasks.

Testing

  • New reasoning-specific tests cover all three estimate paths, both portable middleware estimate branches, threshold-triggered summarization, Tier 1 retention including keep_tool_result=0 and -1, tool-result compression, and non-mutation.
  • Local full suite: 2335 passed, 1 skipped.
  • Ruff and Pyright pass.

zhanghanduo and others added 2 commits October 13, 2026 17:23
With thinking_format=reasoning_content and thinking_in_history, every prior
assistant turn carries its full reasoning on the wire, but estimate_tokens
counted only content and no compactor touched the field. A DeepSeek-flash TBS
run estimated ~3.5k tokens against a 66-80k real prompt, so Tier 2 never
won and reasoning grew unbounded.

- Both estimate_tokens implementations now include reasoning_content.
- New strip_stale_reasoning(); Tier 1 keeps reasoning only on the last
  keep_tool_result assistant turns, compress_tool_results (tool-compression
  candidates and the Tier 2 input) keeps it only on the latest turn.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@zhanghanduo

Copy link
Copy Markdown
Collaborator Author

Proxy check: stripped historical reasoning is accepted (llm-hub deepseek-flash)

I tested the PR's assumption with a real request. First I ran a 3-turn tool-call chain ((17×23) → +41 → ×3) where every assistant turn returned reasoning_content. Then I sent the same history three ways:

Variant HTTP prompt_tokens Answer
Reasoning on all 3 turns 200 599 1296
Latest turn only (this PR) 200 487 1296
Stripped entirely 200 479 1296
  • The proxy doesn't require reasoning_content on earlier turns. Even stripping it entirely gives no error and the correct answer.
  • The drop in prompt_tokens confirms that reasoning sent in history is counted toward the prompt, which is the undercount this PR fixes.

Not covered: the self-hosted DSV4 checkpoint (tbs-1004-v1-e3, service currently down) and any effect on quality in long tasks. A 3-turn arithmetic chain can't show either.

@zhanghanduo
zhanghanduo merged commit 958dc55 into main Oct 13, 2026
5 checks passed
@zhanghanduo
zhanghanduo deleted the fix/compaction-counts-reasoning branch October 13, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant