skill-doctor: parse full Claude/Codex JSONL sessions, not just first 8MB - #88
Draft
warp-factories[bot] wants to merge 1 commit into
Draft
skill-doctor: parse full Claude/Codex JSONL sessions, not just first 8MB#88warp-factories[bot] wants to merge 1 commit into
warp-factories[bot] wants to merge 1 commit into
Conversation
collect_sessions.py truncated each Claude Code and Codex JSONL session file to MAX_FILE_BYTES (8MB) before splitting into lines and parsing, silently dropping any messages/tool calls past that offset for long sessions. The rendered transcript is already condensed independently (per-entry char limits and a head/tail cap on total entries), so the raw-file truncation was unnecessary and just caused data loss. Remove the pre-parse truncation in parse_claude_session() and parse_codex_session() so the full file is parsed; add regression tests that pad a session past the old 8MB cutoff and assert a trailing message still shows up in the parsed entries. Fixes #87
Author
|
This PR was generated with Warp. Comment    |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
collect_sessions.pytruncated each Claude Code and Codex JSONL session file toMAX_FILE_BYTES(8MB) before splitting it into lines and parsing. For sessions whose raw file exceeds 8MB, every record past that byte offset was silently dropped from the generated transcript, even though the file is well-formed JSONL.Fix
Removed the pre-parse truncation in
parse_claude_session()andparse_codex_session(). The full file is now read and every line is parsed. The generated transcript stays condensed independently via the existing per-entry character limits (MAX_MSG_CHARS,MAX_TOOL_CHARS) and the head/tail cap on total entries (MAX_TRANSCRIPT_ENTRIES), so this does not change report size, only which messages get considered.Testing
Added two regression tests in
test_collect_sessions.pythat pad a synthetic session past the old 8MB cutoff with filler tool calls, then assert a trailing message is still present in the parsed entries (one for the Claude parser, one for the Codex parser). Ran the fulltest_collect_sessions.pysuite locally — all 7 tests pass.Closes #87