diff --git a/CHANGELOG.md b/CHANGELOG.md index 47068ca..714f2d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -10,6 +10,11 @@ for GitHub Release notes, so every published version must have a matching ### Added +- Link nested typed JSON `tool_result`/`tool_response` objects to earlier tool + calls by exact ID, alongside the existing direct tool-role result shape. +- Add `r` in interactive JSONL/NDJSON views to toggle an exact, bounded raw + record snapshot. Follow refresh and attached/detached viewport state continue + behind the snapshot, and `r` returns to the structured view. - Add `-F`/`--follow` for interactive JSONL/NDJSON files. Follow mode opens at the committed tail without formatting the whole file, advances on appended records while attached, detaches when scrolled up, reattaches at the bottom, @@ -21,6 +26,10 @@ for GitHub Release notes, so every published version must have a matching ### Changed +- Collapse high-confidence inline base64 media in interactive JSONL/NDJSON + views to media type and validated decoded size without decoding or copying it + into the formatted display spool. Source records, redirected output, diffs, + tool arguments, media bytes, and unknown fields remain token-preserving. - Split the publishable, terminal-independent viewer engine into the new `fmtview-core` workspace crate. Format profiles and transforms, load/index models, file and diff viewer state, search/navigation, highlighting, layout, diff --git a/README.md b/README.md index 687dbf1..1041248 100644 --- a/README.md +++ b/README.md @@ -62,11 +62,16 @@ embedded markup, wrapped records, or formatted diffs. marks the object start, and its colored guide covers the body between the opening and closing braces. Boundary rows and objects without a direct role stay neutral. Role-bearing objects are also preferred for structure jumps. -- Match direct `role: tool` results to recent earlier tool calls by exact ID in - common `tool_call_id`/`tool_use_id` and contextual custom fields. Matched +- Match direct `role: tool` results and nested typed `tool_result` objects to + recent earlier tool calls by exact ID in common `tool_call_id`/`tool_use_id`, + `call_id`, and contextual custom fields. Matched calls and results reuse the existing line-number separator for compact `↓`/`↑` direction markers, without taking another column; press `t` to jump exactly between a matched pair. +- Collapse high-confidence inline base64 media in the interactive JSONL view to + a media type and validated decoded byte size, without allocating a decoded + payload. Press `r` to inspect the current record's original source text and + press it again to return to the structured view. - Preserve data semantics. JSON strings are highlighted for readability, not rewritten. - Keep large outputs responsive by indexing a temporary text file and only @@ -240,6 +245,26 @@ Other types are intentionally passthrough: - Jinja templates are indexed and highlighted as templates, but `fmtview` does not render them, evaluate includes, or rewrite template statements. +### Conversation records and inline media + +Conversation-shaped JSON remains ordinary JSON: direct `role`/`ref` fields, +typed `content` items, reasoning, runtime reminders, artifact metadata, and +unknown future fields are formatted and displayed instead of projected into a +fixed schema. Typed `tool_call`/`tool_use` and `tool_result`/`tool_response` +objects can link by exact IDs even when they are nested in `content` arrays. +Embedded argument strings keep their original JSON token spelling and escapes. + +In the interactive JSONL viewer, an unescaped-ASCII +`data:;base64,...` string is shown as a compact media summary. A +same-object `type: base64` or `type: base64url` plus +`media_type`/`mime_type` and later `data` field is handled too, as is a direct +`attachment` object under a typed image item. Standard and URL-safe alphabets +are validated separately. Metadata must precede that sibling `data` field for +streaming recognition, and arbitrary `media_type` plus `data` objects are not +guessed to be base64. Valid payloads show the decoded byte size; recognized +invalid payloads say `invalid base64` instead of claiming a size. Redirected +output never collapses these strings. + ### Following growing JSONL files Use `-F`/`--follow` with a JSONL or NDJSON file to open directly at its current @@ -319,6 +344,7 @@ The repository includes small sample files that exercise the viewer features: fmtview examples/showcase.json fmtview examples/events.jsonl fmtview examples/chat.jsonl +fmtview examples/conversation.jsonl fmtview examples/long-inline.jsonl fmtview examples/response.xml fmtview examples/page.html @@ -345,6 +371,9 @@ useful for testing structure jumps around partially observed inline content. mixed `system`/`user`/`assistant`/`tool` ordering, an object without a role, and a matched tool call/result pair for testing role scopes, relation markers, pair navigation, and chat-aware structure jumps. +`examples/conversation.jsonl` includes generic typed content, nested tool +call/result objects, an exact embedded arguments string, reasoning/runtime and +artifact metadata, and same-object base64 media. `examples/page.html` is well-formed HTML that also works as XML-compatible markup. `examples/messy.html` is loose HTML with void elements, unclosed optional @@ -376,6 +405,7 @@ Shift+Wheel horizontal scroll in nowrap mode n/N next/previous search match ]/[ next/previous structure t jump between the focused tool call/result pair +r toggle current JSONL record between structured and raw source view m toggle mouse selection mode Digits+Enter jump to a line number, for example 1200 Enter Backspace edit a pending prompt @@ -425,6 +455,18 @@ With `--follow`, this same indexed spool starts from the committed tail, loads older records backward when needed, and extends forward after source refreshes. The footer shows `follow:on`, `follow:detached`, or `follow:off`. +Press `r` on a JSONL/NDJSON record to open a bounded raw snapshot backed by its +exact immutable raw-spool range. An active search match selects its containing +record; otherwise the viewport's top record is used. The snapshot stays on that +record even if the source is appended, truncated, or replaced in the +background; returning to structured mode shows the updated stream without +changing an existing detached anchor. Raw display +uses 32 KiB visual chunks so a huge record is never copied wholesale into the +viewer frame. Search works within those chunks, but a query spanning an +artificial chunk boundary is not matched. Invalid UTF-8 is displayed lossily; +the exact bytes remain in the source/spool and are never reconstructed from +pretty output. Press `q` (or an idle `Esc`) to quit from either view. + To jump to a specific line, type the line number directly and press Enter. While a line jump is pending, the footer shows the target line; Backspace edits it and Esc cancels it. Out-of-range line numbers are clamped to the file. On fully @@ -452,7 +494,8 @@ line numbers. Tool calls and tool results reuse the existing line-number separator instead of adding a separate relation lane. The matcher only considers ID-like fields -in a tool call object or a direct `"role": "tool"` result object, then requires +in a tool call object, a direct `"role": "tool"` result object, or a nested +typed `tool_result`/`tool_response` object, then requires an exact ID value match with an earlier call from bounded recent context. On a cold jump, context recovery reads at most 4,096 preceding lines. A matched call replaces `│` with `↓`, and its result replaces it with `↑`; the @@ -543,6 +586,11 @@ rendered output in memory for browsing. - Lazy preview writes transformed records into a temporary spool and keeps compact offsets, not formatted strings, in memory. The title shows `N+` lines while the session index is still incomplete and idle time extends the index. +- JSONL raw toggles write only ingested source records to a separate immutable + temporary spool, so an open snapshot cannot change after source replacement. +- Viewer-only JSONL formatting scans recognized base64 payloads in buffered + slices, validates padding/alphabet, and writes only the media summary. It does + not allocate a decoded buffer or a payload-sized formatted output buffer. - Passthrough inputs, such as Markdown, TOML, plain text, and Jinja templates, are indexed without content rewriting. - The terminal viewer uses compact ANSI redraws and avoids repainting invisible diff --git a/crates/fmtview-core/README.md b/crates/fmtview-core/README.md index a7824a6..0453cf0 100644 --- a/crates/fmtview-core/README.md +++ b/crates/fmtview-core/README.md @@ -18,6 +18,22 @@ The core viewer owns viewport-anchor preservation and follow state through backend-neutral events, including `ViewerCommand::FollowTail` and `ViewerCommand::ToggleFollowTail`. +Record-backed views can expose exact raw-record snapshots through bounded +virtual lines. Lazy JSONL records are copied once at ingest into an immutable +raw spool, so opening raw mode is an offset lookup and source replacement +cannot change the snapshot. `FileViewer` owns the `r` structured/raw toggle, +separate raw search and wrap state, and synchronization when returning to the +structured viewport. Follow refresh and the main viewport's attached or +detached state continue behind an open raw snapshot. + +Generic conversation handling also stays in core. The JSON package recognizes +direct chat roles, contextual tool calls, direct tool-role results, and nested +typed tool results without depending on an application's storage types. The +viewer-only JSONL display path can collapse explicitly typed inline base64 +media to media type and validated decoded size without decoding the payload. +Normal transforms preserve exact tool-argument tokens, media payloads, and +unknown fields for redirected output and diffs. + Reset overlap uses a bounded, non-consuming probe of the committed source prefix. Exact IDs for the matched prefix are filtered from tail and later older batches, so large replacements remain tail-first without guessing from equal diff --git a/crates/fmtview-core/src/formats/json/tool_links.rs b/crates/fmtview-core/src/formats/json/tool_links.rs index ce5ff69..8aeba72 100644 --- a/crates/fmtview-core/src/formats/json/tool_links.rs +++ b/crates/fmtview-core/src/formats/json/tool_links.rs @@ -54,6 +54,8 @@ struct ToolContainer { start_line: usize, role_tool: bool, toolish_type: bool, + tool_result_type: bool, + contains_typed_result: bool, ids: Vec, provisional_link: Option, pending_child: Option, @@ -88,7 +90,7 @@ struct PendingCall { #[derive(Debug, Clone)] struct ToolDiscovery { start_line: usize, - link: ToolLink, + link: Option, } #[derive(Debug)] @@ -108,7 +110,7 @@ enum ToolMatchDecision { impl ToolLinkTracker { pub(crate) fn apply_line(&mut self, line: &str, line_number: usize) { - self.scan_line(line, line_number); + let _ = self.scan_line(line, line_number); } #[cfg(test)] @@ -128,7 +130,11 @@ impl ToolLinkTracker { for (offset, line) in visible_lines.iter().chain(lookahead_lines).enumerate() { let line_number = first_line.saturating_add(offset); for ToolDiscovery { start_line, link } in self.scan_line(line, line_number) { - results.insert(start_line, (offset, link)); + if let Some(link) = link { + results.insert(start_line, (offset, link)); + } else { + results.remove(&start_line); + } } if let Some((start_line, link)) = self.active_result() { results.insert(start_line, (offset, link.clone())); @@ -189,12 +195,10 @@ impl ToolLinkTracker { '{' => self.push_container(ContainerKind::Object, line_number), '[' => self.push_container(ContainerKind::Array, line_number), '}' => { - if let Some(discovery) = self.pop_container(ContainerKind::Object) { - discoveries.push(discovery); - } + discoveries.extend(self.pop_container(ContainerKind::Object)); } ']' => { - self.pop_container(ContainerKind::Array); + discoveries.extend(self.pop_container(ContainerKind::Array)); } _ => {} } @@ -223,24 +227,49 @@ impl ToolLinkTracker { start_line: line_number, role_tool: false, toolish_type: false, + tool_result_type: false, + contains_typed_result: false, ids: Vec::new(), provisional_link: None, pending_child: None, }); } - fn pop_container(&mut self, expected: ContainerKind) -> Option { + fn pop_container(&mut self, expected: ContainerKind) -> Vec { + let mut discoveries = Vec::new(); while let Some(container) = self.containers.pop() { if container.kind != expected { continue; } if expected == ContainerKind::Object { - if container.role_tool { + if (container.role_tool || container.tool_result_type) + && !container.contains_typed_result + { let link = self.link_result(container.start_line, &container.ids); - return link.map(|link| ToolDiscovery { - start_line: container.start_line, - link, - }); + let mut preserved_ancestor_fallback = false; + if container.tool_result_type { + let authoritative_child_id = container + .ids + .iter() + .any(|candidate| result_id_rank(&candidate.key) >= 3); + let preserve_explicit_fallback = !authoritative_child_id + && link + .as_ref() + .is_none_or(|link| link.status != ToolLinkStatus::Matched); + preserved_ancestor_fallback = self.suppress_ancestor_results( + &mut discoveries, + preserve_explicit_fallback, + ); + } + if let Some(link) = link + && !(preserved_ancestor_fallback && link.status != ToolLinkStatus::Matched) + { + discoveries.push(ToolDiscovery { + start_line: container.start_line, + link: Some(link), + }); + } + return discoveries; } if (container.context == ContainerContext::ToolCallObject || container.toolish_type) && !container.ids.is_empty() @@ -252,12 +281,40 @@ impl ToolLinkTracker { while self.pending_calls.len() > MAX_PENDING_CALLS { self.pending_calls.pop_front(); } - return None; + return discoveries; } } break; } - None + discoveries + } + + fn suppress_ancestor_results( + &mut self, + discoveries: &mut Vec, + preserve_explicit_fallback: bool, + ) -> bool { + let mut preserved = false; + for parent in &mut self.containers { + if preserve_explicit_fallback + && (parent.role_tool || parent.tool_result_type) + && parent + .ids + .iter() + .any(|candidate| result_id_rank(&candidate.key) >= 3) + { + preserved = true; + continue; + } + parent.contains_typed_result = true; + if parent.provisional_link.take().is_some() { + discoveries.push(ToolDiscovery { + start_line: parent.start_line, + link: None, + }); + } + } + preserved } fn apply_property(&mut self, property: JsonProperty) { @@ -268,7 +325,10 @@ impl ToolLinkTracker { if let Some(value) = property.string_value { match key.as_str() { "role" => container.role_tool = value.eq_ignore_ascii_case("tool"), - "type" => container.toolish_type = is_tool_call_type(&value), + "type" => { + container.toolish_type = is_tool_call_type(&value); + container.tool_result_type = is_tool_result_type(&value); + } _ if is_id_key(&key) && key.len() <= MAX_TOOL_ID_KEY_BYTES && value.len() <= MAX_TOOL_ID_BYTES => @@ -297,8 +357,8 @@ impl ToolLinkTracker { let Some(container) = self.containers.last() else { return; }; - let link = container - .role_tool + let link = ((container.role_tool || container.tool_result_type) + && !container.contains_typed_result) .then(|| self.preview_result(container.start_line, &container.ids)) .flatten(); if let Some(container) = self.containers.last_mut() { @@ -533,6 +593,13 @@ fn is_tool_call_type(value: &str) -> bool { ) } +fn is_tool_result_type(value: &str) -> bool { + matches!( + value.to_ascii_lowercase().as_str(), + "tool_result" | "tool_response" | "function_result" | "function_response" + ) +} + fn child_context(key: &str) -> Option { let lower = key.to_ascii_lowercase(); if matches!( diff --git a/crates/fmtview-core/src/formats/json/tool_links/tests.rs b/crates/fmtview-core/src/formats/json/tool_links/tests.rs index f9deb69..4d0660f 100644 --- a/crates/fmtview-core/src/formats/json/tool_links/tests.rs +++ b/crates/fmtview-core/src/formats/json/tool_links/tests.rs @@ -34,6 +34,21 @@ fn links_tool_use_by_shared_id() { assert_eq!(marks[1].link.as_ref().unwrap().call_line, Some(0)); } +#[test] +fn links_nested_typed_tool_call_to_nested_typed_result() { + let marks = marks(&[ + r#"{"ref":"m2","role":"assistant","content":[{"type":"tool_call","id":"call_1","name":"shell","arguments":"{\"cmd\":\"cargo test\"}"}]}"#, + r#"{"ref":"m3","role":"tool","content":[{"type":"tool_result","call_id":"call_1","content":"ok"}]}"#, + ]); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::MatchedResult); + let link = marks[1].link.as_ref().unwrap(); + assert_eq!(link.id.as_ref(), "call_1"); + assert_eq!(link.call_line, Some(0)); + assert_eq!(link.result_line, 1); +} + #[test] fn supports_custom_shared_id_field_without_matching_unrelated_objects() { let marks = marks(&[ @@ -269,3 +284,191 @@ fn provisional_result_context_matches_a_plain_id() { assert_eq!(marks[0].link.as_ref().unwrap().call_line, Some(0)); assert_eq!(marks[0].link.as_ref().unwrap().id.as_ref(), "call_1"); } + +#[test] +fn nested_typed_result_suppresses_a_role_tool_envelope_plain_id() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"m3"}"#, + "{", + r#" "id": "m3","#, + r#" "role": "tool","#, + r#" "content": ["#, + " {", + r#" "type": "tool_result","#, + r#" "call_id": "c1","#, + r#" "content": "ok""#, + " }", + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::None); + assert!(marks[2].link.is_none(), "envelope must not retain a link"); + assert_eq!(marks[6].relation, ToolRelationMark::MatchedResult); + assert_eq!(marks[6].link.as_ref().unwrap().call_line, Some(0)); + assert_eq!(tracker.pending_calls.len(), 1); + assert_eq!(tracker.pending_calls[0].line, 1); +} + +#[test] +fn typed_child_in_lookahead_retracts_visible_envelope_provisional_link() { + let prefix = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"m3"}"#, + ]; + let mut tracker = ToolLinkTracker::default(); + for (line, text) in prefix.iter().enumerate() { + tracker.apply_line(text, line); + } + let visible = [ + "{".to_owned(), + r#" "id": "m3","#.to_owned(), + r#" "role": "tool","#.to_owned(), + r#" "content": ["#.to_owned(), + ]; + let lookahead = [ + " {".to_owned(), + r#" "type": "tool_result","#.to_owned(), + r#" "call_id": "c1""#.to_owned(), + " }".to_owned(), + " ]".to_owned(), + "}".to_owned(), + ]; + + let marks = tracker.mark_lines_with_lookahead(&visible, &lookahead, prefix.len()); + + assert!(marks.iter().all(|mark| mark.link.is_none())); +} + +#[test] +fn explicit_envelope_call_id_is_a_fallback_for_an_unmatched_typed_child() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + "{", + r#" "role": "tool","#, + r#" "tool_call_id": "c1","#, + r#" "content": ["#, + " {", + r#" "type": "tool_result","#, + r#" "id": "result-123","#, + r#" "content": "ok""#, + " }", + " ]", + "}", + ]; + + let marks = marks(&lines); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::MatchedResult); + assert!(marks[5..10].iter().all(|mark| { + mark.link + .as_ref() + .is_none_or(|link| link.status == ToolLinkStatus::Matched) + })); +} + +#[test] +fn authoritative_unmatched_child_id_does_not_fall_back_to_the_envelope() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + "{", + r#" "role": "tool","#, + r#" "tool_call_id": "c1","#, + r#" "content": ["#, + r#" {"type":"tool_result","call_id":"bad"}"#, + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::None); + assert!(marks[1].link.is_none()); + assert_eq!(marks[5].link.as_ref().unwrap().id.as_ref(), "bad"); + assert_eq!( + marks[5].link.as_ref().unwrap().status, + ToolLinkStatus::Unmatched + ); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn ordinary_intermediate_toolish_ids_are_not_envelope_fallbacks() { + let lines = [ + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": [{"#, + r#" "call_id": "message-7","#, + r#" "result": {"type":"tool_result","id":"result-123"}"#, + " }]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::None); + assert!(marks[1].link.is_none()); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn idless_typed_child_still_suppresses_an_envelope_plain_id() { + let lines = [ + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": [{"type":"tool_result","content":"ok"}]"#, + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert!(marks.iter().all(|mark| mark.link.is_none())); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn multiple_typed_children_consume_only_their_exact_calls() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"c2"}"#, + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": ["#, + r#" {"type":"tool_result","call_id":"c1"},"#, + r#" {"type":"tool_result","call_id":"c2"}"#, + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[2].relation, ToolRelationMark::None); + assert_eq!(marks[7].relation, ToolRelationMark::MatchedResult); + assert_eq!(marks[8].relation, ToolRelationMark::MatchedResult); + assert!(marks[3].link.is_none()); + assert_eq!(tracker.pending_calls.len(), 1); + assert_eq!(tracker.pending_calls[0].line, 2); +} diff --git a/crates/fmtview-core/src/formats/json/transform.rs b/crates/fmtview-core/src/formats/json/transform.rs index 3d72cc9..3fab633 100644 --- a/crates/fmtview-core/src/formats/json/transform.rs +++ b/crates/fmtview-core/src/formats/json/transform.rs @@ -72,6 +72,21 @@ pub(crate) fn format_json_value( input, indent: vec![b' '; indent], offset: 0, + collapse_media: false, + }; + formatter.format(output) +} + +pub(crate) fn format_json_value_for_view( + input: R, + output: &mut W, + indent: usize, +) -> Result<()> { + let mut formatter = JsonFormatter { + input, + indent: vec![b' '; indent], + offset: 0, + collapse_media: true, }; formatter.format(output) } @@ -80,6 +95,56 @@ struct JsonFormatter { input: R, indent: Vec, offset: usize, + collapse_media: bool, +} + +#[derive(Default)] +struct ObjectMediaContext { + media_type: Option, + inherited_encoding: Option, + object_type: Option, + encoding_field: Option>, +} + +struct StringPrefix { + bytes: Vec, + ended: bool, +} + +struct Base64Stats { + encoding: Base64Encoding, + data: usize, + padding: usize, + invalid: bool, + saw_padding: bool, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Base64Encoding { + Standard, + UrlSafe, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MediaObjectType { + Encoded(Base64Encoding), + Image, + Other, +} + +impl ObjectMediaContext { + fn encoding(&self) -> Option { + self.encoding_field.unwrap_or(match self.object_type { + Some(MediaObjectType::Encoded(encoding)) => Some(encoding), + Some(MediaObjectType::Image) => Some(Base64Encoding::Standard), + Some(MediaObjectType::Other) => None, + None => self.inherited_encoding, + }) + } + + fn is_typed_image(&self) -> bool { + self.object_type == Some(MediaObjectType::Image) + } } impl JsonFormatter { @@ -113,6 +178,15 @@ impl JsonFormatter { } fn write_object(&mut self, output: &mut W, depth: usize) -> Result<()> { + self.write_object_with_media_hint(output, depth, None) + } + + fn write_object_with_media_hint( + &mut self, + output: &mut W, + depth: usize, + inherited_encoding: Option, + ) -> Result<()> { self.expect_byte(b'{')?; output.write_all(b"{")?; self.skip_ws()?; @@ -121,19 +195,58 @@ impl JsonFormatter { return Ok(()); } - self.write_pretty_object(output, depth) + self.write_pretty_object(output, depth, inherited_encoding) } - fn write_pretty_object(&mut self, output: &mut W, depth: usize) -> Result<()> { + fn write_pretty_object( + &mut self, + output: &mut W, + depth: usize, + inherited_encoding: Option, + ) -> Result<()> { + let mut media = ObjectMediaContext { + inherited_encoding, + ..ObjectMediaContext::default() + }; loop { output.write_all(b"\n")?; self.write_indent(output, depth + 1)?; self.skip_ws()?; - self.write_string(output)?; + let key = self.write_captured_string(output, 64)?; self.skip_ws()?; self.expect_byte(b':')?; output.write_all(b": ")?; - self.write_value(output, depth + 1)?; + if self.collapse_media && self.peek_byte()? == Some(b'"') { + match key.as_deref() { + Some("media_type" | "mime_type") => { + let value = self.write_captured_string(output, 128)?; + media.media_type = value.filter(|value| valid_media_type(value)); + } + Some("type") => { + let value = self.write_captured_string(output, 32)?; + media.object_type = value.as_deref().map(media_object_type); + } + Some("encoding") => { + let value = self.write_captured_string(output, 32)?; + media.encoding_field = Some(value.as_deref().and_then(base64_encoding)); + } + key => self.write_string_for_view(output, key, &media)?, + } + } else { + let typed_attachment = self.collapse_media + && key.as_deref() == Some("attachment") + && media.is_typed_image() + && self.peek_byte()? == Some(b'{'); + if typed_attachment { + self.write_object_with_media_hint( + output, + depth + 1, + Some(Base64Encoding::Standard), + )?; + } else { + self.write_value(output, depth + 1)?; + } + } match self.read_separator(b'}')? { JsonSeparator::Comma => output.write_all(b",")?, @@ -217,6 +330,278 @@ impl JsonFormatter { } } + fn write_captured_string( + &mut self, + output: &mut W, + max_bytes: usize, + ) -> Result> { + self.expect_byte(b'"')?; + output.write_all(b"\"")?; + let mut capture = Some(Vec::with_capacity(max_bytes.min(32))); + let mut utf8 = Utf8Validator::default(); + + loop { + let byte = self.next_required("unterminated JSON string")?; + output.write_all(&[byte])?; + match byte { + b'"' => { + utf8.finish()?; + return Ok(capture.and_then(|bytes| String::from_utf8(bytes).ok())); + } + b'\\' => { + let escaped = self.next_required("unterminated JSON string escape")?; + output.write_all(&[escaped])?; + match escaped { + b'"' => push_capture_byte(&mut capture, b'"', max_bytes), + b'\\' => push_capture_byte(&mut capture, b'\\', max_bytes), + b'/' => push_capture_byte(&mut capture, b'/', max_bytes), + b'b' => push_capture_byte(&mut capture, 0x08, max_bytes), + b'f' => push_capture_byte(&mut capture, 0x0c, max_bytes), + b'n' => push_capture_byte(&mut capture, b'\n', max_bytes), + b'r' => push_capture_byte(&mut capture, b'\r', max_bytes), + b't' => push_capture_byte(&mut capture, b'\t', max_bytes), + b'u' => { + let mut value = 0_u32; + for _ in 0..4 { + let hex = self.next_required("unterminated unicode escape")?; + if !hex.is_ascii_hexdigit() { + bail!("invalid unicode escape digit {}", describe_byte(hex)); + } + output.write_all(&[hex])?; + value = value + .checked_mul(16) + .and_then(|value| value.checked_add(hex_value(hex))) + .expect("four hex digits fit in u32"); + } + if let Some(ch) = char::from_u32(value) + && !(0xd800..=0xdfff).contains(&value) + { + let mut encoded = [0_u8; 4]; + push_capture_bytes( + &mut capture, + ch.encode_utf8(&mut encoded).as_bytes(), + max_bytes, + ); + } else { + capture = None; + } + } + _ => bail!("invalid JSON string escape {}", describe_byte(escaped)), + } + } + 0x00..=0x1f => bail!("unescaped control byte in JSON string"), + byte if byte.is_ascii() => { + if let Some(bytes) = capture.as_mut() { + if bytes.len() < max_bytes { + bytes.push(byte); + } else { + capture = None; + } + } + } + _ => { + capture = None; + utf8.accept(byte)?; + } + } + } + } + + fn write_string_for_view( + &mut self, + output: &mut W, + key: Option<&str>, + media: &ObjectMediaContext, + ) -> Result<()> { + self.expect_byte(b'"')?; + let prefix = self.read_safe_ascii_prefix(256)?; + let data_uri = data_uri_header(&prefix.bytes); + let sibling_media = + key == Some("data") && media.media_type.is_some() && media.encoding().is_some(); + + if key != Some("arguments") + && let Some((media_type, payload_start)) = data_uri + { + return self.write_media_summary( + output, + media_type, + Base64Encoding::Standard, + &prefix.bytes[payload_start..], + prefix.ended, + ); + } + if sibling_media { + return self.write_media_summary( + output, + media + .media_type + .as_deref() + .expect("sibling media type checked"), + media.encoding().expect("sibling media encoding checked"), + &prefix.bytes, + prefix.ended, + ); + } + + output.write_all(b"\"")?; + output.write_all(&prefix.bytes)?; + if prefix.ended { + output.write_all(b"\"")?; + return Ok(()); + } + self.write_open_string_tail(output) + } + + fn read_safe_ascii_prefix(&mut self, limit: usize) -> Result { + let mut bytes = Vec::with_capacity(limit.min(64)); + while bytes.len() < limit { + match self.peek_byte()? { + Some(b'"') => { + self.next_byte()?; + return Ok(StringPrefix { bytes, ended: true }); + } + Some(byte) if is_safe_json_string_ascii(byte) => { + self.next_byte()?; + bytes.push(byte); + } + Some(_) => break, + None => bail!("unterminated JSON string"), + } + } + Ok(StringPrefix { + bytes, + ended: false, + }) + } + + fn write_open_string_tail(&mut self, output: &mut W) -> Result<()> { + let mut utf8 = Utf8Validator::default(); + loop { + if utf8.is_idle() && self.write_safe_ascii_string_span(output)? { + continue; + } + let byte = self.next_required("unterminated JSON string")?; + output.write_all(&[byte])?; + match byte { + b'"' => { + utf8.finish()?; + return Ok(()); + } + b'\\' => { + let escaped = self.next_required("unterminated JSON string escape")?; + output.write_all(&[escaped])?; + match escaped { + b'"' | b'\\' | b'/' | b'b' | b'f' | b'n' | b'r' | b't' => {} + b'u' => { + for _ in 0..4 { + let hex = self.next_required("unterminated unicode escape")?; + if !hex.is_ascii_hexdigit() { + bail!("invalid unicode escape digit {}", describe_byte(hex)); + } + output.write_all(&[hex])?; + } + } + _ => bail!("invalid JSON string escape {}", describe_byte(escaped)), + } + } + 0x00..=0x1f => bail!("unescaped control byte in JSON string"), + _ => utf8.accept(byte)?, + } + } + } + + fn write_media_summary( + &mut self, + output: &mut W, + media_type: &str, + encoding: Base64Encoding, + prefix_payload: &[u8], + ended: bool, + ) -> Result<()> { + let mut stats = Base64Stats::new(encoding); + for &byte in prefix_payload { + stats.accept(byte); + } + if !ended { + self.scan_base64_string_tail(&mut stats)?; + } + + match stats.decoded_bytes() { + Some(bytes) => write!(output, "\"\"")?, + None => write!(output, "\"\"")?, + } + Ok(()) + } + + fn scan_base64_string_tail(&mut self, stats: &mut Base64Stats) -> Result<()> { + let mut utf8 = Utf8Validator::default(); + loop { + if utf8.is_idle() { + let len = { + let buffer = self.input.fill_buf().context("failed to read JSON input")?; + let len = buffer + .iter() + .position(|byte| { + *byte == b'"' || *byte == b'\\' || *byte < 0x20 || *byte >= 0x80 + }) + .unwrap_or(buffer.len()); + stats.accept_slice(&buffer[..len]); + len + }; + if len > 0 { + self.input.consume(len); + self.offset = self.offset.saturating_add(len); + continue; + } + } + + let byte = self.next_required("unterminated JSON string")?; + match byte { + b'"' if utf8.is_idle() => return Ok(()), + b'\\' if utf8.is_idle() => match self.consume_string_escape_ascii()? { + Some(byte) => stats.accept(byte), + None => stats.invalid = true, + }, + 0x00..=0x1f if utf8.is_idle() => { + bail!("unescaped control byte in JSON string") + } + _ => { + stats.invalid = true; + utf8.accept(byte)?; + } + } + } + } + + fn consume_string_escape_ascii(&mut self) -> Result> { + let escaped = self.next_required("unterminated JSON string escape")?; + match escaped { + b'"' => Ok(Some(b'"')), + b'\\' => Ok(Some(b'\\')), + b'/' => Ok(Some(b'/')), + b'b' => Ok(Some(0x08)), + b'f' => Ok(Some(0x0c)), + b'n' => Ok(Some(b'\n')), + b'r' => Ok(Some(b'\r')), + b't' => Ok(Some(b'\t')), + b'u' => { + let mut value = 0_u32; + for _ in 0..4 { + let hex = self.next_required("unterminated unicode escape")?; + if !hex.is_ascii_hexdigit() { + bail!("invalid unicode escape digit {}", describe_byte(hex)); + } + value = value + .checked_mul(16) + .and_then(|value| value.checked_add(hex_value(hex))) + .expect("four hex digits fit in u32"); + } + Ok(u8::try_from(value).ok()) + } + _ => bail!("invalid JSON string escape {}", describe_byte(escaped)), + } + } + fn write_safe_ascii_string_span(&mut self, output: &mut W) -> Result { let len = { let buffer = self.input.fill_buf().context("failed to read JSON input")?; @@ -364,6 +749,139 @@ impl JsonFormatter { } } +impl Base64Stats { + fn new(encoding: Base64Encoding) -> Self { + Self { + encoding, + data: 0, + padding: 0, + invalid: false, + saw_padding: false, + } + } + + fn accept(&mut self, byte: u8) { + match byte { + b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' if !self.saw_padding => { + self.data = self.data.saturating_add(1); + } + b'+' | b'/' if self.encoding == Base64Encoding::Standard && !self.saw_padding => { + self.data = self.data.saturating_add(1); + } + b'-' | b'_' if self.encoding == Base64Encoding::UrlSafe && !self.saw_padding => { + self.data = self.data.saturating_add(1); + } + b'=' if self.padding < 2 => { + self.padding += 1; + self.saw_padding = true; + } + _ => self.invalid = true, + } + } + + fn decoded_bytes(&self) -> Option { + if self.invalid { + return None; + } + let total = self.data.checked_add(self.padding)?; + if self.padding > 0 { + if total % 4 != 0 + || (self.padding == 1 && self.data % 4 != 3) + || (self.padding == 2 && self.data % 4 != 2) + { + return None; + } + return total + .checked_div(4)? + .checked_mul(3)? + .checked_sub(self.padding); + } + let full = self.data.checked_div(4)?.checked_mul(3)?; + match self.data % 4 { + 0 => Some(full), + 2 => full.checked_add(1), + 3 => full.checked_add(2), + _ => None, + } + } + + fn accept_slice(&mut self, bytes: &[u8]) { + for &byte in bytes { + self.accept(byte); + } + } +} + +fn data_uri_header(prefix: &[u8]) -> Option<(&str, usize)> { + let header_end = prefix + .windows(8) + .position(|window| window.eq_ignore_ascii_case(b";base64,"))?; + if !prefix.get(..5)?.eq_ignore_ascii_case(b"data:") { + return None; + } + let metadata = std::str::from_utf8(prefix.get(5..header_end)?).ok()?; + let media_type = metadata.split(';').next()?; + if !valid_media_type(media_type) { + return None; + } + Some((media_type, header_end + 8)) +} + +fn valid_media_type(value: &str) -> bool { + !value.is_empty() + && value.len() <= 127 + && value.contains('/') + && value.bytes().all(|byte| { + byte.is_ascii_alphanumeric() + || matches!( + byte, + b'!' | b'#' | b'$' | b'&' | b'^' | b'_' | b'.' | b'+' | b'-' | b'/' + ) + }) +} + +fn base64_encoding(value: &str) -> Option { + if value.eq_ignore_ascii_case("base64") { + Some(Base64Encoding::Standard) + } else if value.eq_ignore_ascii_case("base64url") { + Some(Base64Encoding::UrlSafe) + } else { + None + } +} + +fn media_object_type(value: &str) -> MediaObjectType { + match base64_encoding(value) { + Some(encoding) => MediaObjectType::Encoded(encoding), + None if value.eq_ignore_ascii_case("image") => MediaObjectType::Image, + None => MediaObjectType::Other, + } +} + +fn push_capture_byte(capture: &mut Option>, byte: u8, max_bytes: usize) { + push_capture_bytes(capture, &[byte], max_bytes); +} + +fn push_capture_bytes(capture: &mut Option>, bytes: &[u8], max_bytes: usize) { + let Some(buffer) = capture.as_mut() else { + return; + }; + if buffer.len().saturating_add(bytes.len()) > max_bytes { + *capture = None; + } else { + buffer.extend_from_slice(bytes); + } +} + +fn hex_value(byte: u8) -> u32 { + match byte { + b'0'..=b'9' => u32::from(byte - b'0'), + b'a'..=b'f' => u32::from(byte - b'a' + 10), + b'A'..=b'F' => u32::from(byte - b'A' + 10), + _ => unreachable!("caller validates ASCII hex digit"), + } +} + #[derive(Default)] struct Utf8Validator { remaining: u8, diff --git a/crates/fmtview-core/src/load.rs b/crates/fmtview-core/src/load.rs index 96808ad..5637cfd 100644 --- a/crates/fmtview-core/src/load.rs +++ b/crates/fmtview-core/src/load.rs @@ -4,6 +4,7 @@ mod lazy_records; mod lines; mod open; mod plan; +mod raw_record; pub(crate) mod record_stream; mod timeline; mod view_file; @@ -14,5 +15,6 @@ pub use open::{ OpenedViewFile, open_follow_view_file, open_view_file, open_view_file_with_fallback, }; pub use plan::LoadPlan; +pub(crate) use raw_record::RawRecordViewFile; pub use timeline::RecordTimelineViewFile; pub use view_file::{ViewFile, ViewFileChange}; diff --git a/crates/fmtview-core/src/load/lazy.rs b/crates/fmtview-core/src/load/lazy.rs index ebb9274..198d0dc 100644 --- a/crates/fmtview-core/src/load/lazy.rs +++ b/crates/fmtview-core/src/load/lazy.rs @@ -10,7 +10,7 @@ use memchr::memchr_iter; use tempfile::NamedTempFile; use super::{ - ViewFile, + RawRecordViewFile, ViewFile, lines::{strip_byte_line_end, strip_line_end}, }; @@ -18,6 +18,10 @@ const DIRECT_READ_LINE_BYTES: u64 = 64 * 1024; pub(crate) trait LazyProducer { fn produce(&mut self, source_offset: u64) -> Result; + + fn write_last_raw_record(&self, _output: &mut dyn Write) -> Result> { + Ok(None) + } } pub(crate) enum LazyBatch { @@ -36,7 +40,17 @@ pub(crate) struct LazyFile

{ } impl LazyFile

{ + #[cfg(test)] pub(crate) fn new(label: String, len: u64, producer: P) -> Result { + Self::with_raw_records(label, len, producer, false) + } + + pub(crate) fn with_raw_records( + label: String, + len: u64, + producer: P, + retain_raw_records: bool, + ) -> Result { Ok(Self { label, len, @@ -49,6 +63,12 @@ impl LazyFile

{ source_line_offsets: Vec::new(), complete: len == 0, units_produced: 0, + raw_spool: retain_raw_records + .then(NamedTempFile::new) + .transpose() + .context("failed to create lazy raw record spool")?, + raw_spool_len: 0, + raw_records: Vec::new(), }), }) } @@ -174,6 +194,28 @@ impl ViewFile for LazyFile

{ Ok(state.line_offsets.len() != start_lines || state.complete) } + + fn open_raw_record(&self, line: usize) -> Result>> { + let state = self.state.borrow(); + let Some(record) = state.raw_records.iter().find(|record| { + line >= record.first_line && line < record.first_line.saturating_add(record.line_count) + }) else { + return Ok(None); + }; + let Some(spool) = state.raw_spool.as_ref() else { + return Ok(None); + }; + let raw = RawRecordViewFile::new( + spool + .reopen() + .context("failed to reopen lazy raw record spool")?, + &self.label, + record.raw_offset, + record.raw_len, + line, + )?; + Ok(Some(Box::new(raw))) + } } struct LazyState

{ @@ -185,6 +227,16 @@ struct LazyState

{ source_line_offsets: Vec, complete: bool, units_produced: usize, + raw_spool: Option, + raw_spool_len: u64, + raw_records: Vec, +} + +struct LazyRawRecordRef { + first_line: usize, + line_count: usize, + raw_offset: u64, + raw_len: u64, } impl LazyState

{ @@ -203,7 +255,32 @@ impl LazyState

{ if source_bytes == 0 && bytes.is_empty() { bail!("lazy producer returned no progress"); } + let first_line = self.line_offsets.len(); + let raw_record = if let Some(raw_spool) = self.raw_spool.as_mut() { + let raw_offset = self.raw_spool_len; + let raw_len = self + .producer + .write_last_raw_record(raw_spool.as_file_mut())? + .context("lazy producer did not expose its raw record")?; + if raw_len != source_bytes { + bail!( + "lazy producer raw record length {raw_len} did not match source progress {source_bytes}" + ); + } + self.raw_spool_len = self.raw_spool_len.saturating_add(raw_len); + Some((raw_offset, raw_len)) + } else { + None + }; self.append_bytes(source_offset, &bytes)?; + if let Some((raw_offset, raw_len)) = raw_record { + self.raw_records.push(LazyRawRecordRef { + first_line, + line_count: self.line_offsets.len().saturating_sub(first_line), + raw_offset, + raw_len, + }); + } source_bytes } }; diff --git a/crates/fmtview-core/src/load/lazy_records.rs b/crates/fmtview-core/src/load/lazy_records.rs index b8c47ac..fb012dd 100644 --- a/crates/fmtview-core/src/load/lazy_records.rs +++ b/crates/fmtview-core/src/load/lazy_records.rs @@ -1,4 +1,4 @@ -use std::{cell::RefCell, fs::File, rc::Rc, time::Duration}; +use std::{cell::RefCell, fs::File, io::Write, rc::Rc, time::Duration}; use anyhow::{Context, Result}; @@ -27,10 +27,11 @@ impl LazyTransformedRecordsFile { .with_context(|| format!("failed to stat {}", source.label()))? .len(); Ok(Self { - inner: LazyFile::new( + inner: LazyFile::with_raw_records( label.clone(), len, RecordTransformProducer::new(label, file, options, Rc::clone(¬ices)), + true, )?, notices, }) @@ -79,6 +80,14 @@ impl ViewFile for LazyTransformedRecordsFile { fn take_notice(&self) -> Option { self.notices.borrow_mut().take_notice() } + + fn open_raw_record(&self, line: usize) -> Result>> { + self.inner.open_raw_record(line) + } + + fn supports_raw_records(&self) -> bool { + true + } } struct RecordTransformProducer { @@ -114,6 +123,14 @@ impl LazyProducer for RecordTransformProducer { bytes: record.bytes, }) } + + fn write_last_raw_record(&self, output: &mut dyn Write) -> Result> { + let raw = self.reader.last_raw_record(); + output + .write_all(raw) + .context("failed to write lazy raw record spool")?; + Ok(Some(raw.len() as u64)) + } } #[derive(Default)] @@ -148,7 +165,10 @@ impl RecordRecoveryNotices { #[cfg(test)] mod tests { - use std::{io::Write, time::Duration}; + use std::{ + io::{Seek, SeekFrom, Write}, + time::Duration, + }; use tempfile::NamedTempFile; @@ -248,6 +268,90 @@ mod tests { assert!(notice.contains("record 1")); } + #[test] + fn lazy_view_collapses_large_data_uri_before_spooling_formatted_lines() { + let mut input = Vec::with_capacity(1024 * 1024 + 64); + input.extend_from_slice(br#"{"content":"data:image/png;base64,"#); + input.extend(std::iter::repeat_n(b'A', 1024 * 1024)); + input.extend_from_slice(b"\"}\n"); + let (_temp, source) = temp_source(&input); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let file = LazyTransformedRecordsFile::new(&source, options).unwrap(); + + let lines = file.read_window(0, 8).unwrap(); + + assert!( + lines + .iter() + .any(|line| { line.contains("") }) + ); + assert!(lines.iter().map(String::len).sum::() < 1024); + } + + #[test] + fn lazy_view_opens_the_exact_source_record_for_a_formatted_line() { + let input = concat!( + r#"{"role":"assistant","content":[{"type":"tool_call","arguments":"{\"cmd\":\"cargo test\"}"}]}"#, + "\n" + ) + .as_bytes(); + let (_temp, source) = temp_source(input); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let file = LazyTransformedRecordsFile::new(&source, options).unwrap(); + let lines = file.read_window(0, 32).unwrap(); + let arguments_line = lines + .iter() + .position(|line| line.contains("arguments")) + .unwrap(); + + let raw = file.open_raw_record(arguments_line).unwrap().unwrap(); + let raw = raw.read_window(0, raw.line_count()).unwrap().concat(); + + assert_eq!(raw.as_bytes(), input.strip_suffix(b"\n").unwrap()); + } + + #[test] + fn lazy_raw_record_uses_the_ingested_snapshot_after_source_rewrites() { + let input = b"{\"value\":\"ORIGINAL\"}\n"; + let (mut temp, source) = temp_source(input); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let file = LazyTransformedRecordsFile::new(&source, options).unwrap(); + let lines = file.read_window(0, 16).unwrap(); + let value_line = lines + .iter() + .position(|line| line.contains("ORIGINAL")) + .unwrap(); + + temp.as_file_mut().seek(SeekFrom::Start(0)).unwrap(); + temp.as_file_mut() + .write_all(b"{\"value\":\"MUTATED!\"}\n") + .unwrap(); + temp.as_file_mut().flush().unwrap(); + + let raw = file.open_raw_record(value_line).unwrap().unwrap(); + assert_eq!( + raw.read_window(0, raw.line_count()).unwrap().concat(), + "{\"value\":\"ORIGINAL\"}" + ); + + temp.as_file_mut().set_len(0).unwrap(); + temp.as_file_mut().write_all(b"replacement\n").unwrap(); + temp.as_file_mut().flush().unwrap(); + assert_eq!( + raw.read_window(0, raw.line_count()).unwrap().concat(), + "{\"value\":\"ORIGINAL\"}" + ); + } + #[test] fn lazy_load_reads_spooled_lines_after_preload() { let (_temp, source) = temp_source(b"{\"a\":1}\n{\"b\":2}\n{\"c\":3}\n"); diff --git a/crates/fmtview-core/src/load/raw_record.rs b/crates/fmtview-core/src/load/raw_record.rs new file mode 100644 index 0000000..f081874 --- /dev/null +++ b/crates/fmtview-core/src/load/raw_record.rs @@ -0,0 +1,201 @@ +use std::{ + cell::RefCell, + fs::File, + io::{Read, Seek, SeekFrom}, +}; + +use anyhow::{Context, Result}; + +use super::ViewFile; + +const RAW_VIEW_CHUNK_BYTES: u64 = 32 * 1024; + +pub(crate) struct RawRecordViewFile { + label: String, + file: RefCell, + offset: u64, + len: u64, +} + +impl RawRecordViewFile { + pub(crate) fn new( + mut file: File, + source_label: &str, + offset: u64, + raw_len: u64, + source_line: usize, + ) -> Result { + let mut len = raw_len; + if len > 0 && byte_at(&mut file, offset.saturating_add(len - 1))? == b'\n' { + len -= 1; + } + if len > 0 && byte_at(&mut file, offset.saturating_add(len - 1))? == b'\r' { + len -= 1; + } + Ok(Self { + label: format!("{source_label} | raw record at line {}", source_line + 1), + file: RefCell::new(file), + offset, + len, + }) + } + + fn chunk_count(&self) -> usize { + usize::try_from(self.len.div_ceil(RAW_VIEW_CHUNK_BYTES)) + .unwrap_or(usize::MAX) + .max(1) + } + + fn boundary(&self, chunk: usize, file: &mut File) -> Result { + let nominal = u64::try_from(chunk) + .unwrap_or(u64::MAX) + .saturating_mul(RAW_VIEW_CHUNK_BYTES) + .min(self.len); + if nominal == 0 || nominal >= self.len { + return Ok(nominal); + } + + file.seek(SeekFrom::Start(self.offset.saturating_add(nominal))) + .context("failed to seek raw record chunk boundary")?; + let mut lookahead = [0_u8; 4]; + let read = file + .read(&mut lookahead) + .context("failed to read raw record chunk boundary")?; + let advance = lookahead[..read] + .iter() + .take_while(|byte| is_utf8_continuation(**byte)) + .take(3) + .count(); + Ok(nominal.saturating_add(advance as u64).min(self.len)) + } +} + +impl ViewFile for RawRecordViewFile { + fn label(&self) -> &str { + &self.label + } + + fn line_count(&self) -> usize { + self.chunk_count() + } + + fn byte_len(&self) -> u64 { + self.len + } + + fn byte_offset_for_line(&self, line: usize) -> u64 { + u64::try_from(line) + .unwrap_or(u64::MAX) + .saturating_mul(RAW_VIEW_CHUNK_BYTES) + .min(self.len) + } + + fn read_window(&self, start: usize, count: usize) -> Result> { + if count == 0 || start >= self.chunk_count() { + return Ok(Vec::new()); + } + let end = start.saturating_add(count).min(self.chunk_count()); + let mut file = self.file.borrow_mut(); + let mut lines = Vec::with_capacity(end - start); + for chunk in start..end { + let chunk_start = self.boundary(chunk, &mut file)?; + let chunk_end = self.boundary(chunk.saturating_add(1), &mut file)?; + let chunk_len = usize::try_from(chunk_end.saturating_sub(chunk_start)) + .context("raw record chunk was too large")?; + file.seek(SeekFrom::Start(self.offset.saturating_add(chunk_start))) + .context("failed to seek raw record spool")?; + let mut bytes = vec![0_u8; chunk_len]; + file.read_exact(&mut bytes) + .context("failed to read raw record spool")?; + lines + .push(String::from_utf8(bytes).unwrap_or_else(|error| { + String::from_utf8_lossy(error.as_bytes()).into_owned() + })); + } + Ok(lines) + } +} + +fn byte_at(file: &mut File, offset: u64) -> Result { + file.seek(SeekFrom::Start(offset)) + .context("failed to seek raw record ending")?; + let mut byte = [0_u8; 1]; + file.read_exact(&mut byte) + .context("failed to read raw record ending")?; + Ok(byte[0]) +} + +fn is_utf8_continuation(byte: u8) -> bool { + byte & 0b1100_0000 == 0b1000_0000 +} + +#[cfg(test)] +mod tests { + use std::io::Write; + + use tempfile::NamedTempFile; + + use super::*; + + #[test] + fn raw_record_view_reads_bounded_utf8_chunks_without_the_delimiter() { + let mut spool = NamedTempFile::new().unwrap(); + let text = format!( + "{{\"text\":\"{}é{}\"}}\r\n", + "a".repeat(32 * 1024 - 11), + "b".repeat(64) + ); + spool.write_all(text.as_bytes()).unwrap(); + spool.flush().unwrap(); + let view = RawRecordViewFile::new( + spool.reopen().unwrap(), + "fixture.jsonl", + 0, + text.len() as u64, + 12, + ) + .unwrap(); + + let chunks = view.read_window(0, view.line_count()).unwrap(); + + assert_eq!(chunks.concat(), text.trim_end()); + assert!(chunks.iter().all(|chunk| chunk.len() <= 32 * 1024 + 3)); + assert!(view.label().contains("raw record at line 13")); + } + + #[test] + fn invalid_utf8_continuations_adjust_a_chunk_boundary_by_at_most_three_bytes() { + let mut spool = NamedTempFile::new().unwrap(); + let mut bytes = vec![b'a'; RAW_VIEW_CHUNK_BYTES as usize]; + bytes.extend_from_slice(&[0x80; 4]); + bytes.push(b'\n'); + spool.write_all(&bytes).unwrap(); + spool.flush().unwrap(); + let view = RawRecordViewFile::new( + spool.reopen().unwrap(), + "invalid.jsonl", + 0, + bytes.len() as u64, + 0, + ) + .unwrap(); + let mut file = spool.reopen().unwrap(); + + assert_eq!( + view.boundary(1, &mut file).unwrap(), + RAW_VIEW_CHUNK_BYTES + 3 + ); + assert_eq!(view.read_window(0, 1).unwrap().len(), 1); + } + + #[test] + fn empty_raw_record_still_has_one_empty_virtual_line() { + let mut spool = NamedTempFile::new().unwrap(); + spool.write_all(b"\n").unwrap(); + spool.flush().unwrap(); + let view = RawRecordViewFile::new(spool.reopen().unwrap(), "empty.jsonl", 0, 1, 0).unwrap(); + + assert_eq!(view.line_count(), 1); + assert_eq!(view.read_window(0, 1).unwrap(), vec![""]); + } +} diff --git a/crates/fmtview-core/src/load/record_stream.rs b/crates/fmtview-core/src/load/record_stream.rs index b0f5007..78954c8 100644 --- a/crates/fmtview-core/src/load/record_stream.rs +++ b/crates/fmtview-core/src/load/record_stream.rs @@ -130,7 +130,7 @@ impl RecoveringFormattedRecordReader { return Ok(None); }; - let bytes = match transform::format_record_bytes(raw.bytes, self.options) { + let bytes = match transform::format_record_display_bytes(raw.bytes, self.options) { Ok(bytes) => bytes, Err(_) => { return Ok(Some(RecoveringFormattedRecordBytes { @@ -151,6 +151,10 @@ impl RecoveringFormattedRecordReader { diagnostic: None, })) } + + pub(crate) fn last_raw_record(&self) -> &[u8] { + &self.raw.line + } } pub(crate) struct RecoveringFormattedRecordBytes { diff --git a/crates/fmtview-core/src/load/timeline.rs b/crates/fmtview-core/src/load/timeline.rs index 817d849..ce47b7d 100644 --- a/crates/fmtview-core/src/load/timeline.rs +++ b/crates/fmtview-core/src/load/timeline.rs @@ -17,7 +17,7 @@ use crate::{ transform::{self, FormatOptions}, }; -use super::{ViewFile, ViewFileChange}; +use super::{RawRecordViewFile, ViewFile, ViewFileChange}; const INITIAL_TAIL_RECORDS: usize = 128; const INITIAL_TAIL_BYTES: usize = 4 * 1024 * 1024; @@ -133,6 +133,28 @@ impl ViewFile for RecordTimelineViewFile { fn take_notice(&self) -> Option { self.state.borrow_mut().notices.pop_front() } + + fn open_raw_record(&self, line_index: usize) -> Result>> { + let state = self.state.borrow(); + let Some(line) = state.lines.get(line_index) else { + return Ok(None); + }; + let raw = RawRecordViewFile::new( + state + .raw_spool + .reopen() + .context("failed to reopen raw timeline spool")?, + &self.label, + line.raw_offset, + u64::try_from(line.raw_len).context("raw timeline record was too large")?, + line_index, + )?; + Ok(Some(Box::new(raw))) + } + + fn supports_raw_records(&self) -> bool { + true + } } struct TimelineViewState { @@ -323,7 +345,7 @@ impl TimelineViewState { let mut parse_failures = 0_usize; let mut first_failure_offset = None; for record in records { - let bytes = match transform::format_record_bytes(&record.raw, self.options) { + let bytes = match transform::format_record_display_bytes(&record.raw, self.options) { Ok(bytes) => bytes, Err(_) => { parse_failures = parse_failures.saturating_add(1); @@ -389,6 +411,8 @@ impl TimelineViewState { spool_offset: record_start + start as u64, len: newline.saturating_sub(start), source_offset: record.id.start_offset, + raw_offset, + raw_len: record.raw.len(), }); start = newline + 1; } @@ -397,6 +421,8 @@ impl TimelineViewState { spool_offset: record_start + start as u64, len: bytes.len().saturating_sub(start), source_offset: record.id.start_offset, + raw_offset, + raw_len: record.raw.len(), }); } let record_ref = TimelineRecordRef { @@ -464,6 +490,8 @@ struct TimelineLine { spool_offset: u64, len: usize, source_offset: u64, + raw_offset: u64, + raw_len: usize, } fn record_ref_matches( diff --git a/crates/fmtview-core/src/load/view_file.rs b/crates/fmtview-core/src/load/view_file.rs index eea9c1d..4e1a0c4 100644 --- a/crates/fmtview-core/src/load/view_file.rs +++ b/crates/fmtview-core/src/load/view_file.rs @@ -46,4 +46,10 @@ pub trait ViewFile { fn take_notice(&self) -> Option { None } + fn supports_raw_records(&self) -> bool { + false + } + fn open_raw_record(&self, _line: usize) -> Result>> { + Ok(None) + } } diff --git a/crates/fmtview-core/src/perf/fixtures.rs b/crates/fmtview-core/src/perf/fixtures.rs index 5678a1b..894bb76 100644 --- a/crates/fmtview-core/src/perf/fixtures.rs +++ b/crates/fmtview-core/src/perf/fixtures.rs @@ -114,3 +114,24 @@ pub(super) fn generated_huge_string_jsonl_source( let source = InputSource::from_arg(temp.path().to_str().unwrap(), None).unwrap(); (temp, source, input_bytes) } + +pub(super) fn generated_huge_media_jsonl_source( + payload_bytes: usize, + suffix: &str, +) -> (NamedTempFile, InputSource, usize) { + let mut temp = TempFileBuilder::new().suffix(suffix).tempfile().unwrap(); + temp.write_all(br#"{"content":"data:image/png;base64,"#) + .unwrap(); + let chunk = vec![b'A'; 64 * 1024]; + let mut remaining = payload_bytes; + while remaining > 0 { + let take = remaining.min(chunk.len()); + temp.write_all(&chunk[..take]).unwrap(); + remaining -= take; + } + temp.write_all(b"\"}\n").unwrap(); + temp.flush().unwrap(); + let input_bytes = temp.as_file().metadata().unwrap().len() as usize; + let source = InputSource::from_arg(temp.path().to_str().unwrap(), None).unwrap(); + (temp, source, input_bytes) +} diff --git a/crates/fmtview-core/src/perf/load.rs b/crates/fmtview-core/src/perf/load.rs index 27f73e4..fba2f11 100644 --- a/crates/fmtview-core/src/perf/load.rs +++ b/crates/fmtview-core/src/perf/load.rs @@ -15,8 +15,9 @@ use ratatui::layout::Size; use super::{ fixtures::{ - HUGE_STRING_FRAGMENT, generated_huge_string_jsonl_source, generated_json_document_source, - generated_jsonl_source, generated_xml_document_source, + HUGE_STRING_FRAGMENT, generated_huge_media_jsonl_source, + generated_huge_string_jsonl_source, generated_json_document_source, generated_jsonl_source, + generated_xml_document_source, }, runner::{BenchCase, BenchSample}, }; @@ -52,6 +53,12 @@ pub(super) const CASES: &[BenchCase] = &[ layer: "transform+spool", run: bench_lazy_huge_string_preload_format, }, + BenchCase { + label: "lazy huge media first-window collapse", + shape: "record-stream/huge-media", + layer: "scan+collapse+spool+readback", + run: bench_lazy_huge_media_first_window, + }, BenchCase { label: "timeline tail-first open+format", shape: "growing-record-stream/tail", @@ -243,6 +250,39 @@ fn bench_lazy_huge_string_preload_format() -> BenchSample { } } +fn bench_lazy_huge_media_first_window() -> BenchSample { + const PAYLOAD_BYTES: usize = 16 * 1024 * 1024; + let (_temp, source, input_bytes) = generated_huge_media_jsonl_source(PAYLOAD_BYTES, ".jsonl"); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + + let started = Instant::now(); + let file = LazyTransformedRecordsFile::new(&source, options).unwrap(); + let lines = file.read_window(0, 8).unwrap(); + let elapsed = started.elapsed(); + + assert_eq!(file.loaded_record_count(), 1); + assert!( + lines + .iter() + .any(|line| { line.contains("") }) + ); + assert!(lines.iter().map(String::len).sum::() < 1024); + BenchSample { + elapsed, + records: 1, + items: 0, + string_bytes: PAYLOAD_BYTES, + lines: lines.len(), + indexed_lines: file.indexed_line_count(), + window_lines: lines.len(), + input_bytes, + output_bytes: lines.iter().map(String::len).sum(), + } +} + fn bench_timeline_tail_open() -> BenchSample { let temp = generated_follow_file(250_000); let input_bytes = temp.as_file().metadata().unwrap().len() as usize; diff --git a/crates/fmtview-core/src/transform.rs b/crates/fmtview-core/src/transform.rs index 98fd882..5f18635 100644 --- a/crates/fmtview-core/src/transform.rs +++ b/crates/fmtview-core/src/transform.rs @@ -7,12 +7,15 @@ mod tests; pub(crate) const IO_BUFFER_BYTES: usize = 256 * 1024; +#[cfg(test)] +pub(crate) use engine::format_record_bytes; +pub(crate) use engine::format_record_display_bytes; #[cfg(test)] pub(crate) use engine::format_record_to_string; #[cfg(test)] pub(crate) use engine::format_source_to_temp; pub(crate) use engine::transform_source_to_temp; pub(crate) use engine::trim_record_line_end; -pub(crate) use engine::{format_record_bytes, format_record_lines, parseable_record_line}; +pub(crate) use engine::{format_record_lines, parseable_record_line}; pub(crate) use types::TransformStrategy; pub use types::{FormatKind, FormatOptions}; diff --git a/crates/fmtview-core/src/transform/engine.rs b/crates/fmtview-core/src/transform/engine.rs index 0f462da..673ec76 100644 --- a/crates/fmtview-core/src/transform/engine.rs +++ b/crates/fmtview-core/src/transform/engine.rs @@ -5,7 +5,9 @@ use tempfile::NamedTempFile; use crate::formats::{ html::transform::format_html_reader, - json::transform::{format_json, format_json_value, format_jsonl, trim_line_end}, + json::transform::{ + format_json, format_json_value, format_json_value_for_view, format_jsonl, trim_line_end, + }, xml::transform::format_xml_reader, }; use crate::input::InputSource; @@ -166,6 +168,10 @@ fn record_output_capacity(input_len: usize) -> usize { input_len.saturating_mul(2).clamp(128, 8192) } +fn record_display_output_capacity(input_len: usize) -> usize { + input_len.saturating_mul(2).clamp(128, 64 * 1024) +} + pub fn trim_record_line_end(line: &[u8]) -> &[u8] { trim_line_end(line) } @@ -219,6 +225,32 @@ pub(crate) fn format_record_bytes(line: &[u8], options: FormatOptions) -> Result Ok(formatted.unwrap_or_else(|| trimmed.to_vec())) } +pub(crate) fn format_record_display_bytes(line: &[u8], options: FormatOptions) -> Result> { + let trimmed = trim_record_line_end(line); + if trim_ascii_ws(trimmed).is_empty() { + return Ok(Vec::new()); + } + + match options.kind { + FormatKind::Auto => match record_format_kind(trimmed) { + Some(FormatKind::Json) => { + let mut output = Vec::with_capacity(record_display_output_capacity(trimmed.len())); + format_json_value_for_view(Cursor::new(trimmed), &mut output, options.indent) + .context("failed to parse JSON record")?; + Ok(output) + } + _ => format_record_bytes(line, options), + }, + FormatKind::Json | FormatKind::Jsonl => { + let mut output = Vec::with_capacity(record_display_output_capacity(trimmed.len())); + format_json_value_for_view(Cursor::new(trimmed), &mut output, options.indent) + .context("failed to parse JSON record")?; + Ok(output) + } + _ => format_record_bytes(line, options), + } +} + fn record_format_kind(line: &[u8]) -> Option { match trim_ascii_ws(line).first().copied() { Some(b'<') => Some(FormatKind::Xml), diff --git a/crates/fmtview-core/src/transform/tests.rs b/crates/fmtview-core/src/transform/tests.rs index c13be25..792aa5c 100644 --- a/crates/fmtview-core/src/transform/tests.rs +++ b/crates/fmtview-core/src/transform/tests.rs @@ -10,3 +10,210 @@ fn preserves_empty_jsonl_lines() { let line = b"\n"; assert!(trim_record_line_end(line).is_empty()); } + +fn jsonl_options() -> FormatOptions { + FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + } +} + +#[test] +fn viewer_display_collapses_valid_data_uri_without_decoding_it() { + let input = br#"{"content":"data:image/png;base64,iVBORw0KGgo="}"#; + + let output = format_record_display_bytes(input, jsonl_options()).unwrap(); + let output = String::from_utf8(output).unwrap(); + + assert!(output.contains(r#""content": """#)); + assert!(!output.contains("iVBORw0KGgo=")); +} + +#[test] +fn viewer_display_collapses_same_object_base64_media() { + let input = br#"{"source":{"type":"base64","media_type":"image/png","data":"iVBORw0KGgo="}}"#; + + let output = format_record_display_bytes(input, jsonl_options()).unwrap(); + let output = String::from_utf8(output).unwrap(); + + assert!(output.contains(r#""data": """#)); +} + +#[test] +fn viewer_display_labels_invalid_high_confidence_media_without_claiming_a_size() { + let input = br#"{"content":"data:image/png;base64,not base64!"}"#; + + let output = format_record_display_bytes(input, jsonl_options()).unwrap(); + let output = String::from_utf8(output).unwrap(); + + assert!(output.contains(r#""content": """#)); + assert!(!output.contains("decoded bytes")); +} + +#[test] +fn invalid_media_escapes_are_consumed_through_the_real_string_boundary() { + let input = br#"{"content":"data:image/png;base64,bad\"still-bad\\tail","after":{"ok":true}}"#; + + let output = format_record_display_bytes(input, jsonl_options()).unwrap(); + let output = String::from_utf8(output).unwrap(); + + assert!(output.contains(r#""content": """#)); + assert!(output.contains(r#""after": {"#)); + assert!(output.contains(r#""ok": true"#)); +} + +#[test] +fn sibling_media_metadata_must_precede_data_in_the_same_object() { + let input = br#"{"source":{"data":"iVBORw0KGgo=","type":"base64","media_type":"image/png"}}"#; + + let output = format_record_display_bytes(input, jsonl_options()).unwrap(); + let output = String::from_utf8(output).unwrap(); + + assert!(output.contains(r#""data": "iVBORw0KGgo=""#)); + assert!(!output.contains(" String { + String::from_utf8(format_record_display_bytes(input, jsonl_options()).unwrap()).unwrap() +} + +#[test] +fn sibling_media_requires_explicit_encoding_or_a_typed_image_shape() { + for input in [ + br#"{"source":{"media_type":"image/png","data":"deadbeef"}}"#.as_slice(), + br#"{"source":{"media_type":"image/png","encoding":"utf8","data":"AAAAAAAAAAAAAAAA"}}"#, + br#"{"attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + br#"{"type":"text","attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + br#"{"type":"image","attachment":{"media_type":"image/png","encoding":"utf8","data":"iVBORw0KGgo="}}"#, + ] { + let output = display_json(input); + assert!(!output.contains(""), + "{same_object}" + ); + + let attachment = display_json( + br#"{"type":"image","attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + ); + assert!( + attachment.contains(""), + "{attachment}" + ); +} + +#[test] +fn escaped_media_metadata_and_payload_use_json_string_semantics() { + let output = display_json( + br#"{"t\u0079pe":"base\u00364","media\u005ftype":"image\u002fpng","data":"iVBORw0KGgo\u003d"}"#, + ); + + assert!( + output.contains(""), + "{output}" + ); + + for (payload, expected) in [ + (r"QU\u004aD", "3 decoded bytes"), + (r"AA\/A", "3 decoded bytes"), + (r"QU\u0022D", "invalid base64"), + (r"QU\u000aD", "invalid base64"), + (r"QU\ud83d\ude80D", "invalid base64"), + ] { + let input = format!(r#"{{"content":"data:image/png;base64,{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "payload={payload:?} output={output}" + ); + } +} + +#[test] +fn base64_alphabet_and_padding_follow_the_declared_encoding() { + for (payload, expected) in [ + ("TQ==", "1 decoded bytes"), + ("TQ", "1 decoded bytes"), + ("TWE=", "2 decoded bytes"), + ("TWE", "2 decoded bytes"), + ("AA+/", "3 decoded bytes"), + ("AA-_", "invalid base64"), + ("A", "invalid base64"), + ("AA=A", "invalid base64"), + ] { + let input = format!(r#"{{"content":"data:image/png;base64,{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "payload={payload:?} output={output}" + ); + } + + for (encoding, payload, expected) in [ + ("base64url", "AA-_", "3 decoded bytes"), + ("base64url", "AA+/", "invalid base64"), + ("base64", "AA+/", "3 decoded bytes"), + ("base64", "AA-_", "invalid base64"), + ] { + let input = + format!(r#"{{"media_type":"image/png","encoding":"{encoding}","data":"{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "encoding={encoding} payload={payload:?} output={output}" + ); + } +} + +#[test] +fn media_collapse_does_not_rewrite_tool_argument_strings_or_invalid_data_uris() { + let arguments = + display_json(br#"{"arguments":"data:image/png;base64,iVBORw0KGgo=","other":"ok"}"#); + assert!( + arguments.contains("data:image/png;base64,iVBORw0KGgo="), + "{arguments}" + ); + assert!(!arguments.contains(", + raw_record: Option, + content_space_changed: bool, +} + +struct RawRecordOverlay { + file: Box, + state: ViewState, + caches: ViewerCaches, + pending_prewarm: Option<(ViewPosition, usize, RenderRequest)>, } impl FileViewer { @@ -76,6 +85,7 @@ impl FileViewer { state.viewport_at_tail = true; set_file_end(&mut state, file.line_count()); } + state.raw_record_available = file.supports_raw_records(); if let Some(message) = notice { state.set_notice(message, Instant::now(), NOTICE_DURATION); } @@ -85,10 +95,15 @@ impl FileViewer { state, caches: ViewerCaches::default(), pending_prewarm: None, + raw_record: None, + content_space_changed: false, } } pub fn advance(&mut self, now: Instant) -> Result { + if let Some(raw) = self.raw_record.as_mut() { + return advance_overlay(raw, now); + } let mut dirty = false; if self.file.has_older_records() && self @@ -148,6 +163,9 @@ impl FileViewer { /// Whether the engine has a search or navigation task ready to advance. pub fn needs_immediate_advance(&self) -> bool { + if let Some(raw) = self.raw_record.as_ref() { + return raw.state.search_task.is_some() || raw.state.structure_task.is_some(); + } self.state.search_task.is_some() || self.state.structure_task.is_some() } @@ -178,7 +196,56 @@ impl FileViewer { } pub fn handle_event(&mut self, event: InputEvent, page: usize) -> ViewerAction { + if self.raw_record.is_some() { + let close_raw = self + .raw_record + .as_ref() + .is_some_and(|raw| !raw.state.has_active_prompt()) + && matches!( + event, + InputEvent::Key { + code: KeyCode::Char('r'), + modifiers + } if modifiers.is_empty() + ); + if close_raw { + let raw = self.raw_record.take().expect("raw overlay checked above"); + self.state.mouse_capture = raw.state.mouse_capture; + if self.state.wrap != raw.state.wrap { + self.state.wrap = raw.state.wrap; + reset_top_row_offset(&mut self.state); + self.state.wrap_bounds_stale = true; + self.caches = ViewerCaches::default(); + self.pending_prewarm = None; + } + self.content_space_changed = true; + return ViewerAction { + dirty: true, + ..ViewerAction::default() + }; + } + let raw = self.raw_record.as_mut().expect("raw overlay checked above"); + return handle_event_with_count( + event, + &mut raw.state, + raw.file.line_count(), + true, + page, + ); + } + let had_active_prompt = self.state.has_active_prompt(); + if !had_active_prompt + && matches!( + event, + InputEvent::Key { + code: KeyCode::Char('r'), + modifiers + } if modifiers.is_empty() + ) + { + return self.open_raw_record(); + } if self.file.is_follow_source() { if matches!(event, InputEvent::Command(ViewerCommand::FollowTail)) { return self.enable_follow_tail(); @@ -243,6 +310,9 @@ impl FileViewer { } pub fn needs_layout(&self) -> bool { + if let Some(raw) = self.raw_record.as_ref() { + return raw.state.wrap_bounds_stale; + } self.state.wrap_bounds_stale } @@ -251,6 +321,23 @@ impl FileViewer { size: Size, previous_position: Option, ) -> Result { + let previous_position = if std::mem::take(&mut self.content_space_changed) { + None + } else { + previous_position + }; + if let Some(raw) = self.raw_record.as_mut() { + let (frame, pending_prewarm) = draw_view( + raw.file.as_ref(), + FormatKind::Plain, + &mut raw.state, + &mut raw.caches, + size, + previous_position, + )?; + raw.pending_prewarm = Some(pending_prewarm); + return Ok(frame); + } let (frame, pending_prewarm) = draw_view( self.file.as_ref(), self.mode, @@ -264,6 +351,21 @@ impl FileViewer { } pub fn prewarm(&mut self) { + if let Some(raw) = self.raw_record.as_mut() { + let Some((position, visible_height, request)) = raw.pending_prewarm.take() else { + return; + }; + prewarm_render_cache( + raw.file.as_ref(), + &mut raw.caches.line, + &mut raw.caches.render, + &mut raw.caches.markdown, + position, + visible_height, + request, + ); + return; + } let Some((position, visible_height, request)) = self.pending_prewarm.take() else { return; }; @@ -317,6 +419,64 @@ impl FileViewer { } } } + + fn open_raw_record(&mut self) -> ViewerAction { + let anchor = self + .state + .search_match_target + .map(|target| target.line) + .unwrap_or(self.state.top); + match self.file.open_raw_record(anchor) { + Ok(Some(file)) => { + let mut state = ViewState { + wrap: self.state.wrap, + mouse_capture: self.state.mouse_capture, + raw_record: true, + ..ViewState::default() + }; + state.viewport_at_tail = false; + self.raw_record = Some(RawRecordOverlay { + file, + state, + caches: ViewerCaches::default(), + pending_prewarm: None, + }); + self.content_space_changed = true; + } + Ok(None) => self.state.set_footer_message( + "raw view is available for record-stream inputs", + input::FooterMessageKind::Warning, + ), + Err(error) => self.state.set_footer_message( + format!("failed to open raw record: {error:#}"), + input::FooterMessageKind::Error, + ), + } + ViewerAction { + dirty: true, + ..ViewerAction::default() + } + } +} + +fn advance_overlay(raw: &mut RawRecordOverlay, now: Instant) -> Result { + let mut dirty = absorb_file_notice(raw.file.as_ref(), &mut raw.state, now); + dirty |= raw.state.expire_footer_message(now); + if raw.state.search_task.is_some() { + dirty |= process_search_step(raw.file.as_ref(), &mut raw.state)?; + } + if raw.state.structure_task.is_some() { + dirty |= process_structure_step(raw.file.as_ref(), &mut raw.state, FormatKind::Plain)?; + } + if raw + .state + .search_index + .as_ref() + .is_some_and(|index| !index.exact) + { + dirty |= process_search_index_step(raw.file.as_ref(), &mut raw.state)?; + } + Ok(dirty) } fn event_moves_away_from_tail(event: InputEvent, wrap: bool) -> bool { diff --git a/crates/fmtview-core/src/viewer/file/input/state.rs b/crates/fmtview-core/src/viewer/file/input/state.rs index 26eda53..408163b 100644 --- a/crates/fmtview-core/src/viewer/file/input/state.rs +++ b/crates/fmtview-core/src/viewer/file/input/state.rs @@ -56,6 +56,8 @@ pub(in crate::viewer) struct ViewState { pub(in crate::viewer) follow: Option, pub(in crate::viewer) follow_reattach_pending: bool, pub(in crate::viewer) mouse_capture: bool, + pub(in crate::viewer) raw_record: bool, + pub(in crate::viewer) raw_record_available: bool, } impl Default for ViewState { @@ -91,6 +93,8 @@ impl Default for ViewState { follow: None, follow_reattach_pending: false, mouse_capture: true, + raw_record: false, + raw_record_available: false, } } } diff --git a/crates/fmtview-core/src/viewer/file/render/footer.rs b/crates/fmtview-core/src/viewer/file/render/footer.rs index 5078c7f..102690c 100644 --- a/crates/fmtview-core/src/viewer/file/render/footer.rs +++ b/crates/fmtview-core/src/viewer/file/render/footer.rs @@ -79,14 +79,21 @@ pub(in crate::viewer) fn idle_footer_text(state: &ViewState) -> String { let position = wrap_position_text(state) .map(|position| format!("{position} | ")) .unwrap_or_default(); + let record_hint = if state.raw_record { + "r structured | " + } else if state.raw_record_available { + "r raw record | " + } else { + "" + }; if let Some(count) = search_count_text(state) { return format!( - " {position}{follow_hint}search: {count} | n/N next/prev | Esc clear search | / new search | {wrap_hint} | ]/[ structure " + " {position}{follow_hint}{record_hint}search: {count} | n/N next/prev | Esc clear search | / new search | {wrap_hint} | ]/[ structure " ); } format!( - " {position}{follow_hint}{wrap_hint} | / search | ]/[ structure | t tool pair | 123 Enter line | m select | {page_hint} " + " {position}{follow_hint}{record_hint}{wrap_hint} | / search | ]/[ structure | t tool pair | 123 Enter line | m select | {page_hint} " ) } diff --git a/crates/fmtview-core/tests/headless_viewer.rs b/crates/fmtview-core/tests/headless_viewer.rs index 59d335d..732d305 100644 --- a/crates/fmtview-core/tests/headless_viewer.rs +++ b/crates/fmtview-core/tests/headless_viewer.rs @@ -31,6 +31,7 @@ fn file_engine_renders_and_searches_without_a_terminal() { let first = viewer.render(size, None).unwrap(); assert!(first.title.contains("headless.json")); + assert!(!first.footer_text.contains("raw record")); assert!(buffer_text(first).contains("alpha")); for code in [ @@ -85,6 +86,249 @@ fn file_engine_navigation_changes_backend_neutral_frame_position() { assert_eq!(next.position.top, 1); } +#[test] +fn record_engine_toggles_bounded_raw_view_and_keeps_core_interactions() { + let source = source( + "conversation.jsonl", + "{\"role\":\"assistant\",\"content\":[{\"type\":\"image\",\"source\":{\"type\":\"base64\",\"media_type\":\"image/png\",\"data\":\"iVBORw0KGgo=\"}}]}\n", + ); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(72, 12); + + let structured = viewer.render(size, None).unwrap(); + let structured_text = buffer_text(structured); + assert!(structured_text.contains("")); + assert!(!structured_text.contains("iVBORw0KGgo=")); + + assert!(send_key(&mut viewer, size, KeyCode::Char('r')).dirty); + let raw = viewer.render(size, None).unwrap(); + assert!(raw.title.contains("raw record"), "{}", raw.title); + assert!( + raw.footer_text.contains("r structured"), + "{}", + raw.footer_text + ); + assert!(buffer_text(raw).contains("iVBORw0KGgo=")); + + assert!( + viewer + .handle_event(InputEvent::Resize, FileViewer::page_for_size(size)) + .dirty + ); + assert!(send_key(&mut viewer, size, KeyCode::Char('w')).dirty); + assert!(viewer.render(size, None).unwrap().title.contains("nowrap")); + + for code in [ + KeyCode::Char('/'), + KeyCode::Char('i'), + KeyCode::Char('V'), + KeyCode::Char('B'), + KeyCode::Enter, + ] { + send_key(&mut viewer, size, code); + } + while viewer.advance(std::time::Instant::now()).unwrap() {} + let searched = viewer.render(size, None).unwrap(); + assert!( + searched.footer_text.contains("1/1 match"), + "{}", + searched.footer_text + ); + + let selection = send_key(&mut viewer, size, KeyCode::Char('m')); + assert_eq!(selection.mouse_capture, Some(false)); + + assert!(send_key(&mut viewer, size, KeyCode::Char('r')).dirty); + let structured = viewer.render(size, None).unwrap(); + assert!(!structured.title.contains("raw record")); + assert!(structured.title.contains("nowrap")); + assert!(structured.footer_text.contains("selection mode")); + assert_eq!(structured.position.row_offset, 0); + assert!(buffer_text(structured).contains("")); + + let restored = send_key(&mut viewer, size, KeyCode::Char('m')); + assert_eq!(restored.mouse_capture, Some(true)); + + send_key(&mut viewer, size, KeyCode::Char('r')); + assert!(send_key(&mut viewer, size, KeyCode::Char('q')).quit); +} + +#[test] +fn raw_toggle_prefers_the_active_search_match_record_over_the_viewport_top() { + let source = source( + "search-focus.jsonl", + concat!( + r#"{"ref":"first","role":"assistant","content":[{"type":"text","text":"before"}]}"#, + "\n", + r#"{"ref":"second","role":"assistant","content":[{"type":"text","text":"after"}]}"#, + "\n" + ), + ); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(72, 30); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('/')); + for ch in "second".chars() { + send_key(&mut viewer, size, KeyCode::Char(ch)); + } + send_key(&mut viewer, size, KeyCode::Enter); + while viewer.advance(std::time::Instant::now()).unwrap() {} + let matched = viewer.render(size, None).unwrap(); + assert_eq!(matched.position.top, 0); + assert!(buffer_text(matched).contains("second")); + + assert!(send_key(&mut viewer, size, KeyCode::Char('r')).dirty); + let raw = viewer.render(size, None).unwrap(); + assert!(raw.title.contains("raw record"), "{}", raw.title); + let raw_text = buffer_text(raw); + assert!(raw_text.contains(r#""ref":"second""#), "{raw_text}"); + assert!(!raw_text.contains(r#""ref":"first""#), "{raw_text}"); +} + +#[test] +fn raw_content_space_transitions_never_reuse_terminal_scroll_hints() { + let record = format!(r#"{{"text":"{}"}}\n"#, "payload ".repeat(2_000)); + let source = source("long-raw.jsonl", &record); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(48, 8); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('r')); + let raw = viewer + .render( + size, + Some(fmtview_core::ScrollPosition { + top: 0, + row_offset: 1, + }), + ) + .unwrap(); + assert!(raw.title.contains("raw record")); + assert_eq!(raw.scroll_hint, None); + + send_key(&mut viewer, size, KeyCode::Char('j')); + let raw_scrolled = viewer.render(size, Some(raw.position)).unwrap(); + assert!(raw_scrolled.position.row_offset > raw.position.row_offset); + + send_key(&mut viewer, size, KeyCode::Char('r')); + let structured = viewer + .render( + size, + Some(fmtview_core::ScrollPosition { + top: 0, + row_offset: 1, + }), + ) + .unwrap(); + assert!(!structured.title.contains("raw record")); + assert_eq!(structured.scroll_hint, None); + assert!(buffer_text(structured).contains("payload")); +} + +#[test] +fn pretty_nested_tool_pair_navigation_round_trips_between_records() { + let source = source( + "conversation.jsonl", + concat!( + r#"{"ref":"m2","role":"assistant","content":[{"type":"tool_call","id":"call_1","name":"shell","arguments":"{\"cmd\":\"cargo test\"}"}]}"#, + "\n", + r#"{"ref":"m3","role":"tool","content":[{"type":"tool_result","call_id":"call_1","content":"ok"}]}"#, + "\n" + ), + ); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(72, 8); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('/')); + for ch in "tool_call".chars() { + send_key(&mut viewer, size, KeyCode::Char(ch)); + } + send_key(&mut viewer, size, KeyCode::Enter); + while viewer.advance(std::time::Instant::now()).unwrap() {} + let call = viewer.render(size, None).unwrap(); + let call_top = call.position.top; + assert!(buffer_text(call).contains("tool_call")); + + assert!(send_key(&mut viewer, size, KeyCode::Char('t')).dirty); + let result = viewer.render(size, None).unwrap(); + let result_top = result.position.top; + assert!(result_top > call_top, "call={call_top} result={result_top}"); + assert!(buffer_text(result).contains("tool_result")); + + assert!(send_key(&mut viewer, size, KeyCode::Char('t')).dirty); + let returned = viewer.render(size, None).unwrap(); + assert!(returned.position.top < result_top); + assert!(buffer_text(returned).contains("tool_call")); +} + +#[test] +fn nested_tool_navigation_ignores_the_role_tool_envelope_message_id() { + let source = source( + "tool-envelope.jsonl", + concat!( + r#"{"role":"assistant","content":[{"type":"tool_call","id":"c1","name":"first"}]}"#, + "\n", + r#"{"role":"assistant","content":[{"type":"tool_call","id":"m3","name":"second"}]}"#, + "\n", + r#"{"id":"m3","role":"tool","content":[{"type":"tool_result","call_id":"c1","content":"ok"}]}"#, + "\n" + ), + ); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(72, 9); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('/')); + for ch in "tool_result".chars() { + send_key(&mut viewer, size, KeyCode::Char(ch)); + } + send_key(&mut viewer, size, KeyCode::Enter); + while viewer.advance(std::time::Instant::now()).unwrap() {} + let searched = viewer.render(size, None).unwrap(); + send_key(&mut viewer, size, KeyCode::Esc); + let result = viewer.render(size, Some(searched.position)).unwrap(); + assert!(result.footer_text.contains("c1"), "{}", result.footer_text); + assert!(!result.footer_text.contains("m3"), "{}", result.footer_text); + + assert!(send_key(&mut viewer, size, KeyCode::Char('t')).dirty); + let call = viewer.render(size, Some(result.position)).unwrap(); + let call_text = buffer_text(call); + assert!(call_text.contains("first"), "{call_text}"); + assert!(!call_text.contains("second"), "{call_text}"); +} + #[test] fn diff_engine_renders_and_navigates_without_a_terminal() { let left = source("left.json", "{\"value\":1}\n"); @@ -119,6 +363,16 @@ fn source(label: &str, text: &str) -> InputSource { InputSource::from_temp(temp, label) } +fn send_key(viewer: &mut FileViewer, size: Size, code: KeyCode) -> fmtview_core::ViewerAction { + viewer.handle_event( + InputEvent::Key { + code, + modifiers: KeyModifiers::NONE, + }, + FileViewer::page_for_size(size), + ) +} + fn buffer_text(frame: fmtview_core::RenderFrame) -> String { let mut buffer = Buffer::empty(Rect::new(0, 0, frame.area.width, frame.area.height)); render_frame_to_buffer(&mut buffer, frame); diff --git a/crates/fmtview-core/tests/record_timeline.rs b/crates/fmtview-core/tests/record_timeline.rs index fa0edf0..765e039 100644 --- a/crates/fmtview-core/tests/record_timeline.rs +++ b/crates/fmtview-core/tests/record_timeline.rs @@ -42,6 +42,171 @@ fn fake_timeline_distinguishes_pending_from_terminal_end() { ); } +#[test] +fn raw_record_mapping_survives_older_prepend() { + let (_handle, timeline) = fake_timeline((0..6).map(record)); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(2, 4096), + ) + .unwrap(); + let newest_line = find_line(&file, "record-5"); + + assert_eq!( + raw_record_text(&file, newest_line), + String::from_utf8(record(5)).unwrap().trim_end() + ); + file.load_older_records(2, 4096).unwrap(); + + assert_eq!( + raw_record_text(&file, find_line(&file, "record-5")), + String::from_utf8(record(5)).unwrap().trim_end() + ); + assert_eq!( + raw_record_text(&file, find_line(&file, "record-2")), + String::from_utf8(record(2)).unwrap().trim_end() + ); +} + +#[test] +fn raw_record_mapping_tracks_newer_append() { + let (handle, timeline) = fake_timeline((0..3).map(record)); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(4, 4096), + ) + .unwrap(); + + handle.append(record(3)); + file.refresh_records(4, 4096).unwrap(); + + assert_eq!( + raw_record_text(&file, find_line(&file, "record-3")), + String::from_utf8(record(3)).unwrap().trim_end() + ); +} + +#[test] +fn raw_record_mapping_uses_the_replacement_epoch_after_reset() { + let (handle, timeline) = fake_timeline((0..3).map(record)); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(4, 4096), + ) + .unwrap(); + + handle.replace([record(20), record(21)]); + file.refresh_records(4, 4096).unwrap(); + + assert_eq!( + raw_record_text(&file, find_line(&file, "record-21")), + String::from_utf8(record(21)).unwrap().trim_end() + ); +} + +#[test] +fn an_open_raw_record_view_remains_a_stable_snapshot_across_source_changes() { + let (handle, timeline) = fake_timeline((0..6).map(record)); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(2, 4096), + ) + .unwrap(); + let raw = file + .open_raw_record(find_line(&file, "record-5")) + .unwrap() + .unwrap(); + + file.load_older_records(2, 4096).unwrap(); + handle.append(record(6)); + file.refresh_records(4, 4096).unwrap(); + handle.replace([record(20), record(21)]); + file.refresh_records(4, 4096).unwrap(); + + assert_eq!( + raw.read_window(0, raw.line_count()).unwrap().concat(), + String::from_utf8(record(5)).unwrap().trim_end() + ); +} + +#[test] +fn follow_raw_overlay_keeps_its_snapshot_until_returning_to_updated_structure() { + let (handle, timeline) = fake_timeline([record(5)]); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(4, 4096), + ) + .unwrap(); + let mut viewer = FileViewer::new(Box::new(file), FormatKind::Jsonl, None); + let size = Size::new(60, 10); + viewer.render(size, None).unwrap(); + + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + let raw = viewer.render(size, None).unwrap(); + assert!(raw.title.contains("raw record")); + assert!(frame_text(raw).contains("record-5")); + + handle.append(record(6)); + viewer.preload().unwrap(); + let stable_after_append = viewer.render(size, None).unwrap(); + assert!(frame_text(stable_after_append).contains("record-5")); + + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + let appended = viewer.render(size, None).unwrap(); + assert!(appended.footer_text.contains("follow:on")); + assert!(frame_text(appended).contains("record-6")); + + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + handle.replace([record(20)]); + viewer.preload().unwrap(); + let stable_raw = viewer.render(size, None).unwrap(); + let stable_raw_text = frame_text(stable_raw); + assert!(stable_raw_text.contains("record-5")); + assert!(!stable_raw_text.contains("record-20")); + + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + let updated = viewer.render(size, None).unwrap(); + assert!(!updated.title.contains("raw record")); + assert!(frame_text(updated).contains("record-20")); +} + +#[test] +fn detached_follow_keeps_receiving_records_behind_a_raw_overlay() { + let (handle, timeline) = fake_timeline((0..12).map(record)); + let file = RecordTimelineViewFile::with_initial_limit( + Box::new(timeline), + JSONL, + RecordLoadLimit::new(16, 4096), + ) + .unwrap(); + let mut viewer = FileViewer::new(Box::new(file), FormatKind::Jsonl, None); + let size = Size::new(60, 8); + viewer.render(size, None).unwrap(); + viewer.handle_event(key(KeyCode::Up), FileViewer::page_for_size(size)); + let detached = viewer.render(size, None).unwrap(); + let detached_top = detached.position.top; + assert!(detached.footer_text.contains("follow:detached")); + + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + handle.append(record(12)); + viewer.preload().unwrap(); + viewer.handle_event(key(KeyCode::Char('r')), FileViewer::page_for_size(size)); + let returned = viewer.render(size, None).unwrap(); + + assert!(returned.footer_text.contains("follow:detached")); + assert_eq!(returned.position.top, detached_top); + viewer.handle_event( + InputEvent::Command(ViewerCommand::FollowTail), + FileViewer::page_for_size(size), + ); + assert!(frame_text(viewer.render(size, None).unwrap()).contains("record-12")); +} + #[test] fn follow_tail_advances_detaches_and_reattaches_headlessly() { let (handle, timeline) = fake_timeline((0..6).map(record)); @@ -1136,6 +1301,22 @@ fn frame_text(frame: fmtview_core::RenderFrame) -> String { .collect::() } +fn find_line(file: &dyn ViewFile, needle: &str) -> usize { + (0..file.line_count()) + .find(|line| { + file.read_window(*line, 1) + .unwrap() + .first() + .is_some_and(|text| text.contains(needle)) + }) + .unwrap_or_else(|| panic!("missing formatted line containing {needle:?}")) +} + +fn raw_record_text(file: &dyn ViewFile, line: usize) -> String { + let raw = file.open_raw_record(line).unwrap().unwrap(); + raw.read_window(0, raw.line_count()).unwrap().join("") +} + #[derive(Clone)] struct FakeHandle { state: Rc>, diff --git a/docs/architecture.md b/docs/architecture.md index 2385f6e..61023b6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -178,8 +178,9 @@ role-colored guide active through nested and wrapped interior rows. The guide is neutral on opening and closing brace rows so adjacent role colors stay visually separated; sibling objects without a direct role are neutral too. The same JSON format package owns a bounded tool-link tracker. It only accepts -ID-like fields from contextual tool-call objects or direct `role: tool` result -objects and matches exact values against bounded earlier pending calls. The +ID-like fields from contextual tool-call objects, direct `role: tool` result +objects, or nested typed tool-result objects and matches exact values against +bounded earlier pending calls. The viewer consumes those relations as compact endpoint directions in the existing line-number separator, footer context, and exact `t` navigation. Role and relation marks are cached in checkpointed windows so adjacent scroll positions @@ -357,6 +358,31 @@ checkpoint-committed producer can implement the same trait by mapping its own opaque stable ordering into epoch/offset identities; it needs no file methods, poll cadence, checkpoint storage rule, or terminal backend type. +Record-backed `ViewFile` implementations can expose an exact raw-record seam. +The seam retains source or raw-spool offsets and `u64` lengths, then presents a +snapshot through nominal 32 KiB virtual-line chunks with up to a three-byte +UTF-8 boundary adjustment. Opening raw mode does not copy or reconstruct the +record from formatted output. An open snapshot +keeps the selected old record stable through later prepend, append, or reset +work, while the main stream continues refreshing behind it. `FileViewer` owns +the `r` toggle plus raw search, wrap, and viewport state. An active search-match +line is the raw-record anchor; without one, the structured viewport top is the +anchor. The viewer synchronizes wrap and mouse-capture state when returning to +the structured view. The terminal package only forwards backend-neutral input +and draws the resulting frame. + +Record streams also have a viewer-only JSON display formatter for explicitly +typed inline base64 media. Unescaped-ASCII data URI headers are +self-describing; sibling `data` fields require earlier media-type and +base64-encoding metadata in the same object. A typed image item's direct +`attachment` object inherits standard base64 unless that object supplies an +explicit encoding. Standard and URL-safe alphabets are validated separately; +arbitrary media-type/data siblings are not inferred from payload characters. +The formatter validates the payload and computes decoded size by scanning +buffered input without allocating a decoded buffer or copying the payload into +the formatted spool. The ordinary transform path remains token-preserving, so +redirected output, diffs, tool arguments, and unknown fields are unchanged. + `probe_prefix` is source-neutral rather than a filesystem seek. It returns committed records from the source's true beginning under the same count/byte limit and single-record exception as directional reads, and it leaves both diff --git a/docs/performance.md b/docs/performance.md index 90d3468..ad6a6c9 100644 --- a/docs/performance.md +++ b/docs/performance.md @@ -99,6 +99,9 @@ shapes. - `record-stream`: newline-delimited records can be transformed independently. - `record-stream/huge-record`: one record dominates the cost, so first-window lazy behavior cannot hide transform/readback work. +- `record-stream/huge-media`: one record contains a large inline base64 payload, + so the viewer must validate and summarize it without decoding or writing the + payload into the formatted spool. - `growing-record-stream/tail`, `/older`, `/newer`, and `/follow`: a bidirectional timeline begins at committed EOF, loads backward, refreshes forward, or keeps an attached viewport at the new tail. @@ -133,6 +136,10 @@ Load metrics: reading the visible window back from the spool. Compare this with the first-window metric to separate transform/spool cost from long-line readback. Shape: `record-stream/huge-record`. +- `lazy huge media first-window collapse` measures opening a JSONL record with + a 16 MiB inline base64 payload, validating it with a buffered scan, and + emitting only the bounded media summary into the formatted spool. Shape: + `record-stream/huge-media`. - `timeline tail-first open+format` measures reverse tail discovery, formatting the initial bounded record batch, spooling/indexing it, and reading the last viewport. Fixture generation is outside the timed region, and correctness diff --git a/docs/visual-verification.md b/docs/visual-verification.md index 7a4d4ee..6362123 100644 --- a/docs/visual-verification.md +++ b/docs/visual-verification.md @@ -52,6 +52,17 @@ FMTVIEW_EMULATOR_FOLLOW_FILE=target/follow-demo.jsonl \ target/debug/fmtview --follow target/follow-demo.jsonl ``` +For the generic conversation fixture, select the dedicated interaction +scenario. It records nested tool-pair navigation, exact raw argument and media +records, a raw-record scroll followed by a structured redraw, the structured +media summary, search, wrap, and clean quit: + +```sh +FMTVIEW_EMULATOR_SCENARIO=conversation \ + scripts/record-emulator-demo.sh target/fmtview-emulator-recordings/conversation -- \ + target/debug/fmtview examples/conversation.jsonl +``` + For release candidates, pass the release binary explicitly after building it: ```sh diff --git a/examples/conversation.jsonl b/examples/conversation.jsonl new file mode 100644 index 0000000..f0eacb6 --- /dev/null +++ b/examples/conversation.jsonl @@ -0,0 +1,4 @@ +{"ref":"m1","role":"system","content":[{"type":"runtime_reminder","text":"Keep output concise."}],"runtime":{"model":"example-model"}} +{"ref":"m2","role":"assistant","content":[{"type":"reasoning","text":"I will run the tests."},{"type":"tool_call","id":"call_1","name":"shell","arguments":"{\"cmd\":\"cargo test --workspace\",\"note\":\"keep spaces\"}"}],"artifact":{"kind":"test-report","path":"target/report.json"}} +{"ref":"m3","role":"tool","content":[{"type":"tool_result","call_id":"call_1","content":"ok"}]} +{"ref":"m4","role":"assistant","content":[{"type":"text","text":"Tests passed."},{"type":"image","source":{"type":"base64","media_type":"image/png","data":"iVBORw0KGgo="}}]} diff --git a/scripts/record-emulator-demo.sh b/scripts/record-emulator-demo.sh index 8dd7dd4..ea4a9ab 100755 --- a/scripts/record-emulator-demo.sh +++ b/scripts/record-emulator-demo.sh @@ -26,6 +26,9 @@ instead of starting Xvfb. Set FMTVIEW_EMULATOR_FOLLOW_FILE to the JSONL file named by a --follow demo command to record append, detach, reattach, and pause/resume actions. + +Set FMTVIEW_EMULATOR_SCENARIO=conversation to record nested tool-pair +navigation, structured/raw record toggles, media collapse, search, and wrap. EOF } @@ -61,6 +64,7 @@ window_h="${FMTVIEW_EMULATOR_WINDOW_H:-820}" fps="${FMTVIEW_EMULATOR_FPS:-24}" use_existing_display="${FMTVIEW_EMULATOR_USE_EXISTING_DISPLAY:-0}" follow_file="${FMTVIEW_EMULATOR_FOLLOW_FILE:-}" +scenario="${FMTVIEW_EMULATOR_SCENARIO:-default}" class_name="fmtview-emulator-$$" display="" xvfb_pid="" @@ -255,6 +259,47 @@ if [[ -n "$follow_file" ]]; then sleep 0.8 send_key "resume_follow" "f" sleep 0.8 +elif [[ "$scenario" == "conversation" ]]; then + send_key "tool_search_open" "slash" + sleep 0.2 + send_type "tool_search_query" "tool_call" + sleep 0.2 + send_key "tool_search_enter" "Return" + sleep 0.7 + send_key "tool_jump_to_result" "t" + sleep 0.7 + send_key "tool_jump_to_call" "t" + sleep 0.7 + send_key "raw_arguments" "r" + sleep 0.7 + send_key "raw_search_open" "slash" + sleep 0.2 + send_type "raw_search_query" "cargo test" + sleep 0.2 + send_key "raw_search_enter" "Return" + sleep 0.7 + send_key "raw_scroll" "j" + sleep 0.7 + send_key "structured_from_arguments" "r" + sleep 0.7 + send_key "media_search_open" "slash" + sleep 0.2 + send_type "media_search_query" "image/png" + sleep 0.2 + send_key "media_search_enter" "Return" + sleep 0.7 + send_key "raw_media" "r" + sleep 0.7 + send_key "structured_from_media" "r" + sleep 0.7 + send_key "toggle_wrap" "w" + sleep 0.7 + send_key "metadata_search_open" "slash" + sleep 0.2 + send_type "metadata_search_query" "runtime_reminder" + sleep 0.2 + send_key "metadata_search_enter" "Return" + sleep 0.7 else send_key "scroll_down" "j" sleep 0.5 diff --git a/tests/cli.rs b/tests/cli.rs index dd638d6..a831f42 100644 --- a/tests/cli.rs +++ b/tests/cli.rs @@ -199,6 +199,23 @@ fn formats_chat_jsonl_showcase() { .stdout(predicate::str::contains(" \"role\": \"assistant\"")); } +#[test] +fn formats_generic_conversation_jsonl_without_rewriting_embedded_arguments() { + let example = + std::path::Path::new(env!("CARGO_MANIFEST_DIR")).join("examples/conversation.jsonl"); + + let mut cmd = Command::cargo_bin("fmtview").unwrap(); + let assert = cmd.arg(example).assert().success(); + let stdout = String::from_utf8(assert.get_output().stdout.clone()).unwrap(); + + assert!(stdout.contains( + r#""arguments": "{\"cmd\":\"cargo test --workspace\",\"note\":\"keep spaces\"}""# + )); + assert!(stdout.contains(r#""type": "tool_result""#)); + assert!(stdout.contains(r#""type": "runtime_reminder""#)); + assert!(stdout.contains(r#""artifact": {"#)); +} + #[test] fn plain_type_passthrough_keeps_stdout_scriptable() { let mut cmd = Command::cargo_bin("fmtview").unwrap();