From d80e7254ee76aa44e05a2b71a8ac2d9635729edc Mon Sep 17 00:00:00 2001 From: "Xinyao (Morry) Niu" Date: Wed, 22 Jul 2026 16:54:02 +0800 Subject: [PATCH] fix(viewer): harden conversation record inspection Blockers addressed: - require explicit media typing and validate standard versus URL-safe alphabets - suppress duplicate tool-result envelopes while preserving canonical fallbacks - snapshot lazy raw records so source rewrites cannot change an open record - force a full redraw when switching structured and raw content spaces Invariants preserved: - redirected output and tool arguments remain token-preserving - media payloads stay streaming and decoded-size-only in the display spool - raw record reads remain bounded to 32 KiB plus at most 3 UTF-8 boundary bytes - terminal lifecycle stays outside fmtview-core --- CHANGELOG.md | 4 +- README.md | 26 +- crates/fmtview-core/README.md | 18 +- .../src/formats/json/tool_links.rs | 88 +++++-- .../src/formats/json/tool_links/tests.rs | 188 ++++++++++++++ .../src/formats/json/transform.rs | 243 ++++++++++++++---- crates/fmtview-core/src/load/lazy.rs | 57 ++-- crates/fmtview-core/src/load/lazy_records.rs | 55 +++- crates/fmtview-core/src/load/raw_record.rs | 37 +++ crates/fmtview-core/src/load/record_stream.rs | 4 + crates/fmtview-core/src/transform/tests.rs | 121 +++++++++ crates/fmtview-core/src/viewer/file.rs | 9 + crates/fmtview-core/tests/headless_viewer.rs | 88 +++++++ docs/architecture.md | 19 +- docs/visual-verification.md | 3 +- scripts/record-emulator-demo.sh | 2 + 16 files changed, 846 insertions(+), 116 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index d35dd86..714f2d6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,8 +28,8 @@ for GitHub Release notes, so every published version must have a matching - Collapse high-confidence inline base64 media in interactive JSONL/NDJSON views to media type and validated decoded size without decoding or copying it - into the formatted display spool. Redirected output, tool arguments, media - bytes, and unknown fields remain token-preserving. + into the formatted display spool. Source records, redirected output, diffs, + tool arguments, media bytes, and unknown fields remain token-preserving. - Split the publishable, terminal-independent viewer engine into the new `fmtview-core` workspace crate. Format profiles and transforms, load/index models, file and diff viewer state, search/navigation, highlighting, layout, diff --git a/README.md b/README.md index 0e1f348..1041248 100644 --- a/README.md +++ b/README.md @@ -254,12 +254,16 @@ fixed schema. Typed `tool_call`/`tool_use` and `tool_result`/`tool_response` objects can link by exact IDs even when they are nested in `content` arrays. Embedded argument strings keep their original JSON token spelling and escapes. -In the interactive JSONL viewer, a `data:;base64,...` string is -shown as a compact media summary. A same-object `type: base64` plus -`media_type`/`mime_type` and later `data` field is handled too. Metadata must -precede that sibling `data` field for streaming recognition. Valid payloads -show the decoded byte size; recognized invalid payloads say `invalid base64` -instead of claiming a size. Redirected output never collapses these strings. +In the interactive JSONL viewer, an unescaped-ASCII +`data:;base64,...` string is shown as a compact media summary. A +same-object `type: base64` or `type: base64url` plus +`media_type`/`mime_type` and later `data` field is handled too, as is a direct +`attachment` object under a typed image item. Standard and URL-safe alphabets +are validated separately. Metadata must precede that sibling `data` field for +streaming recognition, and arbitrary `media_type` plus `data` objects are not +guessed to be base64. Valid payloads show the decoded byte size; recognized +invalid payloads say `invalid base64` instead of claiming a size. Redirected +output never collapses these strings. ### Following growing JSONL files @@ -452,11 +456,11 @@ older records backward when needed, and extends forward after source refreshes. The footer shows `follow:on`, `follow:detached`, or `follow:off`. Press `r` on a JSONL/NDJSON record to open a bounded raw snapshot backed by its -exact source or spool range. An active search match selects its containing +exact immutable raw-spool range. An active search match selects its containing record; otherwise the viewport's top record is used. The snapshot stays on that -record while a followed source appends or resets in the background; returning -to structured mode shows the updated stream without changing an existing -detached anchor. Raw display +record even if the source is appended, truncated, or replaced in the +background; returning to structured mode shows the updated stream without +changing an existing detached anchor. Raw display uses 32 KiB visual chunks so a huge record is never copied wholesale into the viewer frame. Search works within those chunks, but a query spanning an artificial chunk boundary is not matched. Invalid UTF-8 is displayed lossily; @@ -582,6 +586,8 @@ rendered output in memory for browsing. - Lazy preview writes transformed records into a temporary spool and keeps compact offsets, not formatted strings, in memory. The title shows `N+` lines while the session index is still incomplete and idle time extends the index. +- JSONL raw toggles write only ingested source records to a separate immutable + temporary spool, so an open snapshot cannot change after source replacement. - Viewer-only JSONL formatting scans recognized base64 payloads in buffered slices, validates padding/alphabet, and writes only the media summary. It does not allocate a decoded buffer or a payload-sized formatted output buffer. diff --git a/crates/fmtview-core/README.md b/crates/fmtview-core/README.md index 9d2b8dc..0453cf0 100644 --- a/crates/fmtview-core/README.md +++ b/crates/fmtview-core/README.md @@ -19,18 +19,20 @@ backend-neutral events, including `ViewerCommand::FollowTail` and `ViewerCommand::ToggleFollowTail`. Record-backed views can expose exact raw-record snapshots through bounded -virtual lines. `FileViewer` owns the `r` structured/raw toggle, separate raw -search and wrap state, and synchronization when returning to the structured -viewport. Follow refresh and the main viewport's attached or detached state -continue behind an open raw snapshot. +virtual lines. Lazy JSONL records are copied once at ingest into an immutable +raw spool, so opening raw mode is an offset lookup and source replacement +cannot change the snapshot. `FileViewer` owns the `r` structured/raw toggle, +separate raw search and wrap state, and synchronization when returning to the +structured viewport. Follow refresh and the main viewport's attached or +detached state continue behind an open raw snapshot. Generic conversation handling also stays in core. The JSON package recognizes direct chat roles, contextual tool calls, direct tool-role results, and nested typed tool results without depending on an application's storage types. The -viewer-only JSONL display path can collapse high-confidence inline base64 media -to media type and validated decoded size without decoding the payload. Normal -transforms preserve exact tool-argument tokens, media payloads, and unknown -fields for redirected output and diffs. +viewer-only JSONL display path can collapse explicitly typed inline base64 +media to media type and validated decoded size without decoding the payload. +Normal transforms preserve exact tool-argument tokens, media payloads, and +unknown fields for redirected output and diffs. Reset overlap uses a bounded, non-consuming probe of the committed source prefix. Exact IDs for the matched prefix are filtered from tail and later older diff --git a/crates/fmtview-core/src/formats/json/tool_links.rs b/crates/fmtview-core/src/formats/json/tool_links.rs index f000c96..8aeba72 100644 --- a/crates/fmtview-core/src/formats/json/tool_links.rs +++ b/crates/fmtview-core/src/formats/json/tool_links.rs @@ -55,6 +55,7 @@ struct ToolContainer { role_tool: bool, toolish_type: bool, tool_result_type: bool, + contains_typed_result: bool, ids: Vec, provisional_link: Option, pending_child: Option, @@ -89,7 +90,7 @@ struct PendingCall { #[derive(Debug, Clone)] struct ToolDiscovery { start_line: usize, - link: ToolLink, + link: Option, } #[derive(Debug)] @@ -109,7 +110,7 @@ enum ToolMatchDecision { impl ToolLinkTracker { pub(crate) fn apply_line(&mut self, line: &str, line_number: usize) { - self.scan_line(line, line_number); + let _ = self.scan_line(line, line_number); } #[cfg(test)] @@ -129,7 +130,11 @@ impl ToolLinkTracker { for (offset, line) in visible_lines.iter().chain(lookahead_lines).enumerate() { let line_number = first_line.saturating_add(offset); for ToolDiscovery { start_line, link } in self.scan_line(line, line_number) { - results.insert(start_line, (offset, link)); + if let Some(link) = link { + results.insert(start_line, (offset, link)); + } else { + results.remove(&start_line); + } } if let Some((start_line, link)) = self.active_result() { results.insert(start_line, (offset, link.clone())); @@ -190,12 +195,10 @@ impl ToolLinkTracker { '{' => self.push_container(ContainerKind::Object, line_number), '[' => self.push_container(ContainerKind::Array, line_number), '}' => { - if let Some(discovery) = self.pop_container(ContainerKind::Object) { - discoveries.push(discovery); - } + discoveries.extend(self.pop_container(ContainerKind::Object)); } ']' => { - self.pop_container(ContainerKind::Array); + discoveries.extend(self.pop_container(ContainerKind::Array)); } _ => {} } @@ -225,24 +228,48 @@ impl ToolLinkTracker { role_tool: false, toolish_type: false, tool_result_type: false, + contains_typed_result: false, ids: Vec::new(), provisional_link: None, pending_child: None, }); } - fn pop_container(&mut self, expected: ContainerKind) -> Option { + fn pop_container(&mut self, expected: ContainerKind) -> Vec { + let mut discoveries = Vec::new(); while let Some(container) = self.containers.pop() { if container.kind != expected { continue; } if expected == ContainerKind::Object { - if container.role_tool || container.tool_result_type { + if (container.role_tool || container.tool_result_type) + && !container.contains_typed_result + { let link = self.link_result(container.start_line, &container.ids); - return link.map(|link| ToolDiscovery { - start_line: container.start_line, - link, - }); + let mut preserved_ancestor_fallback = false; + if container.tool_result_type { + let authoritative_child_id = container + .ids + .iter() + .any(|candidate| result_id_rank(&candidate.key) >= 3); + let preserve_explicit_fallback = !authoritative_child_id + && link + .as_ref() + .is_none_or(|link| link.status != ToolLinkStatus::Matched); + preserved_ancestor_fallback = self.suppress_ancestor_results( + &mut discoveries, + preserve_explicit_fallback, + ); + } + if let Some(link) = link + && !(preserved_ancestor_fallback && link.status != ToolLinkStatus::Matched) + { + discoveries.push(ToolDiscovery { + start_line: container.start_line, + link: Some(link), + }); + } + return discoveries; } if (container.context == ContainerContext::ToolCallObject || container.toolish_type) && !container.ids.is_empty() @@ -254,12 +281,40 @@ impl ToolLinkTracker { while self.pending_calls.len() > MAX_PENDING_CALLS { self.pending_calls.pop_front(); } - return None; + return discoveries; } } break; } - None + discoveries + } + + fn suppress_ancestor_results( + &mut self, + discoveries: &mut Vec, + preserve_explicit_fallback: bool, + ) -> bool { + let mut preserved = false; + for parent in &mut self.containers { + if preserve_explicit_fallback + && (parent.role_tool || parent.tool_result_type) + && parent + .ids + .iter() + .any(|candidate| result_id_rank(&candidate.key) >= 3) + { + preserved = true; + continue; + } + parent.contains_typed_result = true; + if parent.provisional_link.take().is_some() { + discoveries.push(ToolDiscovery { + start_line: parent.start_line, + link: None, + }); + } + } + preserved } fn apply_property(&mut self, property: JsonProperty) { @@ -302,7 +357,8 @@ impl ToolLinkTracker { let Some(container) = self.containers.last() else { return; }; - let link = (container.role_tool || container.tool_result_type) + let link = ((container.role_tool || container.tool_result_type) + && !container.contains_typed_result) .then(|| self.preview_result(container.start_line, &container.ids)) .flatten(); if let Some(container) = self.containers.last_mut() { diff --git a/crates/fmtview-core/src/formats/json/tool_links/tests.rs b/crates/fmtview-core/src/formats/json/tool_links/tests.rs index 833bf60..4d0660f 100644 --- a/crates/fmtview-core/src/formats/json/tool_links/tests.rs +++ b/crates/fmtview-core/src/formats/json/tool_links/tests.rs @@ -284,3 +284,191 @@ fn provisional_result_context_matches_a_plain_id() { assert_eq!(marks[0].link.as_ref().unwrap().call_line, Some(0)); assert_eq!(marks[0].link.as_ref().unwrap().id.as_ref(), "call_1"); } + +#[test] +fn nested_typed_result_suppresses_a_role_tool_envelope_plain_id() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"m3"}"#, + "{", + r#" "id": "m3","#, + r#" "role": "tool","#, + r#" "content": ["#, + " {", + r#" "type": "tool_result","#, + r#" "call_id": "c1","#, + r#" "content": "ok""#, + " }", + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::None); + assert!(marks[2].link.is_none(), "envelope must not retain a link"); + assert_eq!(marks[6].relation, ToolRelationMark::MatchedResult); + assert_eq!(marks[6].link.as_ref().unwrap().call_line, Some(0)); + assert_eq!(tracker.pending_calls.len(), 1); + assert_eq!(tracker.pending_calls[0].line, 1); +} + +#[test] +fn typed_child_in_lookahead_retracts_visible_envelope_provisional_link() { + let prefix = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"m3"}"#, + ]; + let mut tracker = ToolLinkTracker::default(); + for (line, text) in prefix.iter().enumerate() { + tracker.apply_line(text, line); + } + let visible = [ + "{".to_owned(), + r#" "id": "m3","#.to_owned(), + r#" "role": "tool","#.to_owned(), + r#" "content": ["#.to_owned(), + ]; + let lookahead = [ + " {".to_owned(), + r#" "type": "tool_result","#.to_owned(), + r#" "call_id": "c1""#.to_owned(), + " }".to_owned(), + " ]".to_owned(), + "}".to_owned(), + ]; + + let marks = tracker.mark_lines_with_lookahead(&visible, &lookahead, prefix.len()); + + assert!(marks.iter().all(|mark| mark.link.is_none())); +} + +#[test] +fn explicit_envelope_call_id_is_a_fallback_for_an_unmatched_typed_child() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + "{", + r#" "role": "tool","#, + r#" "tool_call_id": "c1","#, + r#" "content": ["#, + " {", + r#" "type": "tool_result","#, + r#" "id": "result-123","#, + r#" "content": "ok""#, + " }", + " ]", + "}", + ]; + + let marks = marks(&lines); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::MatchedResult); + assert!(marks[5..10].iter().all(|mark| { + mark.link + .as_ref() + .is_none_or(|link| link.status == ToolLinkStatus::Matched) + })); +} + +#[test] +fn authoritative_unmatched_child_id_does_not_fall_back_to_the_envelope() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + "{", + r#" "role": "tool","#, + r#" "tool_call_id": "c1","#, + r#" "content": ["#, + r#" {"type":"tool_result","call_id":"bad"}"#, + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::None); + assert!(marks[1].link.is_none()); + assert_eq!(marks[5].link.as_ref().unwrap().id.as_ref(), "bad"); + assert_eq!( + marks[5].link.as_ref().unwrap().status, + ToolLinkStatus::Unmatched + ); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn ordinary_intermediate_toolish_ids_are_not_envelope_fallbacks() { + let lines = [ + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": [{"#, + r#" "call_id": "message-7","#, + r#" "result": {"type":"tool_result","id":"result-123"}"#, + " }]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::None); + assert!(marks[1].link.is_none()); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn idless_typed_child_still_suppresses_an_envelope_plain_id() { + let lines = [ + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": [{"type":"tool_result","content":"ok"}]"#, + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert!(marks.iter().all(|mark| mark.link.is_none())); + assert_eq!(tracker.pending_calls.len(), 1); +} + +#[test] +fn multiple_typed_children_consume_only_their_exact_calls() { + let lines = [ + r#"{"type":"tool_call","id":"c1"}"#, + r#"{"type":"tool_call","id":"c2"}"#, + r#"{"type":"tool_call","id":"message-7"}"#, + "{", + r#" "id": "message-7","#, + r#" "role": "tool","#, + r#" "content": ["#, + r#" {"type":"tool_result","call_id":"c1"},"#, + r#" {"type":"tool_result","call_id":"c2"}"#, + " ]", + "}", + ]; + let owned = lines.map(str::to_owned); + let mut tracker = ToolLinkTracker::default(); + + let marks = tracker.mark_lines(&owned, 0); + + assert_eq!(marks[0].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[1].relation, ToolRelationMark::MatchedCall); + assert_eq!(marks[2].relation, ToolRelationMark::None); + assert_eq!(marks[7].relation, ToolRelationMark::MatchedResult); + assert_eq!(marks[8].relation, ToolRelationMark::MatchedResult); + assert!(marks[3].link.is_none()); + assert_eq!(tracker.pending_calls.len(), 1); + assert_eq!(tracker.pending_calls[0].line, 2); +} diff --git a/crates/fmtview-core/src/formats/json/transform.rs b/crates/fmtview-core/src/formats/json/transform.rs index 51639a8..3fab633 100644 --- a/crates/fmtview-core/src/formats/json/transform.rs +++ b/crates/fmtview-core/src/formats/json/transform.rs @@ -101,7 +101,9 @@ struct JsonFormatter { #[derive(Default)] struct ObjectMediaContext { media_type: Option, - base64_encoding: bool, + inherited_encoding: Option, + object_type: Option, + encoding_field: Option>, } struct StringPrefix { @@ -109,14 +111,42 @@ struct StringPrefix { ended: bool, } -#[derive(Default)] struct Base64Stats { + encoding: Base64Encoding, data: usize, padding: usize, invalid: bool, saw_padding: bool, } +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum Base64Encoding { + Standard, + UrlSafe, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum MediaObjectType { + Encoded(Base64Encoding), + Image, + Other, +} + +impl ObjectMediaContext { + fn encoding(&self) -> Option { + self.encoding_field.unwrap_or(match self.object_type { + Some(MediaObjectType::Encoded(encoding)) => Some(encoding), + Some(MediaObjectType::Image) => Some(Base64Encoding::Standard), + Some(MediaObjectType::Other) => None, + None => self.inherited_encoding, + }) + } + + fn is_typed_image(&self) -> bool { + self.object_type == Some(MediaObjectType::Image) + } +} + impl JsonFormatter { fn format(&mut self, output: &mut W) -> Result<()> { self.skip_ws()?; @@ -148,6 +178,15 @@ impl JsonFormatter { } fn write_object(&mut self, output: &mut W, depth: usize) -> Result<()> { + self.write_object_with_media_hint(output, depth, None) + } + + fn write_object_with_media_hint( + &mut self, + output: &mut W, + depth: usize, + inherited_encoding: Option, + ) -> Result<()> { self.expect_byte(b'{')?; output.write_all(b"{")?; self.skip_ws()?; @@ -156,11 +195,19 @@ impl JsonFormatter { return Ok(()); } - self.write_pretty_object(output, depth) + self.write_pretty_object(output, depth, inherited_encoding) } - fn write_pretty_object(&mut self, output: &mut W, depth: usize) -> Result<()> { - let mut media = ObjectMediaContext::default(); + fn write_pretty_object( + &mut self, + output: &mut W, + depth: usize, + inherited_encoding: Option, + ) -> Result<()> { + let mut media = ObjectMediaContext { + inherited_encoding, + ..ObjectMediaContext::default() + }; loop { output.write_all(b"\n")?; self.write_indent(output, depth + 1)?; @@ -175,19 +222,30 @@ impl JsonFormatter { let value = self.write_captured_string(output, 128)?; media.media_type = value.filter(|value| valid_media_type(value)); } - Some("type" | "encoding") => { + Some("type") => { let value = self.write_captured_string(output, 32)?; - if value.as_deref().is_some_and(|value| { - value.eq_ignore_ascii_case("base64") - || value.eq_ignore_ascii_case("base64url") - }) { - media.base64_encoding = true; - } + media.object_type = value.as_deref().map(media_object_type); + } + Some("encoding") => { + let value = self.write_captured_string(output, 32)?; + media.encoding_field = Some(value.as_deref().and_then(base64_encoding)); } key => self.write_string_for_view(output, key, &media)?, } } else { - self.write_value(output, depth + 1)?; + let typed_attachment = self.collapse_media + && key.as_deref() == Some("attachment") + && media.is_typed_image() + && self.peek_byte()? == Some(b'{'); + if typed_attachment { + self.write_object_with_media_hint( + output, + depth + 1, + Some(Base64Encoding::Standard), + )?; + } else { + self.write_value(output, depth + 1)?; + } } match self.read_separator(b'}')? { @@ -291,18 +349,41 @@ impl JsonFormatter { return Ok(capture.and_then(|bytes| String::from_utf8(bytes).ok())); } b'\\' => { - capture = None; let escaped = self.next_required("unterminated JSON string escape")?; output.write_all(&[escaped])?; match escaped { - b'"' | b'\\' | b'/' | b'b' | b'f' | b'n' | b'r' | b't' => {} + b'"' => push_capture_byte(&mut capture, b'"', max_bytes), + b'\\' => push_capture_byte(&mut capture, b'\\', max_bytes), + b'/' => push_capture_byte(&mut capture, b'/', max_bytes), + b'b' => push_capture_byte(&mut capture, 0x08, max_bytes), + b'f' => push_capture_byte(&mut capture, 0x0c, max_bytes), + b'n' => push_capture_byte(&mut capture, b'\n', max_bytes), + b'r' => push_capture_byte(&mut capture, b'\r', max_bytes), + b't' => push_capture_byte(&mut capture, b'\t', max_bytes), b'u' => { + let mut value = 0_u32; for _ in 0..4 { let hex = self.next_required("unterminated unicode escape")?; if !hex.is_ascii_hexdigit() { bail!("invalid unicode escape digit {}", describe_byte(hex)); } output.write_all(&[hex])?; + value = value + .checked_mul(16) + .and_then(|value| value.checked_add(hex_value(hex))) + .expect("four hex digits fit in u32"); + } + if let Some(ch) = char::from_u32(value) + && !(0xd800..=0xdfff).contains(&value) + { + let mut encoded = [0_u8; 4]; + push_capture_bytes( + &mut capture, + ch.encode_utf8(&mut encoded).as_bytes(), + max_bytes, + ); + } else { + capture = None; } } _ => bail!("invalid JSON string escape {}", describe_byte(escaped)), @@ -335,14 +416,16 @@ impl JsonFormatter { self.expect_byte(b'"')?; let prefix = self.read_safe_ascii_prefix(256)?; let data_uri = data_uri_header(&prefix.bytes); - let sibling_media = key == Some("data") - && media.media_type.is_some() - && (media.base64_encoding || prefix_looks_base64(&prefix.bytes)); + let sibling_media = + key == Some("data") && media.media_type.is_some() && media.encoding().is_some(); - if let Some((media_type, payload_start)) = data_uri { + if key != Some("arguments") + && let Some((media_type, payload_start)) = data_uri + { return self.write_media_summary( output, media_type, + Base64Encoding::Standard, &prefix.bytes[payload_start..], prefix.ended, ); @@ -350,7 +433,11 @@ impl JsonFormatter { if sibling_media { return self.write_media_summary( output, - media.media_type.as_deref(), + media + .media_type + .as_deref() + .expect("sibling media type checked"), + media.encoding().expect("sibling media encoding checked"), &prefix.bytes, prefix.ended, ); @@ -426,11 +513,12 @@ impl JsonFormatter { fn write_media_summary( &mut self, output: &mut W, - media_type: Option<&str>, + media_type: &str, + encoding: Base64Encoding, prefix_payload: &[u8], ended: bool, ) -> Result<()> { - let mut stats = Base64Stats::default(); + let mut stats = Base64Stats::new(encoding); for &byte in prefix_payload { stats.accept(byte); } @@ -438,7 +526,6 @@ impl JsonFormatter { self.scan_base64_string_tail(&mut stats)?; } - let media_type = media_type.unwrap_or("unknown media type"); match stats.decoded_bytes() { Some(bytes) => write!(output, "\"\"")?, None => write!(output, "\"\"")?, @@ -471,10 +558,10 @@ impl JsonFormatter { let byte = self.next_required("unterminated JSON string")?; match byte { b'"' if utf8.is_idle() => return Ok(()), - b'\\' if utf8.is_idle() => { - stats.invalid = true; - self.consume_string_escape()?; - } + b'\\' if utf8.is_idle() => match self.consume_string_escape_ascii()? { + Some(byte) => stats.accept(byte), + None => stats.invalid = true, + }, 0x00..=0x1f if utf8.is_idle() => { bail!("unescaped control byte in JSON string") } @@ -486,18 +573,30 @@ impl JsonFormatter { } } - fn consume_string_escape(&mut self) -> Result<()> { + fn consume_string_escape_ascii(&mut self) -> Result> { let escaped = self.next_required("unterminated JSON string escape")?; match escaped { - b'"' | b'\\' | b'/' | b'b' | b'f' | b'n' | b'r' | b't' => Ok(()), + b'"' => Ok(Some(b'"')), + b'\\' => Ok(Some(b'\\')), + b'/' => Ok(Some(b'/')), + b'b' => Ok(Some(0x08)), + b'f' => Ok(Some(0x0c)), + b'n' => Ok(Some(b'\n')), + b'r' => Ok(Some(b'\r')), + b't' => Ok(Some(b'\t')), b'u' => { + let mut value = 0_u32; for _ in 0..4 { let hex = self.next_required("unterminated unicode escape")?; if !hex.is_ascii_hexdigit() { bail!("invalid unicode escape digit {}", describe_byte(hex)); } + value = value + .checked_mul(16) + .and_then(|value| value.checked_add(hex_value(hex))) + .expect("four hex digits fit in u32"); } - Ok(()) + Ok(u8::try_from(value).ok()) } _ => bail!("invalid JSON string escape {}", describe_byte(escaped)), } @@ -651,11 +750,25 @@ impl JsonFormatter { } impl Base64Stats { + fn new(encoding: Base64Encoding) -> Self { + Self { + encoding, + data: 0, + padding: 0, + invalid: false, + saw_padding: false, + } + } + fn accept(&mut self, byte: u8) { match byte { - b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' | b'+' | b'/' | b'-' | b'_' - if !self.saw_padding => - { + b'A'..=b'Z' | b'a'..=b'z' | b'0'..=b'9' if !self.saw_padding => { + self.data = self.data.saturating_add(1); + } + b'+' | b'/' if self.encoding == Base64Encoding::Standard && !self.saw_padding => { + self.data = self.data.saturating_add(1); + } + b'-' | b'_' if self.encoding == Base64Encoding::UrlSafe && !self.saw_padding => { self.data = self.data.saturating_add(1); } b'=' if self.padding < 2 => { @@ -699,15 +812,18 @@ impl Base64Stats { } } -fn data_uri_header(prefix: &[u8]) -> Option<(Option<&str>, usize)> { +fn data_uri_header(prefix: &[u8]) -> Option<(&str, usize)> { let header_end = prefix .windows(8) .position(|window| window.eq_ignore_ascii_case(b";base64,"))?; if !prefix.get(..5)?.eq_ignore_ascii_case(b"data:") { return None; } - let media_type = std::str::from_utf8(prefix.get(5..header_end)?).ok()?; - let media_type = (!media_type.is_empty() && valid_media_type(media_type)).then_some(media_type); + let metadata = std::str::from_utf8(prefix.get(5..header_end)?).ok()?; + let media_type = metadata.split(';').next()?; + if !valid_media_type(media_type) { + return None; + } Some((media_type, header_end + 8)) } @@ -724,21 +840,46 @@ fn valid_media_type(value: &str) -> bool { }) } -fn prefix_looks_base64(prefix: &[u8]) -> bool { - prefix.len() >= 8 - && prefix.iter().all(|byte| { - matches!( - byte, - b'A'..=b'Z' - | b'a'..=b'z' - | b'0'..=b'9' - | b'+' - | b'/' - | b'-' - | b'_' - | b'=' - ) - }) +fn base64_encoding(value: &str) -> Option { + if value.eq_ignore_ascii_case("base64") { + Some(Base64Encoding::Standard) + } else if value.eq_ignore_ascii_case("base64url") { + Some(Base64Encoding::UrlSafe) + } else { + None + } +} + +fn media_object_type(value: &str) -> MediaObjectType { + match base64_encoding(value) { + Some(encoding) => MediaObjectType::Encoded(encoding), + None if value.eq_ignore_ascii_case("image") => MediaObjectType::Image, + None => MediaObjectType::Other, + } +} + +fn push_capture_byte(capture: &mut Option>, byte: u8, max_bytes: usize) { + push_capture_bytes(capture, &[byte], max_bytes); +} + +fn push_capture_bytes(capture: &mut Option>, bytes: &[u8], max_bytes: usize) { + let Some(buffer) = capture.as_mut() else { + return; + }; + if buffer.len().saturating_add(bytes.len()) > max_bytes { + *capture = None; + } else { + buffer.extend_from_slice(bytes); + } +} + +fn hex_value(byte: u8) -> u32 { + match byte { + b'0'..=b'9' => u32::from(byte - b'0'), + b'a'..=b'f' => u32::from(byte - b'a' + 10), + b'A'..=b'F' => u32::from(byte - b'A' + 10), + _ => unreachable!("caller validates ASCII hex digit"), + } } #[derive(Default)] diff --git a/crates/fmtview-core/src/load/lazy.rs b/crates/fmtview-core/src/load/lazy.rs index 7923cc0..198d0dc 100644 --- a/crates/fmtview-core/src/load/lazy.rs +++ b/crates/fmtview-core/src/load/lazy.rs @@ -18,6 +18,10 @@ const DIRECT_READ_LINE_BYTES: u64 = 64 * 1024; pub(crate) trait LazyProducer { fn produce(&mut self, source_offset: u64) -> Result; + + fn write_last_raw_record(&self, _output: &mut dyn Write) -> Result> { + Ok(None) + } } pub(crate) enum LazyBatch { @@ -38,14 +42,14 @@ pub(crate) struct LazyFile

{ impl LazyFile

{ #[cfg(test)] pub(crate) fn new(label: String, len: u64, producer: P) -> Result { - Self::with_raw_source(label, len, producer, None) + Self::with_raw_records(label, len, producer, false) } - pub(crate) fn with_raw_source( + pub(crate) fn with_raw_records( label: String, len: u64, producer: P, - raw_source: Option, + retain_raw_records: bool, ) -> Result { Ok(Self { label, @@ -59,7 +63,11 @@ impl LazyFile

{ source_line_offsets: Vec::new(), complete: len == 0, units_produced: 0, - raw_source, + raw_spool: retain_raw_records + .then(NamedTempFile::new) + .transpose() + .context("failed to create lazy raw record spool")?, + raw_spool_len: 0, raw_records: Vec::new(), }), }) @@ -194,16 +202,16 @@ impl ViewFile for LazyFile

{ }) else { return Ok(None); }; - let Some(source) = state.raw_source.as_ref() else { + let Some(spool) = state.raw_spool.as_ref() else { return Ok(None); }; let raw = RawRecordViewFile::new( - source - .try_clone() - .context("failed to clone raw record source")?, + spool + .reopen() + .context("failed to reopen lazy raw record spool")?, &self.label, - record.source_offset, - record.source_bytes, + record.raw_offset, + record.raw_len, line, )?; Ok(Some(Box::new(raw))) @@ -219,15 +227,16 @@ struct LazyState

{ source_line_offsets: Vec, complete: bool, units_produced: usize, - raw_source: Option, + raw_spool: Option, + raw_spool_len: u64, raw_records: Vec, } struct LazyRawRecordRef { first_line: usize, line_count: usize, - source_offset: u64, - source_bytes: u64, + raw_offset: u64, + raw_len: u64, } impl LazyState

{ @@ -247,13 +256,29 @@ impl LazyState

{ bail!("lazy producer returned no progress"); } let first_line = self.line_offsets.len(); + let raw_record = if let Some(raw_spool) = self.raw_spool.as_mut() { + let raw_offset = self.raw_spool_len; + let raw_len = self + .producer + .write_last_raw_record(raw_spool.as_file_mut())? + .context("lazy producer did not expose its raw record")?; + if raw_len != source_bytes { + bail!( + "lazy producer raw record length {raw_len} did not match source progress {source_bytes}" + ); + } + self.raw_spool_len = self.raw_spool_len.saturating_add(raw_len); + Some((raw_offset, raw_len)) + } else { + None + }; self.append_bytes(source_offset, &bytes)?; - if self.raw_source.is_some() { + if let Some((raw_offset, raw_len)) = raw_record { self.raw_records.push(LazyRawRecordRef { first_line, line_count: self.line_offsets.len().saturating_sub(first_line), - source_offset, - source_bytes, + raw_offset, + raw_len, }); } source_bytes diff --git a/crates/fmtview-core/src/load/lazy_records.rs b/crates/fmtview-core/src/load/lazy_records.rs index 26e1c49..fb012dd 100644 --- a/crates/fmtview-core/src/load/lazy_records.rs +++ b/crates/fmtview-core/src/load/lazy_records.rs @@ -1,4 +1,4 @@ -use std::{cell::RefCell, fs::File, rc::Rc, time::Duration}; +use std::{cell::RefCell, fs::File, io::Write, rc::Rc, time::Duration}; use anyhow::{Context, Result}; @@ -27,11 +27,11 @@ impl LazyTransformedRecordsFile { .with_context(|| format!("failed to stat {}", source.label()))? .len(); Ok(Self { - inner: LazyFile::with_raw_source( + inner: LazyFile::with_raw_records( label.clone(), len, RecordTransformProducer::new(label, file, options, Rc::clone(¬ices)), - Some(source.open()?), + true, )?, notices, }) @@ -123,6 +123,14 @@ impl LazyProducer for RecordTransformProducer { bytes: record.bytes, }) } + + fn write_last_raw_record(&self, output: &mut dyn Write) -> Result> { + let raw = self.reader.last_raw_record(); + output + .write_all(raw) + .context("failed to write lazy raw record spool")?; + Ok(Some(raw.len() as u64)) + } } #[derive(Default)] @@ -157,7 +165,10 @@ impl RecordRecoveryNotices { #[cfg(test)] mod tests { - use std::{io::Write, time::Duration}; + use std::{ + io::{Seek, SeekFrom, Write}, + time::Duration, + }; use tempfile::NamedTempFile; @@ -305,6 +316,42 @@ mod tests { assert_eq!(raw.as_bytes(), input.strip_suffix(b"\n").unwrap()); } + #[test] + fn lazy_raw_record_uses_the_ingested_snapshot_after_source_rewrites() { + let input = b"{\"value\":\"ORIGINAL\"}\n"; + let (mut temp, source) = temp_source(input); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let file = LazyTransformedRecordsFile::new(&source, options).unwrap(); + let lines = file.read_window(0, 16).unwrap(); + let value_line = lines + .iter() + .position(|line| line.contains("ORIGINAL")) + .unwrap(); + + temp.as_file_mut().seek(SeekFrom::Start(0)).unwrap(); + temp.as_file_mut() + .write_all(b"{\"value\":\"MUTATED!\"}\n") + .unwrap(); + temp.as_file_mut().flush().unwrap(); + + let raw = file.open_raw_record(value_line).unwrap().unwrap(); + assert_eq!( + raw.read_window(0, raw.line_count()).unwrap().concat(), + "{\"value\":\"ORIGINAL\"}" + ); + + temp.as_file_mut().set_len(0).unwrap(); + temp.as_file_mut().write_all(b"replacement\n").unwrap(); + temp.as_file_mut().flush().unwrap(); + assert_eq!( + raw.read_window(0, raw.line_count()).unwrap().concat(), + "{\"value\":\"ORIGINAL\"}" + ); + } + #[test] fn lazy_load_reads_spooled_lines_after_preload() { let (_temp, source) = temp_source(b"{\"a\":1}\n{\"b\":2}\n{\"c\":3}\n"); diff --git a/crates/fmtview-core/src/load/raw_record.rs b/crates/fmtview-core/src/load/raw_record.rs index dbdab5e..f081874 100644 --- a/crates/fmtview-core/src/load/raw_record.rs +++ b/crates/fmtview-core/src/load/raw_record.rs @@ -64,6 +64,7 @@ impl RawRecordViewFile { let advance = lookahead[..read] .iter() .take_while(|byte| is_utf8_continuation(**byte)) + .take(3) .count(); Ok(nominal.saturating_add(advance as u64).min(self.len)) } @@ -161,4 +162,40 @@ mod tests { assert!(chunks.iter().all(|chunk| chunk.len() <= 32 * 1024 + 3)); assert!(view.label().contains("raw record at line 13")); } + + #[test] + fn invalid_utf8_continuations_adjust_a_chunk_boundary_by_at_most_three_bytes() { + let mut spool = NamedTempFile::new().unwrap(); + let mut bytes = vec![b'a'; RAW_VIEW_CHUNK_BYTES as usize]; + bytes.extend_from_slice(&[0x80; 4]); + bytes.push(b'\n'); + spool.write_all(&bytes).unwrap(); + spool.flush().unwrap(); + let view = RawRecordViewFile::new( + spool.reopen().unwrap(), + "invalid.jsonl", + 0, + bytes.len() as u64, + 0, + ) + .unwrap(); + let mut file = spool.reopen().unwrap(); + + assert_eq!( + view.boundary(1, &mut file).unwrap(), + RAW_VIEW_CHUNK_BYTES + 3 + ); + assert_eq!(view.read_window(0, 1).unwrap().len(), 1); + } + + #[test] + fn empty_raw_record_still_has_one_empty_virtual_line() { + let mut spool = NamedTempFile::new().unwrap(); + spool.write_all(b"\n").unwrap(); + spool.flush().unwrap(); + let view = RawRecordViewFile::new(spool.reopen().unwrap(), "empty.jsonl", 0, 1, 0).unwrap(); + + assert_eq!(view.line_count(), 1); + assert_eq!(view.read_window(0, 1).unwrap(), vec![""]); + } } diff --git a/crates/fmtview-core/src/load/record_stream.rs b/crates/fmtview-core/src/load/record_stream.rs index 64ea3cc..78954c8 100644 --- a/crates/fmtview-core/src/load/record_stream.rs +++ b/crates/fmtview-core/src/load/record_stream.rs @@ -151,6 +151,10 @@ impl RecoveringFormattedRecordReader { diagnostic: None, })) } + + pub(crate) fn last_raw_record(&self) -> &[u8] { + &self.raw.line + } } pub(crate) struct RecoveringFormattedRecordBytes { diff --git a/crates/fmtview-core/src/transform/tests.rs b/crates/fmtview-core/src/transform/tests.rs index aae3725..792aa5c 100644 --- a/crates/fmtview-core/src/transform/tests.rs +++ b/crates/fmtview-core/src/transform/tests.rs @@ -96,3 +96,124 @@ fn redirected_record_format_never_collapses_media() { assert!(output.contains("data:image/png;base64,iVBORw0KGgo=")); assert!(!output.contains(" String { + String::from_utf8(format_record_display_bytes(input, jsonl_options()).unwrap()).unwrap() +} + +#[test] +fn sibling_media_requires_explicit_encoding_or_a_typed_image_shape() { + for input in [ + br#"{"source":{"media_type":"image/png","data":"deadbeef"}}"#.as_slice(), + br#"{"source":{"media_type":"image/png","encoding":"utf8","data":"AAAAAAAAAAAAAAAA"}}"#, + br#"{"attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + br#"{"type":"text","attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + br#"{"type":"image","attachment":{"media_type":"image/png","encoding":"utf8","data":"iVBORw0KGgo="}}"#, + ] { + let output = display_json(input); + assert!(!output.contains(""), + "{same_object}" + ); + + let attachment = display_json( + br#"{"type":"image","attachment":{"media_type":"image/png","data":"iVBORw0KGgo="}}"#, + ); + assert!( + attachment.contains(""), + "{attachment}" + ); +} + +#[test] +fn escaped_media_metadata_and_payload_use_json_string_semantics() { + let output = display_json( + br#"{"t\u0079pe":"base\u00364","media\u005ftype":"image\u002fpng","data":"iVBORw0KGgo\u003d"}"#, + ); + + assert!( + output.contains(""), + "{output}" + ); + + for (payload, expected) in [ + (r"QU\u004aD", "3 decoded bytes"), + (r"AA\/A", "3 decoded bytes"), + (r"QU\u0022D", "invalid base64"), + (r"QU\u000aD", "invalid base64"), + (r"QU\ud83d\ude80D", "invalid base64"), + ] { + let input = format!(r#"{{"content":"data:image/png;base64,{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "payload={payload:?} output={output}" + ); + } +} + +#[test] +fn base64_alphabet_and_padding_follow_the_declared_encoding() { + for (payload, expected) in [ + ("TQ==", "1 decoded bytes"), + ("TQ", "1 decoded bytes"), + ("TWE=", "2 decoded bytes"), + ("TWE", "2 decoded bytes"), + ("AA+/", "3 decoded bytes"), + ("AA-_", "invalid base64"), + ("A", "invalid base64"), + ("AA=A", "invalid base64"), + ] { + let input = format!(r#"{{"content":"data:image/png;base64,{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "payload={payload:?} output={output}" + ); + } + + for (encoding, payload, expected) in [ + ("base64url", "AA-_", "3 decoded bytes"), + ("base64url", "AA+/", "invalid base64"), + ("base64", "AA+/", "3 decoded bytes"), + ("base64", "AA-_", "invalid base64"), + ] { + let input = + format!(r#"{{"media_type":"image/png","encoding":"{encoding}","data":"{payload}"}}"#); + let output = display_json(input.as_bytes()); + assert!( + output.contains(expected), + "encoding={encoding} payload={payload:?} output={output}" + ); + } +} + +#[test] +fn media_collapse_does_not_rewrite_tool_argument_strings_or_invalid_data_uris() { + let arguments = + display_json(br#"{"arguments":"data:image/png;base64,iVBORw0KGgo=","other":"ok"}"#); + assert!( + arguments.contains("data:image/png;base64,iVBORw0KGgo="), + "{arguments}" + ); + assert!(!arguments.contains(", raw_record: Option, + content_space_changed: bool, } struct RawRecordOverlay { @@ -95,6 +96,7 @@ impl FileViewer { caches: ViewerCaches::default(), pending_prewarm: None, raw_record: None, + content_space_changed: false, } } @@ -216,6 +218,7 @@ impl FileViewer { self.caches = ViewerCaches::default(); self.pending_prewarm = None; } + self.content_space_changed = true; return ViewerAction { dirty: true, ..ViewerAction::default() @@ -318,6 +321,11 @@ impl FileViewer { size: Size, previous_position: Option, ) -> Result { + let previous_position = if std::mem::take(&mut self.content_space_changed) { + None + } else { + previous_position + }; if let Some(raw) = self.raw_record.as_mut() { let (frame, pending_prewarm) = draw_view( raw.file.as_ref(), @@ -433,6 +441,7 @@ impl FileViewer { caches: ViewerCaches::default(), pending_prewarm: None, }); + self.content_space_changed = true; } Ok(None) => self.state.set_footer_message( "raw view is available for record-stream inputs", diff --git a/crates/fmtview-core/tests/headless_viewer.rs b/crates/fmtview-core/tests/headless_viewer.rs index ae5d8cf..732d305 100644 --- a/crates/fmtview-core/tests/headless_viewer.rs +++ b/crates/fmtview-core/tests/headless_viewer.rs @@ -198,6 +198,52 @@ fn raw_toggle_prefers_the_active_search_match_record_over_the_viewport_top() { assert!(!raw_text.contains(r#""ref":"first""#), "{raw_text}"); } +#[test] +fn raw_content_space_transitions_never_reuse_terminal_scroll_hints() { + let record = format!(r#"{{"text":"{}"}}\n"#, "payload ".repeat(2_000)); + let source = source("long-raw.jsonl", &record); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(48, 8); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('r')); + let raw = viewer + .render( + size, + Some(fmtview_core::ScrollPosition { + top: 0, + row_offset: 1, + }), + ) + .unwrap(); + assert!(raw.title.contains("raw record")); + assert_eq!(raw.scroll_hint, None); + + send_key(&mut viewer, size, KeyCode::Char('j')); + let raw_scrolled = viewer.render(size, Some(raw.position)).unwrap(); + assert!(raw_scrolled.position.row_offset > raw.position.row_offset); + + send_key(&mut viewer, size, KeyCode::Char('r')); + let structured = viewer + .render( + size, + Some(fmtview_core::ScrollPosition { + top: 0, + row_offset: 1, + }), + ) + .unwrap(); + assert!(!structured.title.contains("raw record")); + assert_eq!(structured.scroll_hint, None); + assert!(buffer_text(structured).contains("payload")); +} + #[test] fn pretty_nested_tool_pair_navigation_round_trips_between_records() { let source = source( @@ -241,6 +287,48 @@ fn pretty_nested_tool_pair_navigation_round_trips_between_records() { assert!(buffer_text(returned).contains("tool_call")); } +#[test] +fn nested_tool_navigation_ignores_the_role_tool_envelope_message_id() { + let source = source( + "tool-envelope.jsonl", + concat!( + r#"{"role":"assistant","content":[{"type":"tool_call","id":"c1","name":"first"}]}"#, + "\n", + r#"{"role":"assistant","content":[{"type":"tool_call","id":"m3","name":"second"}]}"#, + "\n", + r#"{"id":"m3","role":"tool","content":[{"type":"tool_result","call_id":"c1","content":"ok"}]}"#, + "\n" + ), + ); + let options = FormatOptions { + kind: FormatKind::Jsonl, + indent: 2, + }; + let profile = TypeProfile::resolve(&source, &options).unwrap(); + let opened = open_view_file(&source, &options, profile).unwrap(); + let mut viewer = FileViewer::new(opened.file, opened.content, opened.notice); + let size = Size::new(72, 9); + viewer.render(size, None).unwrap(); + + send_key(&mut viewer, size, KeyCode::Char('/')); + for ch in "tool_result".chars() { + send_key(&mut viewer, size, KeyCode::Char(ch)); + } + send_key(&mut viewer, size, KeyCode::Enter); + while viewer.advance(std::time::Instant::now()).unwrap() {} + let searched = viewer.render(size, None).unwrap(); + send_key(&mut viewer, size, KeyCode::Esc); + let result = viewer.render(size, Some(searched.position)).unwrap(); + assert!(result.footer_text.contains("c1"), "{}", result.footer_text); + assert!(!result.footer_text.contains("m3"), "{}", result.footer_text); + + assert!(send_key(&mut viewer, size, KeyCode::Char('t')).dirty); + let call = viewer.render(size, Some(result.position)).unwrap(); + let call_text = buffer_text(call); + assert!(call_text.contains("first"), "{call_text}"); + assert!(!call_text.contains("second"), "{call_text}"); +} + #[test] fn diff_engine_renders_and_navigates_without_a_terminal() { let left = source("left.json", "{\"value\":1}\n"); diff --git a/docs/architecture.md b/docs/architecture.md index e4aed88..61023b6 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -371,14 +371,17 @@ anchor. The viewer synchronizes wrap and mouse-capture state when returning to the structured view. The terminal package only forwards backend-neutral input and draws the resulting frame. -Record streams also have a viewer-only JSON display formatter for -high-confidence inline base64 media. Data URIs are self-describing; sibling -`data` fields require earlier media-type and base64-encoding metadata in the -same object. The formatter validates the payload and computes decoded size by -scanning buffered input without allocating a decoded buffer or copying the -payload into the formatted spool. The ordinary transform path remains -token-preserving, so redirected output, diffs, tool arguments, and unknown -fields are unchanged. +Record streams also have a viewer-only JSON display formatter for explicitly +typed inline base64 media. Unescaped-ASCII data URI headers are +self-describing; sibling `data` fields require earlier media-type and +base64-encoding metadata in the same object. A typed image item's direct +`attachment` object inherits standard base64 unless that object supplies an +explicit encoding. Standard and URL-safe alphabets are validated separately; +arbitrary media-type/data siblings are not inferred from payload characters. +The formatter validates the payload and computes decoded size by scanning +buffered input without allocating a decoded buffer or copying the payload into +the formatted spool. The ordinary transform path remains token-preserving, so +redirected output, diffs, tool arguments, and unknown fields are unchanged. `probe_prefix` is source-neutral rather than a filesystem seek. It returns committed records from the source's true beginning under the same count/byte diff --git a/docs/visual-verification.md b/docs/visual-verification.md index 0c27e2c..6362123 100644 --- a/docs/visual-verification.md +++ b/docs/visual-verification.md @@ -54,7 +54,8 @@ FMTVIEW_EMULATOR_FOLLOW_FILE=target/follow-demo.jsonl \ For the generic conversation fixture, select the dedicated interaction scenario. It records nested tool-pair navigation, exact raw argument and media -records, the structured media summary, search, wrap, and clean quit: +records, a raw-record scroll followed by a structured redraw, the structured +media summary, search, wrap, and clean quit: ```sh FMTVIEW_EMULATOR_SCENARIO=conversation \ diff --git a/scripts/record-emulator-demo.sh b/scripts/record-emulator-demo.sh index f6768fe..ea4a9ab 100755 --- a/scripts/record-emulator-demo.sh +++ b/scripts/record-emulator-demo.sh @@ -278,6 +278,8 @@ elif [[ "$scenario" == "conversation" ]]; then sleep 0.2 send_key "raw_search_enter" "Return" sleep 0.7 + send_key "raw_scroll" "j" + sleep 0.7 send_key "structured_from_arguments" "r" sleep 0.7 send_key "media_search_open" "slash"