Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,8 +28,8 @@ for GitHub Release notes, so every published version must have a matching

- Collapse high-confidence inline base64 media in interactive JSONL/NDJSON
views to media type and validated decoded size without decoding or copying it
into the formatted display spool. Redirected output, tool arguments, media
bytes, and unknown fields remain token-preserving.
into the formatted display spool. Source records, redirected output, diffs,
tool arguments, media bytes, and unknown fields remain token-preserving.
- Split the publishable, terminal-independent viewer engine into the new
`fmtview-core` workspace crate. Format profiles and transforms, load/index
models, file and diff viewer state, search/navigation, highlighting, layout,
Expand Down
26 changes: 16 additions & 10 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -254,12 +254,16 @@ fixed schema. Typed `tool_call`/`tool_use` and `tool_result`/`tool_response`
objects can link by exact IDs even when they are nested in `content` arrays.
Embedded argument strings keep their original JSON token spelling and escapes.

In the interactive JSONL viewer, a `data:<media-type>;base64,...` string is
shown as a compact media summary. A same-object `type: base64` plus
`media_type`/`mime_type` and later `data` field is handled too. Metadata must
precede that sibling `data` field for streaming recognition. Valid payloads
show the decoded byte size; recognized invalid payloads say `invalid base64`
instead of claiming a size. Redirected output never collapses these strings.
In the interactive JSONL viewer, an unescaped-ASCII
`data:<media-type>;base64,...` string is shown as a compact media summary. A
same-object `type: base64` or `type: base64url` plus
`media_type`/`mime_type` and later `data` field is handled too, as is a direct
`attachment` object under a typed image item. Standard and URL-safe alphabets
are validated separately. Metadata must precede that sibling `data` field for
streaming recognition, and arbitrary `media_type` plus `data` objects are not
guessed to be base64. Valid payloads show the decoded byte size; recognized
invalid payloads say `invalid base64` instead of claiming a size. Redirected
output never collapses these strings.

### Following growing JSONL files

Expand Down Expand Up @@ -452,11 +456,11 @@ older records backward when needed, and extends forward after source refreshes.
The footer shows `follow:on`, `follow:detached`, or `follow:off`.

Press `r` on a JSONL/NDJSON record to open a bounded raw snapshot backed by its
exact source or spool range. An active search match selects its containing
exact immutable raw-spool range. An active search match selects its containing
record; otherwise the viewport's top record is used. The snapshot stays on that
record while a followed source appends or resets in the background; returning
to structured mode shows the updated stream without changing an existing
detached anchor. Raw display
record even if the source is appended, truncated, or replaced in the
background; returning to structured mode shows the updated stream without
changing an existing detached anchor. Raw display
uses 32 KiB visual chunks so a huge record is never copied wholesale into the
viewer frame. Search works within those chunks, but a query spanning an
artificial chunk boundary is not matched. Invalid UTF-8 is displayed lossily;
Expand Down Expand Up @@ -582,6 +586,8 @@ rendered output in memory for browsing.
- Lazy preview writes transformed records into a temporary spool and keeps compact
offsets, not formatted strings, in memory. The title shows `N+` lines while
the session index is still incomplete and idle time extends the index.
- JSONL raw toggles write only ingested source records to a separate immutable
temporary spool, so an open snapshot cannot change after source replacement.
- Viewer-only JSONL formatting scans recognized base64 payloads in buffered
slices, validates padding/alphabet, and writes only the media summary. It does
not allocate a decoded buffer or a payload-sized formatted output buffer.
Expand Down
18 changes: 10 additions & 8 deletions crates/fmtview-core/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,20 @@ backend-neutral events, including `ViewerCommand::FollowTail` and
`ViewerCommand::ToggleFollowTail`.

Record-backed views can expose exact raw-record snapshots through bounded
virtual lines. `FileViewer` owns the `r` structured/raw toggle, separate raw
search and wrap state, and synchronization when returning to the structured
viewport. Follow refresh and the main viewport's attached or detached state
continue behind an open raw snapshot.
virtual lines. Lazy JSONL records are copied once at ingest into an immutable
raw spool, so opening raw mode is an offset lookup and source replacement
cannot change the snapshot. `FileViewer` owns the `r` structured/raw toggle,
separate raw search and wrap state, and synchronization when returning to the
structured viewport. Follow refresh and the main viewport's attached or
detached state continue behind an open raw snapshot.

Generic conversation handling also stays in core. The JSON package recognizes
direct chat roles, contextual tool calls, direct tool-role results, and nested
typed tool results without depending on an application's storage types. The
viewer-only JSONL display path can collapse high-confidence inline base64 media
to media type and validated decoded size without decoding the payload. Normal
transforms preserve exact tool-argument tokens, media payloads, and unknown
fields for redirected output and diffs.
viewer-only JSONL display path can collapse explicitly typed inline base64
media to media type and validated decoded size without decoding the payload.
Normal transforms preserve exact tool-argument tokens, media payloads, and
unknown fields for redirected output and diffs.

Reset overlap uses a bounded, non-consuming probe of the committed source
prefix. Exact IDs for the matched prefix are filtered from tail and later older
Expand Down
88 changes: 72 additions & 16 deletions crates/fmtview-core/src/formats/json/tool_links.rs
Original file line number Diff line number Diff line change
Expand Up @@ -55,6 +55,7 @@ struct ToolContainer {
role_tool: bool,
toolish_type: bool,
tool_result_type: bool,
contains_typed_result: bool,
ids: Vec<IdCandidate>,
provisional_link: Option<ToolLink>,
pending_child: Option<ContainerContext>,
Expand Down Expand Up @@ -89,7 +90,7 @@ struct PendingCall {
#[derive(Debug, Clone)]
struct ToolDiscovery {
start_line: usize,
link: ToolLink,
link: Option<ToolLink>,
}

#[derive(Debug)]
Expand All @@ -109,7 +110,7 @@ enum ToolMatchDecision {

impl ToolLinkTracker {
pub(crate) fn apply_line(&mut self, line: &str, line_number: usize) {
self.scan_line(line, line_number);
let _ = self.scan_line(line, line_number);
}

#[cfg(test)]
Expand All @@ -129,7 +130,11 @@ impl ToolLinkTracker {
for (offset, line) in visible_lines.iter().chain(lookahead_lines).enumerate() {
let line_number = first_line.saturating_add(offset);
for ToolDiscovery { start_line, link } in self.scan_line(line, line_number) {
results.insert(start_line, (offset, link));
if let Some(link) = link {
results.insert(start_line, (offset, link));
} else {
results.remove(&start_line);
}
}
if let Some((start_line, link)) = self.active_result() {
results.insert(start_line, (offset, link.clone()));
Expand Down Expand Up @@ -190,12 +195,10 @@ impl ToolLinkTracker {
'{' => self.push_container(ContainerKind::Object, line_number),
'[' => self.push_container(ContainerKind::Array, line_number),
'}' => {
if let Some(discovery) = self.pop_container(ContainerKind::Object) {
discoveries.push(discovery);
}
discoveries.extend(self.pop_container(ContainerKind::Object));
}
']' => {
self.pop_container(ContainerKind::Array);
discoveries.extend(self.pop_container(ContainerKind::Array));
}
_ => {}
}
Expand Down Expand Up @@ -225,24 +228,48 @@ impl ToolLinkTracker {
role_tool: false,
toolish_type: false,
tool_result_type: false,
contains_typed_result: false,
ids: Vec::new(),
provisional_link: None,
pending_child: None,
});
}

fn pop_container(&mut self, expected: ContainerKind) -> Option<ToolDiscovery> {
fn pop_container(&mut self, expected: ContainerKind) -> Vec<ToolDiscovery> {
let mut discoveries = Vec::new();
while let Some(container) = self.containers.pop() {
if container.kind != expected {
continue;
}
if expected == ContainerKind::Object {
if container.role_tool || container.tool_result_type {
if (container.role_tool || container.tool_result_type)
&& !container.contains_typed_result
{
let link = self.link_result(container.start_line, &container.ids);
return link.map(|link| ToolDiscovery {
start_line: container.start_line,
link,
});
let mut preserved_ancestor_fallback = false;
if container.tool_result_type {
let authoritative_child_id = container
.ids
.iter()
.any(|candidate| result_id_rank(&candidate.key) >= 3);
let preserve_explicit_fallback = !authoritative_child_id
&& link
.as_ref()
.is_none_or(|link| link.status != ToolLinkStatus::Matched);
preserved_ancestor_fallback = self.suppress_ancestor_results(
&mut discoveries,
preserve_explicit_fallback,
);
}
if let Some(link) = link
&& !(preserved_ancestor_fallback && link.status != ToolLinkStatus::Matched)
{
discoveries.push(ToolDiscovery {
start_line: container.start_line,
link: Some(link),
});
}
return discoveries;
}
if (container.context == ContainerContext::ToolCallObject || container.toolish_type)
&& !container.ids.is_empty()
Expand All @@ -254,12 +281,40 @@ impl ToolLinkTracker {
while self.pending_calls.len() > MAX_PENDING_CALLS {
self.pending_calls.pop_front();
}
return None;
return discoveries;
}
}
break;
}
None
discoveries
}

fn suppress_ancestor_results(
&mut self,
discoveries: &mut Vec<ToolDiscovery>,
preserve_explicit_fallback: bool,
) -> bool {
let mut preserved = false;
for parent in &mut self.containers {
if preserve_explicit_fallback
&& (parent.role_tool || parent.tool_result_type)
&& parent
.ids
.iter()
.any(|candidate| result_id_rank(&candidate.key) >= 3)
{
preserved = true;
continue;
}
parent.contains_typed_result = true;
if parent.provisional_link.take().is_some() {
discoveries.push(ToolDiscovery {
start_line: parent.start_line,
link: None,
});
}
}
preserved
}

fn apply_property(&mut self, property: JsonProperty) {
Expand Down Expand Up @@ -302,7 +357,8 @@ impl ToolLinkTracker {
let Some(container) = self.containers.last() else {
return;
};
let link = (container.role_tool || container.tool_result_type)
let link = ((container.role_tool || container.tool_result_type)
&& !container.contains_typed_result)
.then(|| self.preview_result(container.start_line, &container.ids))
.flatten();
if let Some(container) = self.containers.last_mut() {
Expand Down
Loading