Skip to content

feat(viewer): render run traces from a payload with a JS renderer - #987

Open
Polarisyjr wants to merge 1 commit into
benchflow-ai:mainfrom
Polarisyjr:feat/trace-viewer-payload-renderer
Open

feat(viewer): render run traces from a payload with a JS renderer#987
Polarisyjr wants to merge 1 commit into
benchflow-ai:mainfrom
Polarisyjr:feat/trace-viewer-payload-renderer

Conversation

@Polarisyjr

Copy link
Copy Markdown

Summary

This is Step 1 of the BenchFlow trace viewer work. More features will be added in follow-up PRs; at this stage, the main goal is to establish the rendering architecture and bring the viewer's overall style and trace presentation closer to PostTrainBench.

Refactors bench eval view around a shared payload-based renderer:

rollout artifacts → viewer.py → versioned payload → render.js → HTML

Instead of generating trace HTML directly in Python, Python now handles artifact normalization while JavaScript handles presentation. This also creates a shared renderer that can later be reused by a hosted/static viewer.

What changed

  • Add a versioned viewer payload generated from rollout artifacts.
  • Treat trajectory/acp_trajectory.jsonl as the canonical trace source, with legacy turn*.txt as a fallback.
  • Move trace HTML rendering into render.js.
  • Separate browser/DOM setup into boot.js, keeping the renderer independently testable.
  • Rework the viewer layout and styling to roughly align with the PostTrainBench trace viewer.
  • Vendor the PostTrainBench stylesheet as the base visual layer, with BenchFlow-specific overrides isolated in benchflow.css.
  • Add BenchFlow-specific run information including:
    • reward and run status
    • token usage
    • environment / agent / verifier timing
    • skill mode and skill invocations
    • agent/verifier errors and partial trajectories
  • Improve tool-call rendering:
    • preserve complete ACP content blocks
    • render diffs separately
    • handle binary output per block
    • truncate very large outputs while preserving the beginning and end
    • support ANSI formatting and a limited escaped Markdown subset
  • Bundle viewer assets locally so generated pages remain portable and work without network access.

Compatibility

This does not introduce a new on-disk trace format.

The viewer payload is derived at render time and is not persisted. Existing ACP trajectory artifacts remain unchanged, and legacy Claude Code turn*.txt traces continue to work as a fallback.

When both formats exist, ACP is used as the source of truth rather than merging or rendering both.

Tests

Added two layers of viewer tests:

  • tests/test_trajectory_viewer.py

    • payload structure
    • run status
    • token metadata
    • turn grouping
    • tool-output completeness and truncation
    • payload/script escaping
  • tests/test_trajectory_viewer_render.py

    • runs render.js through Node
    • tool output and diff rendering
    • Markdown and ANSI rendering
    • binary placeholders
    • injection protection
    • reward 0 vs. unscored
    • schema-version mismatch

Also updates the trajectory integration tests for the new payload-based rendering path.

Validation

uv run pytest \
  tests/test_trajectory_viewer.py \
  tests/test_trajectory_viewer_render.py \
  tests/test_trajectory_integration.py -q

Manual E2E:

bench eval view <rollout-dir>

Next steps

This PR intentionally focuses on the basic single-run viewer and visual alignment with PostTrainBench.

Follow-up work will add more viewer features such as job-level navigation, verifier details, review/audit results, cost and per-call metrics, system metrics, and additional trace metadata.

image image

Replaces the server-rendered HTML in trajectories/viewer.py with a
payload -> render.js pipeline: Python emits data only, JS emits HTML, so
`bench eval view` and the future static site share one renderer instead of
drifting into two. Vendors PostTrainBench's stylesheet byte-for-byte with our
additions kept in a separate bf-prefixed layer.

Adds payload tests and node-driven renderer tests; updates the viewer
compatibility tests to assert on the payload's turn structure.
@1digitaldesign

Copy link
Copy Markdown

Cool cool!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants