Skip to content

V2 chat: inline audio and video players, image cards, and rich agent-posted group messages #1611

Open

Description

@paullizer

Summary

Agents and actions increasingly return media: call recordings, short camera clips, photos and document scans, usually as short-lived signed links from an OpenAPI action. The React V2 chat does not present these well:

  • Audio and video links render as plain links that open the file on a new page, so listening to a recording means leaving the conversation.
  • The content security policy only allows media-src 'self' blob:, so audio and video served from an action's own host cannot play inline even if a player were added.
  • When an agent posts into a group (collaboration) conversation with the SimpleChat action (add_conversation_message), the message is stored as an ordinary user message and V2 renders it as plain text. Headings, tables, links and images in an agent-written briefing show up as raw markdown.

User Value

People reviewing an agent's findings can listen to recordings, watch clips and see photos in place, and a briefing an agent posts to a group reads the same way as the agent's own reply.

Proposed Behavior

  1. Inline audio player. A markdown link whose URL path ends in an audio extension renders as a compact bar with play/pause, the title and elapsed/total time. Expanding it adds a seek slider, stop, volume and mute, and playback speed. Starting one recording pauses any other. Keyboard accessible, with ARIA labels.
  2. Inline video cards. Links to video files render as a card with native controls (play, seek, volume, fullscreen) and the link text as the caption.
  3. Image cards. Markdown images render as captioned cards that open in the existing image lightbox.
  4. Rich agent-posted group messages. Messages created through add_conversation_message carry metadata marking them as agent-authored markdown, and V2 renders them with the same XSS-safe markdown renderer used for assistant replies (no raw HTML). Messages typed by people stay plain text.
  5. Media source allowlist. An administrator-configured list of external https:// origins is appended to the CSP media-src directive. Entries are validated (scheme, host, optional leading wildcard label and port only; no paths, quotes or semicolons) so the setting cannot inject other directives.

Acceptance Criteria

  • Audio links in assistant replies render as the inline player; play/pause, stop, seek, volume/mute and speed work, and only one recording plays at a time.
  • Video links render as inline video cards, and image markdown renders as captioned cards with click-to-enlarge.
  • Media that fails to load falls back to a normal link.
  • Agent-posted group conversation messages render as markdown; messages typed by people are unchanged.
  • External media plays only from configured origins; invalid allowlist entries are rejected and logged.
  • Functional tests cover link classification, CSP allowlist validation and the agent-message metadata; docs and release notes are updated.

Notes

Activity

  1. moved this from Pending Evaluation to In progress in Simple Chat Roadmapon Oct 2, 2026
  2. paullizer commented on Oct 2, 2026

    @paullizer
    ContributorAuthor

    Status update: implemented in #1612 (version 0.261.222), stacked on #1608.

    What changed

    • Audio links in replies render as an inline player: play/pause, title and elapsed/total time; expanded, it adds seek, stop, volume and mute, speed and open-in-new-tab. One recording or clip plays at a time.
    • Video links render as cards with native controls; markdown images render as captioned cards that open the image viewer.
    • Messages an agent posts through add_conversation_message (personal and shared) are marked posted_via: agent_action, content_format: markdown, and V2 renders only those as markdown, labeled "posted through an agent".
    • New CSP_MEDIA_SRC_ORIGINS app setting adds validated bare https:// origins to media-src; other entries are rejected and logged.

    Validation

    • test_v2_inline_media_players.py (including the Node run of test_v2_inline_media_logic.mjs), test_v2_rich_rendering.py, test_v2_message_masking.py, test_v2_citations.py: 37 passed. Docs tests: 13 passed.
    • XSS sink checker on the added lines and V2 type-check: passed.
    • Deployed to a SimpleChat POC deployment. An agent workflow posted a group briefing with 13 images, 2 video clips and 2 recordings; the message carried the marker and the media host was allowed through CSP_MEDIA_SRC_ORIGINS.

    Follow-ups

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P2Priority 2: important, scheduled after P0/P1enhancementNew feature or request

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions