Feature/multimodal observations - #388

Open

fatemehpesaran310 wants to merge 8 commits into

ServiceNow:mainfrom

fatemehpesaran310:feature/multimodal-observations

Commits on Apr 14, 2026

Add audio capture support to BrowserGym observations

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
2c13954
Rewrite audio capture using JavaScript MediaRecorder API

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
c0577e4

Commits on Apr 16, 2026

Add Whisper API support and LLM agent test scripts

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
5efda3e
Add webarena_pro benchmark with audio/video tasks

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
9df3450
Add quick start README for webarena_pro audio tasks

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
6db33da

Commits on Apr 17, 2026

Add video frame extraction support to BrowserGym observations

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
66acc2b
Add VLM-based video frame captioning for LLM agents

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
cb94d25
Complete video + audio pipeline with VLM and LLM agent paths

Fatemeh Pesaran Zadeh [Affiliate]
and
claude
committed
32ce1ce