diff --git a/index.html b/index.html index 4475279..ab27c1a 100644 --- a/index.html +++ b/index.html @@ -3,734 +3,659 @@ -FlashRT — ML infrastructure, rewritten by agents - - +FlashRT - + - + + -
-
- - ⚡ - FlashRT - - -
-
- -
+
- -
-
-
-
-
- Infini-AI Lab · Carnegie Mellon -
-

- Real-time multimodal applications, - each deserves its own system. -

-

- Multimodal applications are essentially complicated. Each sits at - its own point on the latency–frame-rate–throughput tradeoff. They - land on different hardware budgets. And their models compute - differently: autoregressive LLMs, diffusion transformers, streaming - VAEs. No shared serving stack fits them all — so a general agent - customizes a system for each, from your plain - single-GPU reference, verifying every step along the way. And - specialization pays: the same models, on the same GPUs, run - up to 70× faster. -

-
- View code -
-
- + +
+

FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications

+
+ Krish Agarwal1, + Zhuoming Chen1, + Yanyuan Qin2, + Zhenyu Gu2, + Atri Rudra3, + Beidi Chen1
-
+
+ 1Carnegie Mellon University  ·  + 2AMD  ·  + 3University at Buffalo +
+
+ + + arXiv + + + + Code + +
+ - -
-
-
-
Experimental Results
-

One agent. Many real-time applications.

-

- For each application the agent is given two things: the reference - implementation and a GPU budget. No hints, no templates. It works out - streaming, disaggregation, sequence parallelism, and pipeline - parallelism on its own, and checks every variant against the reference - before keeping it. All numbers below come from one node of 8 NVIDIA - B200 GPUs, using Claude Code with Opus 4.8. -

-
+ +
+

+ Real-time multimodal applications each deserve their own serving systems. + Multimodal applications are often quite complicated. Each sits at its own set of tradeoffs on key metrics like latency and throughput, and balancing these metrics can require different hardware budgets for each application. Each application also consists models compute that differently: autoregressive LLMs, diffusion transformers, etc. Ultimately, no shared serving stack can fit them all. Instead, we introduce FlashRT, an agent harness that guides a coding agent to customize a system for each application. From a plain user-provided reference, it directs the agent to develop a customized system to serve it. The specialization pays off: Across five real-time applications, on both NVIDIA B200 and AMD MI355X GPUs, FlashRT reaches up to + 70× lower latency and 3.6× higher throughput, all with no hand-written serving code. +

+
-
-
-
70×
-
lower latency on the face-to-face conversational avatar
-
-
-
2.8×
-
higher frame rate on the real-time video background editor
-
-
-
25%
-
lower latency than hand-tuned vLLM-Omni on Qwen3-Omni
-
-
-
2.4×
-
higher frame rate on the real-time video narrator
-
-
+ +
+

📊One agent, Many Real-time Applications

+

+ For each application the agent is given two things: the reference implementation and a GPU + budget. No hints, no templates. It works out streaming, disaggregation, sequence parallelism, + and pipeline parallelism on its own, and checks every variant against the reference before + keeping it. We run the agent separately on a node of 8 NVIDIA B200 GPUs and a node of 8 AMD MI355X + GPUs; use the toggle to switch between them. (Agent: Claude Code with Opus 4.8.) +

-
- - - - - -
+
+ + + +
- -
-
-

Face-to-Face Conversational Agent

-

The pipeline runs ASR → LLM → streaming TTS → an audio-driven video - avatar (Live-Avatar). The agent overlaps the stages with chunk-level - streaming, splits the TTS and video engines onto separate GPUs, and - pipelines the four denoising steps inside the video model. A ~108-second - offline baseline becomes a ~1.6-second interactive stream that runs well - above real-time frame rates.

-
-
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential, no streaming)107.92 s—
1FlashRT — streaming3.94 s16.26 FPS
3FlashRT — streaming + disaggregation1.57 s40.88 FPS
8FlashRT — streaming + disagg. + S2V pipeline parallelism1.66 s173.67 FPS (theoretical)
-
-

- About 70× lower latency than the hand-written single-GPU reference - (108 s → 1.6 s). The 8-GPU frame rate is the pipeline's theoretical - sustainable throughput, far beyond what real-time display needs, so - latency is the binding constraint here. -

+
+
+ + + + +
- - +
+ +
+
- - - - - - - - -
-
+ +
+

Face-to-Face Conversational Agent Live-Avatar-14B (S2V)

+

+ The pipeline runs ASR → LLM → streaming TTS → an audio-driven video avatar (Live-Avatar). + The agent overlaps the stages with chunk-level streaming, splits the TTS and video engines + onto separate GPUs, and pipelines the four denoising steps inside the video model. A + minutes-long offline baseline becomes a roughly one-second interactive stream that runs well above + real-time frame rates. +

+
+
+ + + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential, no streaming)107.92 s—
1FlashRT — streaming3.94 s16.26 FPS
3FlashRT — streaming + disaggregation1.57 s40.88 FPS
8FlashRT — streaming + disagg. + S2V pipeline parallelism1.66 s173.67 FPS (theoretical)
+
+

+ About 70× lower latency than the hand-written single-GPU reference (108 s → 1.6 s). The + 8-GPU frame rate is the pipeline's theoretical sustainable throughput, far beyond what + real-time display needs, so latency is the binding constraint here. +

+
+
+
+ + + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
2Baseline (sequential, no streaming)101.17 s—
2FlashRT — streaming10.58 s34.2 FPS
3FlashRT — streaming + disaggregation1.59 s37.8 FPS
8FlashRT — streaming + disagg. + S2V pipeline parallelism1.47 s78.5 FPS
+
+

+ About 69× lower latency than the sequential reference (101 s → 1.5 s), with the 8-GPU + frame rate well above what real-time display needs. +

+
+
- -
-
-
The vision
-

- Development efficiency and - intelligence
- change how ML infrastructure is designed. -

-

- For decades we have built ML software as a layered stack: hand-written - kernels beneath frameworks, frameworks beneath compilers and serving - systems, each layer written once and shared by every application above - it. Sharing was never the goal — it was the workaround for two - constraints of human-written software. AI agents remove both. -

- -
-
-
01 · Efficient development
-

Every application gets its own system.

-
- Then + +
+

Qwen3-Omni Multimodal LLM

- Code was expensive to write. Every line an engineer produced had - to be amortized across many applications, so everyone settled - for the average case of a shared stack. + Qwen3-Omni is a natively multimodal model with three stages to support text input → audio output: + a Thinker LLM, a Talker LLM, and a vocoder. We compare with vLLM-Omni, which disaggregates and streams + these stages with hand-written code. Given only a synchronous baseline, the agent recovers the same + structure on its own, then edges past it with lighter-weight data transfer between components than + vLLM-Omni's more general implementation.

-
-
- Now +
+
+ + + + + + + + + +
DeploymentGPUsLatency ↓RTF < 1
Sequential (no streaming)142.713 s✓
vLLM-Omni (hand-engineered)30.433 s✓
FlashRT30.323 s✓
+
+

+ 25% lower latency than vLLM-Omni's hand-engineered deployment, while keeping the real-time + factor below 1. +

+
+
+
+ + + + + + + + + +
DeploymentGPUsLatency ↓RTF < 1
Sequential (no streaming)129.66 s✓
vLLM-Omni (hand-engineered)30.779 s✓
FlashRT30.276 s✓
+
+

+ 65% lower latency than vLLM-Omni's hand-engineered deployment — a wider margin than on + B200 — while keeping the real-time factor below 1. +

+
+
+ + +
+

Video Background Editor Krea-Realtime-14B + SAM 3

- For an agent, lines of code are cheap. It writes, rewrites, and - discards code at negligible cost — so a system specialized to - your application, your hardware, - and your efficiency targets stops being a - luxury. And specialization is where efficiency comes from: a - system shaped to one application outruns a general stack on the - same hardware. + A live webcam stream runs down two paths at once. Krea-Realtime restyles each frame while + SAM 3 segments the person, and the two results are composited so only the background changes. + The agent puts SAM 3 on its own GPU, off the restyling path, and parallelizes the DiT and VAE + that do the restyling. +

+
+
+ + + + + + + + + + +
DeploymentGPUsLatency ↓Frame rate ↑
Baseline (sequential)11715 ms6.82 FPS
FlashRT — parallel21014 ms11.54 FPS
FlashRT — latency-optimized4491 ms17.18 FPS
FlashRT — frame-rate-optimized5517 ms19.41 FPS
+
+

+ About 3.5× lower latency (1.7 s → 0.5 s) and 2.8× the baseline frame rate (6.8 → 19.4 FPS), + with SAM 3 kept on its own GPU throughout.

-
-
-
-
02 · Intelligent
-

Reuse ideas, not frozen code.

-
- Then +
+
+
+ + + + + + + + + +
DeploymentGPUsLatency ↓Frame rate ↑
Baseline (sequential)12393 ms5.40 FPS
FlashRT — latency-optimized4612 ms7.93 FPS
FlashRT — frame-rate-optimized5646 ms8.90 FPS
+
+

+ About 3.9× lower latency. Here the pipeline is bound by the SAM 3 segmentation path, + so disaggregating the VAE adds little frame rate. +

+
+
+ + +
+

Video World Model WorldPlay-5B

- Traditional stacks compose only through exact, strictly - compatible interfaces. Code that is even slightly incompatible - cannot be reused at all — so reuse meant frozen, portable - artifacts. + A video world model turns keyboard actions (WASD, arrow keys) into a video stream you can + move through in real time. We use WorldPlay, which splits into an autoregressive DiT and a + streaming VAE that share no state, so the agent can disaggregate them onto separate GPUs and + pipeline across frames for a higher frame rate, or co-locate them under sequence parallelism + for lower latency. It scales whichever choice it makes to the GPU budget. +

+
+
+ + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential)869 ms20.4 FPS
2FlashRT — frame-rate-optimized625 ms31.0 FPS
2FlashRT — latency-optimized493 ms25.5 FPS
+
+

+ 1.5× the baseline frame rate or 43% lower latency, depending on which target the agent + optimizes. See paper for extended results with scaling to more GPUs. +

+
+
+
+ + + + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential)1102 ms15.6 FPS
2FlashRT — latency-optimized576 ms19.8 FPS
2FlashRT — frame-rate-optimized723 ms29.5 FPS
4FlashRT — latency-optimized320 ms31.8 FPS
8FlashRT — frame-rate-optimized326 ms56.8 FPS
+
+

+ At 2 GPUs, the same co-location (latency) vs. disaggregation (frame rate) split as on B200; the + frame rate scales to 3.6× the baseline (56.8 FPS) at 8 GPUs.

-
-
- Now +
+
+ + +
+

Video Narrator LongLive-2.0-5B

- An agent reads code the way an engineer does. It takes the - ideas from a system that almost fits — kernels, - schedules, parallelism strategies — and re-implements them for - the case at hand. + The user speaks, an ASR model transcribes each prompt, and an autoregressive video model + (LongLive-2.0-5B) keeps rendering a stream that updates to match. The agent runs the ASR on + its own GPU so transcription never stalls generation, then trades latency against frame rate: + co-locate the DiT and VAE at a higher sequence-parallel degree for lower latency, or split them + apart and tile the VAE for a higher frame rate.

-
- -
+
+
+ + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential)674 ms25.8 FPS
2FlashRT — latency-optimized512 ms36.5 FPS
2FlashRT — frame-rate-optimized630 ms47.8 FPS
+
+

+ 1.9× the baseline frame rate, or 1.3× lower latency when the agent optimizes for latency + instead. See paper for extended results with scaling to more GPUs. +

+
+
+
+ + + + + + + + + + + +
GPUsDeploymentLatency ↓Frame rate ↑
1Baseline (sequential)784 ms18.9 FPS
2FlashRT — latency-optimized660 ms20.4 FPS
2FlashRT — frame-rate-optimized907 ms30.5 FPS
4FlashRT — latency-optimized343 ms38.8 FPS
8FlashRT — frame-rate-optimized442 ms56.5 FPS
+
+

+ At 2 GPUs, the same co-location (latency) vs. disaggregation (frame rate) split as on B200; the + frame rate scales to 3.0× the baseline (56.5 FPS) at 8 GPUs. +

+
+ -
-
One reference — many systems, written online
-
- latency-oriented - frame-rate-oriented - balanced - 1 → 8 GPUs
-

- This is not hypothetical — every deployment in the results above is - such a system. From one reference implementation, the agent writes a - latency-oriented system when you are chasing responsiveness, a - frame-rate-oriented one when you are chasing smoothness, a balanced - one in between — and re-specializes it for however many GPUs you - actually have. -

+
+
- -

- And there is a cost we rarely question. A shared stack must serve - everyone, so it accretes — compatibility paths, configuration surfaces, - layers upon layers — until maintaining and extending it becomes a - project of its own, and understanding and debugging it becomes a - tremendous challenge for humans and agents alike. In the era of agents, - it is worth asking whether that structure is still a - helper or a blocker — especially for ML - infrastructure, whose entire job is to connect models, systems, and - hardware. -

- -

- The foundation of ML software shifts - from shared infrastructure to a general agent -  — and FlashRT is our first concrete step. -

- -
-
-
-
Why we build this
-

Real-time models for people. Agents for the systems work.

-

- FlashRT sits between two problems we care about: making generative - models fast enough to use interactively, and handing the repetitive - systems work behind them to an agent instead of an engineer. -

-
+ +
-
-
-
01
-

Interactive generative models

+

🧭The Vision

+

Development efficiency and intelligence change how ML infrastructure is designed.

+

+ For decades we have built ML software as a layered stack: hand-written kernels beneath + frameworks, frameworks beneath compilers and serving systems, each layer written once and + shared by every application above it. Sharing was never the goal — it was the workaround + for two constraints of human-written software. AI agents remove both. +

+ +
+

01 · Every application gets its own system.

+
+
+
Then

- Most generative models are still used in batches: send a prompt, wait, - get a file back. We are after the interactive case, where audio and - video models run fast enough to hold a conversation or react to a live - webcam. That takes streaming generation, low latency - across modalities, and pipelines that compose models from different - runtimes without falling apart. + Code was expensive to write. Every line an engineer produced had to be amortized across + many applications, so everyone settled for the average case of a shared stack.

-
    -
  • Streaming generation at sub-second latency
    • -
  • Audio and video models that respond in real time
    • -
  • Heterogeneous pipelines, composed from off-the-shelf models
    • -
-
- -
-
02
-

Agents that do the systems work

+
+
+
Now

- Making these pipelines run efficiently is slow, specialized work, and - it has to be redone for every new application. FlashRT is our first - step toward letting an agent do it: write the deployment, - parallelize the models, and check the result against a reference, - with a person supplying only that reference. + For an agent, lines of code are cheap. It writes, rewrites, and discards code at + negligible cost — so a system specialized to your application, your + hardware, and your efficiency targets stops being a + luxury.

-
    -
  • Agent-written placement, streaming, and parallelism
    • -
  • Every variant checked against a ground-truth reference
    • -
  • Decisions gated on measurements, not guesses
    • -
- +
-
- - -
-
-
-
The system
-

An agent that deploys real-time multimodal applications.

-

- You write a simple single-GPU reference. FlashRT lifts it into an - optimized multi-GPU deployment, choosing placement, streaming, and - parallelism for itself. It does this through a chain-of-program - workflow: lower the reference one pass at a time, and check the work at - every step. -

-
-
-
- Before vs FlashRT: where teams once hand-engineered each pipeline, a single agent now produces all of them from references. -
- Before vs. FlashRT. Where each new pipeline once - demanded its own systems team, a single agent now produces all of them - from references. -
-
-
-

- Until now, every new real-time multimodal application (a world model, - a live avatar, a speech agent) came with its own round of systems - engineering: deciding what to disaggregate, what to stream, how to - shard, and where to put each model. -

+
+

02 · Reuse ideas, not frozen code.

+
+
+
Then

- FlashRT hands that work to one agent. You write a - plain single-GPU reference for the application, and the agent makes - and verifies every placement, streaming, and parallelism decision - inside a measurement-gated loop. + Traditional stacks compose only through exact, strictly compatible interfaces. Code that + is even slightly incompatible cannot be reused at all — so reuse meant frozen, + portable artifacts.

+
+
+
Now

- What comes out is a single serving system that targets very different - applications, with no per-application code on the systems side. + An agent reads code the way an engineer does. It takes the ideas from a + system that almost fits — kernels, schedules, parallelism strategies — and + re-implements them for the case at hand.

+
-
- FlashRT workflow: from human-written baseline code, the agent constructs and analyzes a hierarchical graph IR, then runs a self-driven validation loop that implements, verifies and benchmarks, and re-hypothesizes each variant before composing the best strategies. -
- The chain-of-program workflow. From a human-written - baseline, the agent builds and analyzes a hierarchical graph IR, then - runs a self-driven loop that implements, verifies, and benchmarks each - variant. Strategies that hold up (here, streaming and a disaggregated - DiT + VAE pipeline) get composed; ones that don't (co-located sequence - parallelism, for this throughput target) are dropped. -
-
- -
-
-
1
-

Lift

-

Turn the reference into a hierarchical graph IR that makes data - dependencies, persistent state, and streaming edges explicit.

-
-
→
-
-
2
-

Validate

-

Run the IR through a sequential interpreter and diff it against the - reference, so every later step stands on verified ground.

-
-
→
-
-
3
-

Analyze

-

Run static analyses over the IR to surface a finite list of legal - moves: streaming, disaggregation, and intra-model parallelism.

-
-
→
-
-
4
-

Iterate

-

Work the candidates in a measurement-gated loop: implement, verify - against the reference, benchmark, then re-plan from the numbers.

-
-
+

One reference, many systems, written online

+

+ This is not at all hypothetical — every deployment in the results above is, in fact, such a system. From + one reference implementation, the agent can write a latency-oriented system when you are chasing + responsiveness, a throughput-oriented one when you are chasing raw throughput, or a balanced + one in between, then re-specializes it for however many GPUs you actually have. +

-
-
-
Insight 1
-

Plan in passes, not one jump

-

An agent can't turn a reference into an efficient deployment in a - single jump. Make it build an IR, analyze that, and only then lower it - into a deployment, and the quality climbs. We call this - chain-of-program, after the way compilers lower code in passes.

-
-
-
Insight 2
-

Ground the loop in real measurements

-

Prompted naively, the agent writes broken code and stops exploring - early. Instead it writes a test harness that drives simulated input - through the deployment's buffers, the way a real frontend would, then - proposes a change, verifies it against the reference, - and re-plans from the measured latency and frame rate.

-
-
-
+

+ And there is a cost we rarely question. A shared stack must serve everyone, so it piles up + — compatibility paths, configuration surfaces, layers upon layers — until + maintaining and extending it becomes a project of its own, and understanding and debugging it + becomes a tremendous challenge for humans and agents alike. In the era of agents, it is worth + asking whether that structure is still a helper or a blocker — + especially for ML infrastructure, whose entire job is to connect models, systems, and hardware. +

+

+ The foundation of ML software shifts from shared infrastructure to a general + agent — and FlashRT is our first concrete step. +

- -
-
-
-
Team
-

Built at Infini-AI Lab.

-

- A research group across Carnegie Mellon and the University at Buffalo, - in collaboration with AMD, working at the intersection of generative - models, systems, and agents. +

+

🎯Why We Build This

+ +

+ FlashRT sits between two problems we care about: making generative models fast enough to use + interactively, and handing the repetitive systems work behind them to an agent instead of an + engineer. +

+ +
+
+

Interactive generative models

+

+ Most generative models are still used in batches: send a prompt, wait, get a file back. + We are after the interactive case, where audio and video models run fast enough to hold a + conversation or react to a live webcam.

+
    +
  • Streaming generation at sub-second latency
    • +
  • Audio and video models that respond in real time
    • +
  • Heterogeneous pipelines, composed from off-the-shelf models
    • +
- -
- -
KA
-
Krish Agarwal
-
CMU
- - -
ZC
-
Zhuoming Chen
-
CMU
- - -
ZG
-
Zhenyu Gu
-
AMD
- - -
AR
-
Atri Rudra
-
University at Buffalo
- - -
BC
-
Beidi Chen
-
CMU · PI
- +
+

Agents that do the systems work

+

+ Making these pipelines run efficiently is slow, specialized work, and it has to be redone + for every new application. FlashRT is our first step toward letting an agent do it: write + the deployment, parallelize the models, and check the result against a reference. +

+
    +
  • Agent-written placement, streaming, and parallelism
    • +
  • Every variant checked against a ground-truth reference
    • +
  • Decisions gated on measurements, not guesses
    • +
-
-
-
-
- ⚡ - FlashRT · Infini-AI Lab +
+ + + +
+

⚙️The System

+

+ A developer writes a simple single-GPU reference. FlashRT guides a coding agent to lift it into + an optimized multi-GPU deployment, choosing placement, streaming, and parallelism, through a + chain-of-program workflow: lower the reference one pass at a time, and check the work + at every step. +

+ +
+ Before vs FlashRT: where teams once hand-engineered each pipeline, a single agent now produces all of them from references. +
+ Before vs. FlashRT. Where each new pipeline once demanded its own systems + team, a single agent now produces all of them from references. +
+
+ +

+ Until now, every new real-time multimodal application (a world model, a live avatar, a speech + agent) came with its own round of systems engineering: deciding what to disaggregate, what to + stream, how to shard, and where to put each model. +

+

+ FlashRT hands that work to one agent. A developer writes a plain single-GPU reference + for the application, and the agent makes and verifies every placement, streaming, and + parallelism decision inside a measurement-gated loop. +

+

+ What comes out is a single serving system that targets very different applications, with no + per-application code on the systems side. +

+ +
+ FlashRT workflow: from human-written baseline code, the agent constructs and analyzes a hierarchical graph IR, then runs a self-driven validation loop that implements, verifies and benchmarks, and re-hypothesizes each variant before composing the best strategies. +
+ The chain-of-program workflow. From a human-written baseline, the agent + builds and analyzes a hierarchical graph IR, then runs a self-driven loop that implements, + verifies, and benchmarks each variant. Strategies that hold up (here, streaming and a + disaggregated DiT + VAE pipeline) get composed; ones that don't (co-located sequence + parallelism, for this throughput target) are dropped. +
+
+ +
+
+
1
+

Lift

+

Turn the reference into a hierarchical graph IR that makes data dependencies, persistent state, and streaming edges explicit.

+
+
+
2
+

Validate

+

Run the IR through a sequential interpreter and diff it against the reference, so every later step stands on verified ground.

-
- GitHub - Contact - Back to top ↑ +
+
3
+

Analyze

+

Run static analyses over the IR to surface a finite list of legal moves: streaming, disaggregation, and intra-model parallelism.

-
© 2026 Infini-AI Lab. Carnegie Mellon University.
+
+
4
+

Iterate

+

Work the candidates in a measurement-gated loop: implement, verify against the reference, benchmark, then re-plan from the numbers.

+
+
+ +
+

Insight 1 — Plan in passes, not one jump. An agent can't turn a + reference into an efficient deployment in a single jump. Make it build an IR, analyze that, and + only then lower it into a deployment, and the quality climbs. We call this + chain-of-program, after the way compilers lower code in passes.

+
+
+

Insight 2 — Ground the loop in real measurements. Prompted naively, + the agent writes broken code and stops exploring early. Instead it writes a test harness that + drives simulated input through the deployment's buffers, the way a real frontend would, then + proposes a change, verifies it against the reference, and re-plans + from the measured latency and frame rate.

+
+
+ + +
+

📄Citation

+ 
@misc{agarwal2026flashrtagentharnessguiding,
+  title         = {FlashRT: Agent Harness for Guiding Agents to Deploy Real-Time Multimodal Applications},
+  author        = {Krish Agarwal and Zhuoming Chen and Yanyuan Qin and Zhenyu Gu and Atri Rudra and Beidi Chen},
+  year          = {2026},
+  eprint        = {2607.18171},
+  archivePrefix = {arXiv},
+  primaryClass  = {cs.LG},
+  url           = {https://arxiv.org/abs/2607.18171}
+}
+
+ + + -
+ + +