Skip to content

fix(webv2): audio uploads as Ref2VA references - #206

Open
lstein wants to merge 4 commits into
mainfrom
fix/audio-reference-default-conditioning
Open

fix(webv2): audio uploads as Ref2VA references#206
lstein wants to merge 4 commits into
mainfrom
fix/audio-reference-default-conditioning

Conversation

@lstein

@lstein lstein commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

What

Follow-ups to the audio-upload ingest from #195, in the Ref2VA reference list:

  1. A reference made from an uploaded audio file starts on "Audio only" instead of "Video + audio" — it was conditioning the generation on a picture of the sound as well as the sound.
  2. The add button is now "Add video or audio", and its file picker offers the accepted media extensions explicitly.
  3. Windows Media (.wmv/.asf/.wma) uploads are accepted.
  4. CI: the frontend-webv2-tests job's timeout-minutes raised from 10 to 20.

How

Audio-only default. The ingest route already stamps media_origin: audio_upload on wrapped audio uploads. getDefaultReferenceConditioning(metadata) reads that stamp, and VideoReferenceListField fetches the clip's metadata in parallel with the gallery resolve, so the add costs no extra round trip; a failed fetch falls back to the ordinary video default. Reading the server's stamp rather than the picked file's MIME type also covers a clip uploaded through the gallery first and then dragged onto the reference list. Recall (which restores the recorded conditioning) and the reference-extend anchor are untouched.

One button, not two. An uploaded audio file becomes a waveform video, so it fills a video reference slot and shares that cap. Two buttons would show the same counter and grey out together — "Add audio (3/3)" disabled because you added three videos — so the single button is relabeled instead. The picker keeps the video/* and audio/* wildcards and adds the extension list: a file whose type the OS cannot map arrives as application/octet-stream, and then only the extension matches.

Windows Media. Verified against the bundled ffmpeg (v7.0.2, which demuxes ASF and decodes wmv1/2/3, VC-1, wmav1/2/pro): a .wmv fixture transcodes to H.264/AAC and a .wma fixture wraps into a waveform clip. .wmv/.asf/.wma added to the upload route's extension lists and to the gallery's client-side classifier so the two agree.

CI timeout. check:architecture is lint + the unit/browser/fixture suites + a production build + the performance and project-file gates — ~9-10 minutes against a 10-minute cap (recent runs: 571s, 585s, 594s, 597s), so slightly slow runners cancelled healthy runs; two were cancelled on this branch and one on an unrelated PR. Happy to split this commit into its own PR if you'd rather.

Testing

  • getDefaultReferenceConditioning unit tests (stamped / unstamped / absent metadata).
  • _classify_upload parametrized tests for the octet-stream extension fallback (.wmv, .asf, .wma, .MOV, .M4A, rejection of .txt); classifyGalleryUpload rows for .wmv/.WMA.
  • pytest tests/app/routers/test_videos_multiuser.py — 66 passed.
  • pnpm run check:architecture green locally (509 unit + 158 browser files, perf and project-file gates included).

🤖 Generated with Claude Code

https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A

An uploaded audio file is stored as a rendered-waveform video, so a
reference made from one arrived on "Video + audio" -- conditioning the
generation on a picture of the sound as well as the sound.

The ingest pipeline already stamps `media_origin: audio_upload` on those
clips. The reference add path now reads that stamp (fetched alongside the
gallery resolve, so it costs no extra round trip) and starts such a
reference on "Audio only"; the selector still offers all three modes.

Reading the server's stamp rather than the picked file's type also covers
a clip uploaded through the gallery first and then dragged onto the
reference list. An unreadable metadata record is not an error -- it just
means the ordinary video default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
`check:architecture` runs lint, the unit/browser/fixture suites, a
production build, and the performance and project-file gates -- ~9-10
minutes on a standard runner against a 10-minute cap, so a slightly slow
runner cancelled a healthy run. Recent runs: 571s, 585s, 594s, 597s, and
two cancellations at 617s/624s (one on an unrelated PR).

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
@github-actions github-actions Bot added the ci-cd label Sep 3, 2026
An uploaded audio file becomes a waveform video, so it fills a VIDEO
reference slot and shares that cap -- a separate "Add audio" button would
carry the same counter and grey out with it, which reads as a bug. The
one button is relabeled instead, and its picker now lists the accepted
extensions explicitly alongside the video/* and audio/* wildcards: a file
whose type the OS cannot map arrives as application/octet-stream, and
then only the extension matches.

Also accepts Windows Media (.wmv/.asf/.wma), which the bundled ffmpeg
demuxes and decodes -- verified end to end: .wmv transcodes to H.264/AAC
and .wma wraps into a waveform clip. Added to the upload route's
extension lists and the gallery's client-side classifier so the two
agree.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S4B5exWsbC2z2Uu2tA167A
@lstein lstein changed the title fix(webv2): default wrapped audio references to "Audio only" fix(webv2): audio uploads as Ref2VA references Sep 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant