External WebCodecs samples

[tiuvi]

I am currently developing a demo in WebCodecs that covers several key functionalities. It uses getUserMedia and MediaStreamTrackProcessor to capture and process video in real time. Additionally, it includes support for decoding WebM and mp4 files using the JSFile and MP4Box repositories.

This demo is modular and supports the integration of any image editing library. I am currently incorporating PixiJS and Canvas to explore various visual manipulation techniques.

Additionally, I plan to add artificial intelligence using MediaPipe to the image editing tools. This expansion not only enriches functionality, but also provides a concrete example of the potential of the WebCodecs API.

My goal with this project goes beyond creating a robust demo on GitHub. I want to offer a practical guide for developers interested in WebCodecs, highlighting the emerging possibilities and future of this API. This demo will serve as an educational and inspirational resource for the developer community.

DEMO
https://tiuvi.com/videoEditorWebCodecs/

This library only has two files
https://cell1.tiuvi.com:1030/videoStreamWorkerPixi.js
https://cell1.tiuvi.com:1030/videoStreamWorker.js

The project is moving very quickly and I can't upload all the changes daily on github but I will keep it updated.
We also need to add support for npm since jsFidle does not compile with Babel, I have to investigate.
https://github.com/tiuvi/tiuvi-editorVideo

[dalecurtis]

Thanks! I think it's probably easier just to keep this discussion up with the link to your demo.

To land it in repo you'd need to rework it the existing samples directory structure and work from GitHub pages.

I think using this discussion page for folks to link there demos for folks to find would work well. Folks feel free to link your samples in this show & tell thread. Thanks!

[breezefeng]

Adding one: videocompress.dev, a video compressor
that runs entirely client side. WebCodecs where the browser can handle the
container, ffmpeg.wasm as the fallback, mediabunny for demux and remux.

Two things from building it that might be useful to others working with
VideoEncoder and VideoDecoder:

Containers the browser cannot demux are often worth rewrapping rather than
giving up on. Plenty of AVI files carry H.264, and a lossless rewrap to
Matroska puts them back on the WebCodecs path. Measured on a 20 second 1080p30
clip against a geometry matched control that had to take the ffmpeg path: 7.5 s
versus 53.4 s, with output sizes within 2 percent. The catch is that AVI stores
no per-packet timestamps, so -c copy into Matroska fails outright and emits a
1151 byte header with no clusters. -fflags +genpts is what makes it work.

Encode capability is a property of the target configuration, not of the file.
I originally probed canEncodeVideo once per file and cached the answer, which
meant a source rejected at 4K stayed on the slow path after the user picked
720p. Keying that cache by codec, dimensions and bitrate fixed it.

The two pages that get the most use are
compress video for Discord
and compress video to 10 MB,
if you want a specific case to poke at rather than the generic one.

[tiuvi]

We'd need to see your code to analyze the problem. If you're using the webcodec demuxer, be aware that it's not a frame-by-frame field recording, and if the system gets overloaded, the voice will distort.

To copy and compress frame by frame, there are only two demux libraries: one for WebM and one for MP4. The WebM library is abandoned and has some bugs. Then, to mux frame by frame, you also only have MP4. You can use fmpeg to output the frames on the web with WASM; I don't know of an option that does it in a stream, but if you're doing it on a server, then yes.

Furthermore, ffmpeg WASM is much slower than the times you mentioned.