✨ feat(video-engine)!: a browser video engine, and annotations that draw on the frame it presents - #93
Draft
cfviotti wants to merge 337 commits into
Draft
✨ feat(video-engine)!: a browser video engine, and annotations that draw on the frame it presents#93cfviotti wants to merge 337 commits into
cfviotti wants to merge 337 commits into
Conversation
…e scene Ellipse, marker, box-corner and mask-halo renderers reached the renderer but stopped at the scene, so a presentation carrying any of them drew nothing. The scene now composes the three shape kinds onto whatever shape style the caller passed, and forwards the halo style to the mask layer. Region renderers cropping to a detection's own mask get the full pipeline: the scene keys the prepared artifact by which targets crop that way, emits an invisible coverage instruction for a target with no mask of its own to draw, and hands the layer the active coverage frame and the media texture. Mask preparation now also runs for a halo-only or coverage-only presentation, which previously had no reason to cook an artifact at all. The shared Pixi test double gains the alpha mask and blur filter the region and halo paths construct, keeping this branch's buffer-image source alongside.
…d picking Hiding a class left it lit: ambient focus targets every detection in the frame, so a class the caller had hidden was still cut out of the dim overlay. The focus layer now takes a visibility predicate and drops those targets before it decides whether it has anything to draw. Dragging a detection had the same shape of bug from the other side. Its hover and selection presentation redrew from the stored geometry while the base layers were already drawing the gesture, so the highlight trailed behind the shape. The interaction presentation layer already knew how to suppress that; nothing was telling it when. The renderer core's detection loads carry the media coordinate space, so its assertions now expect the third argument.
Covers what the halo and visibility ports actually promise: two detections asking for different spreads get one blur pass each with only their own id in the palette, a halo still draws when mask preparation falls back to the RGBA composite, and a hidden detection draws no interaction presentation at all.
…ource The tracking playground needs to sit between the fixture's chunked source and the session so it can re-track loaded frames, and it needs to pause, seek, refresh and resume around a re-track. The fixture loader now takes a source wrap, and the renderer hook exposes the playback and refresh controls those steps need. Playback control names follow the hook's own vocabulary rather than the props of the one component that used to consume them, and seeking resolves when the seek has settled instead of returning before it. The region-effects sample opened on a blank picture in the sample picker, which its own showcase hid because that playground pins every layer off. It now opens with masks drawn, the geometry its detections are built around.
… fixtures The public API guide and the internal ingestion note each described half of the merged tree. They now describe all of it: coordinate-space projection, tracking post-processors, region media crops, the new renderer kinds, typed media failures, live append and coverage finalization, and bounded retention, next to this branch's video-engine source and its rule that playback never waits for annotations. The keypoint showcase fixture the tracking playground opens on was missing, so that playground silently fell back to a different sample. It is here now, on the 30fps proxy its detections were computed against. The geometry fixture tool takes upstream's head-region pipeline with this branch's guard on top: a pose run measured against a different frame grid is rejected rather than warned about, because every frame index resolves against any grid and the skeletons would simply land on the wrong frames. The React Native patches now apply on install, and the lockfile carries both new workspaces and the tool that applies them.
Scene fixtures now carry the four renderer styles the options require, and the halo test names the prepared frame by the representation this branch keeps.
…aces A caller reading `requiredForPlayback` on a composed detection source sees a name that promises a playback gate, and this build has none. The web-side option and the multiple-sources recipe both say so outright; the core type, which is the one a library consumer reads first, stopped at "waitForRange skips this entry" and left the gate question open. The docs contract test checks the file, not the doc comment, so the core surface satisfied it on prose written for `playbackGate` several hundred lines away. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Upstream renders through WebGL. This branch's push presentation forces WebGPU. A shader written in GLSL alone therefore draws nothing here, and does it in the worst possible way: no error, no failing test, a clean typecheck, and a picture that is simply missing something. Two arrived that way in the merge. Exact-mask media regions, upstream's headline feature, drew nothing at all: bare video where the cropped heads should be. The mask halo drew nothing either, which is harder to notice because its absence looks like a style choice. Regions had two further breaks stacked under the shader, both invisible to a compiler. The region layer was handed the canvas the pull path uploads into, which nothing writes to under GPU compositing, so crops sampled an empty surface. Handing over the presented texture then exposed the last one: the demo plays a proxy smaller than its media, and the region layer addresses in media coordinates, so every crop was computed against the proxy's pixel size and sampled the wrong part of the frame. That is the misalignment a human spotted before any test did. Attribution was proved by isolation rather than argued: forcing WebGL drew the crops while WebGPU did not, removing the WGSL alone returned it to bare video, and disabling the alpha mask alone gave rectangles. The halo is pinned the same way: against a paused frame, the fixed build changes 19 percent of the canvas when the halo is enabled, and the build from HEAD changes zero pixels. A test now walks the renderers directory, finds every shader by reading it, and fails when one lacks a program for either backend or names a resource its WGSL does not declare. Discovery rather than a list, so a shader written next month is covered by nobody remembering anything. It fails on both shapes of this defect, verified by introducing each. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The horse trail's detection chunks were stored one JSON token per line. That is 2.1 million lines for 71 files, and it made every regeneration look like a rewrite of the repository: a pull request that touched them read as eight hundred thousand changed lines, which is not a diff anybody can review. They are minified now, which is what the basketball fixtures already were. The loader parses them, so the formatting was never load-bearing. 2,139,080 lines become 102, and the tracked payload drops about thirty percent. No detection, mask, polygon or keypoint value changes: the files are reparsed and re-serialised, not rebuilt. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The sample picker offered three basketball clips that a viewer could not tell apart. Two were the same nine seconds of the same game and differed only in which model run produced their detections and whether the demo played the source file or a 30fps transcode of it. There is one now, and it is the better of the two: five annotation kinds on the clip's own 25fps frames, where the removed fixture had four on a resampled proxy. The three documentation playgrounds, the annotation renderers, the homepage basketball demo and the tracking post processor, all open on it. Nothing on those pages looks busier or emptier than before. The merged fixture draws 10.9 detections a frame at its own confidence gate against the removed one's 11.0, and the per-second profile matches: 4.4 at the opening rising to a plateau of 11.8 to 13.1. Two pages needed care to keep looking right: - The polylines page pins its confidence gate to zero. That page scopes itself to the ball's one derived trace, and the fixture's 0.5 gate hides 200 of the 224 trace segments, so the page drew the ball with no trail at all. - The tracking page stopped inventing a frame count. It read "0/270" while loading, which was the removed fixture's length quoted as a fact. It reads "0/0" until the real number arrives. The clip's media also stopped being tracked twice. `basketball_sample.mp4` was committed in two fixture directories; the second was a Git LFS pointer, so a clone was fetching the same 22MB payload a second time for nothing. Both fixtures share the one copy. `benchmark/masks/run.mjs` located that media through the manifest's provenance record, which names a path that no longer exists, so it now reads `fixture.meta.json`, which is what the demo itself uses. The removed fixture's pose run moves to `basketball_regions`, the only thing that still reads it, and the fixture builder's defaults follow the fixture that survives. Its README now carries the geometry coverage and the provenance the removed one held, including the two model runs that cannot be reproduced from this repo as it stands. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Anyone installing `supervision` and reaching for video got told to install
`supervision-js-video-engine`, and that package did not exist. The release
workflow built and published one tarball, the browser package, and the engine
sat at version 0.0.0 with nothing to publish it.
The release now ships both. The engine goes first, so the browser package never
reaches the registry naming a peer that cannot be installed, and it publishes
through the same guards the browser package already uses: the manifest version
is the source of truth, an already-published version is a silent no-op rather
than a failure, and a prerelease and its dist-tag have to agree.
The engine starts at 0.1.0, matching how this project released its first
browser version, and the browser package moves to 0.1.8. It had been sitting at
0.1.7, which is what npm already serves, so the next release would have been
refused as a republish.
The optional peer range narrows from `"*"` to `"^0.1.0"`. The old range would
have accepted a future incompatible major, and the failure would have surfaced
at run time inside `openVideoEngineMediaSource` rather than at install.
One step still needs a person, once. npm will not attach a trusted publisher to
a package name that does not exist yet, so the very first engine release fails
until someone registers the name from their own machine:
npm trust github supervision-js-video-engine \
--file publish-npm.yml \
--repository roboflow/supervision-js \
--environment npm-publish
The engine also gets the LICENSE and README that npm always ships regardless of
the `files` list, so its package page is not blank. Its entry-point table was
checked against the manifest's own `exports` map rather than written from
memory.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The halo glow vanished entirely on any scene where some masked detections are filtered out of the picture. Every player kept their silhouette, and not one of them glowed. A single wide mask was enough to do it. The id mask raster holds one identity per pixel, so the last mask written to a pixel owns it, and the halo reads the identities back to find the silhouettes it paints. The layer that prepares that raster admitted every detection carrying a mask, including the ones the halo declines to paint. A full-frame detection at low confidence therefore claimed every pixel on screen, buried the identities of the players above it, and left the halo with a palette full of entries and no pixels to match them. The preparation and the paint now ask the same question through one predicate, so they cannot disagree again. A halo that paints nothing, whether because it has no mask, no instruction, no opacity or no spread, also claims nothing. That last case was live: a demo halo style reports its configured opacity unconditionally, so at zero glow opacity every masked detection was still claiming raster pixels while painting nothing at all. Preparing that coverage is expensive, so the scene reuses it until the set of detections the halo admits actually changes. It compares the two styles over the detections currently buffered instead of assuming any restyle is a new set, which is what a style whose only member is an arbitrary function allows. Moving the spread slider through twelve steps cooked mask coverage 204 times before and cooks it zero times now; driving glow opacity off zero still costs the 17 cooks that genuinely have to come back. One gap stays open and is documented where it lives: a prepared artifact can outlive the buffered window, so a restyle that moves the admission boundary on a frame the buffer has rolled past keeps a stale artifact. Closing it exactly needs `MaskHaloStyle` to carry the identity `MaskStyle` already carries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scrub group reported four seek timings above a population nobody could see, and two counters the engine broadcasts every tick were read by nothing at all. A reader looking at four timings and a zero had no way to tell "zero because nothing asked" from "zero because the counter is broken". A `Cursor seeks` row now reads them, split as exact and key. Reading it against `Seeks` in the group below answers a question that has cost real time twice: a seek issued while the video plays re-anchors playback instead of moving the cursor, so it lands in neither count and times nowhere. Paused, seven seeks read seven exact. Playing, the same seven seeks read zero here and seven there. Both ledgers are on screen and visibly disjoint. No engine counter was changed to make the panel look busier. The engine was already counting these correctly. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three ways the demo eval could report a number that was not true. **A retried scenario measured a warmer page than its first try.** When a scenario was disturbed and retried, the retry ran on the page the first attempt had already warmed, so it looked dramatically better and the harness kept the better number. Measured on the drag scenario, attempt one to attempt two went 32.6ms to 8.1ms stale and 50 to 91 frames a second, on the same build in the same minute. A retry now reloads first, and the two attempts land together: 32.3 to 27.1ms and 50.8 to 62.2fps. The reload costs about 0.6 seconds and only a disturbed attempt pays it. `cadence` keeps its page on purpose, because it selects its own fixture and a reload would drop the demo back to the default clip while every number still named the other one. **The paints scenario could not see a pause that keeps drawing.** It waited six seconds before it started tracing, so anything that decayed after a pause was already over. It now starts the trace first and pauses inside the window. Twenty passes put the settling burst at 167 to 177ms and 11 to 15 paints, with zero paints once settled, so the new budget sits five times wider than the widest pass: the gate is for a pause that keeps drawing, not for the transition. **No recorded number said which tree it came from.** A report could be compared against a baseline taken on different code with nothing to catch it. Reports now carry the commit, whether the tree was dirty, and the fixture the scenario ran on, and the baseline comparison warns before it prints a single delta. The ten guessed noise floors are untouched. Picking numbers without measuring is the failure being fixed here, and the paints scenario's neighbours now read slightly outside two of them, which makes those floors the next thing to measure rather than the next thing to widen. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Eleven documents in this repo were false at the same time and every gate stayed green. A fixture README told a reader to run a script that has never existed. The root README named a release that had already shipped. A rebuild command silently read a different input than the one it named. The gate checked that links resolve and that the API facades cover every export. It never read a claim. It now checks six kinds of claim across all 81 tracked Markdown files: - a path a document names has to exist - an `npm run` script it shows has to be declared, workspace forms included - a flag it passes has to be one that script actually parses - a checksum it quotes beside a path has to match that file - a version it states beside a package has to match that manifest - a module it imports has to export what it imports Each was proven able to fail by injecting the failure and watching the gate catch it, including the two real ones above. The link check widened from a subset to all 95 links in the corpus. Six live violations turned up, all in planning documents: a module that never existed, a proposed filename read as an existing path, three references to a module that was renamed before it shipped, and an API sketch importing three symbols under names the package does not use. Counted claims like "nine tsc projects" are not checked. A number in a sentence has no mechanical link to the set it counts, and a gate that guesses is worse than none. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`npm run verify` was failing on this branch before any of today's work, on two counts that had nothing to do with each other. The mask benchmarks declared `setTimeout` and `clearTimeout` in a `/* global */` comment. This branch had already added those to the shared eslint globals, so every one of them was reported as redeclaring a built-in. The comments keep only the globals the config does not supply. And a fixture tool had drifted out of Prettier's shape. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The branch had upstream's code but not upstream's history. Whoever integrated the 23 commits upstream shipped after the fork did it by copying files, so git recorded no second parent. Every one of those commits still counted as absent, GitHub computed the pull request's diff from the fork point, and the branch would not have merged at all. It merges now, and the tree it produces differs from this branch by eight files. That is the whole point: our side was already a superset, and this records the history to prove it rather than asserting it. Resolving 52 conflicts, the rule was never "which side wins" but "does upstream have something we lack". Every hunk that answered no took ours; the ones that answered yes are ported below. **Ported from upstream** - The docs for `pruneFrames` and `appendLiveFrame`. We carry both and had no written account of either. - The annotation roadmap's current renderer vocabulary and its guidance on the smallest public addition to reach for. - Eight tests: four interaction-presentation tests, two `basketball_regions` fixture tests, a halo-only renderer list, and screen-sized region assets holding steady across a paused zoom. - Two stylesheet rules that fix real gaps here. Without the first, the tracking playground cannot scroll on a narrow viewport. Without the second, hiding a class had no visual affordance even though the markup already emits the modifier. - Three behaviours the resolution would otherwise have dropped: a hidden detection is no longer pickable, region badges redraw on a viewport-style change instead of holding stale geometry, and the editing overlay draws keypoints with the style its host configured. Upstream added 45 stylesheet selectors since the fork and only those two rules were missing here; everything else names a class these components never emit. **What resolving this taught, recorded because it nearly went wrong** Ten places where git auto-applied an upstream hunk outside every conflict region, because our copy-based integration had moved the same code elsewhere. Keeping the HEAD half of each marker would have shipped a duplicated key, four duplicated function definitions, a duplicate block-scoped constant that does not compile, and a reference to a variable this branch never declares. Files were resolved by whole-file replacement, never by patching the marked regions. **What this merge does not take, and why** Two upstream regression tests for seeking while buffering. The production fix they guard is already here; only the coverage is lost. They cannot be ported as written because they drive the renderer into buffering through the playback gate this branch removed, and the option that gate reads is still accepted and ignored, so they would compile and never reach the state they assert. Two upstream tests asserting that playback waits for detection coverage, which this branch's own tests assert it never does. They are direct opposites and cannot coexist. Upstream's editing-gesture hide, which stops the base layers drawing a detection while a gesture previews it. This branch keeps drawing it and moves it instead. That difference stays a deliberate decision rather than a merge artifact. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pressing play when the renderer cannot start it did nothing visible. No error, no state change, no hint that anything had been attempted. The renderer's own `togglePlayback()` decides pause or play and drops the rejection from the play it starts. That decision has to stay where it is: it has a branch for a drag in flight, where the producer sits mechanically paused and reads as not playing, so a caller that reads the state and calls play itself would resume a clip the viewer had just paused mid-drag. So the reporting goes onto `play` instead, on the renderer the demo adopts. The play that `togglePlayback()` starts and drops now reaches the same error line every other failure in the demo uses, and callers that already handle the rejection keep handling it. Also drops a `createImageBitmap` stub from a session test. Its comment said it was there to make the pipeline take a mask path that this branch does not have, and a counting probe confirmed the global is called zero times on that path. The test still bites: removing the halo renderer from the presentation fails it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five options let a consumer ask playback to wait until detections and prepared masks cover the moment being played. All five compiled, all five were documented as doing nothing, and all five did nothing. A consumer who set one got silence. They work again, and the default is unchanged: the picture still moves first and annotations still catch up behind it. Setting nothing gets exactly what setting nothing got yesterday. Ask for the gate and playback waits, on play and through the playback loop after a seek, which is what it did before it was removed. The reason for the option is that both behaviours are legitimate. A viewer scrubbing through footage wants the picture immediately and can accept annotations arriving a beat later. A reviewer stepping frame by frame to judge a model would rather wait than see a frame with nothing drawn on it. That choice belongs to the application, not to us. **The trap this nearly shipped with.** Two session defaults still resolved the gate as enabled. They were harmless while nothing read them. Reviving the option without touching them would have turned waiting on for every media session and every render preparation, which is the opposite of the intent. Both now resolve off, and their lookahead numbers stay, so an application that opts in still inherits sensible tuning. Five existing tests set the flag and asserted that nothing waited, which was only true while the flag was inert. Each now tests the real default with nothing set, and the gated case sits beside it. Four tests come back that could not exist while the gate was gone, including two regressions covering a seek taken while buffering. The fix they guard was never lost, but nothing had been able to reach the buffering state to prove it. Documentation stops describing a no-op. Every surface that names the gate now says what it does, what it does not do, and that it ships off, and the contract test that pins those surfaces was rewritten to check for that instead. Two limits worth stating. The gate is a pull-path feature: a push producer never builds the controller that owns the wait, so a push session that enables it gets no gate, exactly as before. And enabling it for detections without a lookahead is inert, because the required coverage ends where playback already is. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
… canvas This is a computer-vision library, and its performance gate spent its headline budget on how often the repo's own demo repainted a control bar. Ten metrics measured the demo application rather than the library: DOM paint, style-recalc and layout rates, the settling time after a pause, a keyboard shortcut scenario whose entire code path lives in a demo component, and a playhead position parsed out of a demo element's CSS transform against a limit calibrated to that component's half-pixel quantizer. None of them says anything about what a consumer installs. They are gone. The paint budget was worse than useless. It excluded the canvas by matching a paint event's rectangle against the canvas box, and that never matched once, so the number it judged was always the whole page. Chrome hands the node name directly, and in a playing window all 366 paint events named a demo element: the timecode, a cell value, a timeline segment, the inspector column. Not one named the canvas. The rectangles show why the match could not work, since a paint clip is a cull rect and not a damaged region: the root document reported 3000x2300 on a 1500x1150 viewport. **What replaced it answers the question that was actually worth asking.** A canvas presenting video has to paint once per presented frame; painting more than that is waste. Nothing compared the two, though the harness collected both. It does now, and the answer is that the renderer draws exactly once per presented frame: seven windows, ratio 1.0000 every time, and zero draws while paused. The budget is 1.05 with no tolerance. Measuring that also priced the thing the paint gate was standing in front of. In a six second window at 27.1 percent main-thread occupancy, every paint event combined costs 0.106ms per frame, while handing each decoded frame across the worker boundary costs 1.862ms. The gate was watching something 17.6 times cheaper than the cost beside it, and that cost is now written down where the next reader will find it. One metric was retargeted rather than deleted. Whether the playhead drifts while the transport is stopped is a real question, so it now reads the library's own clock instead of a demo element's transform. Its limit was re-derived from measurement and came out at zero, because a stopped transport's time is a stored number and does not jitter. Five surviving metrics have library numerators scaled by demo input, and each now says so where its number is read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…t it does Two public shapes that made a reader work harder than the code does. **A highlight had two ways to be styled, and they shadowed each other.** Four fields on the base interaction style set the hover and selection rectangle directly, while `hovered.boxStyle` and `selected.boxStyle` did the same job through the ordinary style path. Setting only one state silently switched the four fields off for that state and left them running for the other, which is a half-migrated config that compiles and quietly draws two different highlights. The four are gone. The remaining path has full parity, including the thing that could have made it inadequate: a box style handed to both states can still tell them apart, because the renderer forwards hover and selection into the style context. It also reaches mask, label, keypoint, polygon and polyline highlights, which the removed fields never did. A default highlight looks exactly as it did. **`requiredForPlayback` had nothing to do with playback.** It picks which detection sources a composite source waits for when it reports a range as covered. Every document that mentioned it spent its second paragraph explaining that the name was wrong, which is a strong signal to change the name rather than keep apologising for it. It is `requiredForCoverage`. Coverage is the word these files already use for what that wait is about, the boolean keeps its polarity and its default, and it no longer reads like a second setting on the playback gate sitting beside it. The paragraphs that existed to walk the old name back are gone, and what is left says what the flag does. **Both are breaking, and one fails quietly.** A TypeScript consumer gets a compile error either way, which is the kind that fixes itself in a minute. A plain JavaScript consumer passing the removed style fields reverts to the built-in highlight. A plain JavaScript consumer who had set the renamed flag to false starts waiting on that source again, and with the playback gate enabled that means playback starts waiting too. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three changes on this branch alter what an application already using `supervision` gets without changing a line of its own code, and a patch number would not say so. The playback gate ships off by default, where upstream shipped it on, so an application that never configured it now sees the picture move before the annotations do. Four interaction-style fields are gone. And a detection-source flag changed its name. Two of those fail quietly in plain JavaScript. Removed style fields revert a custom highlight to the built-in one. A renamed flag reverts to its default, which under an enabled gate means playback starts waiting where it did not. Under the 0.x convention a minor is the signal for that, so this is 0.2.0 instead of 0.1.8. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An earlier pass replaced a paint gate with three canvas metrics, on the reading that the question worth asking was whether the renderer draws more than once per presented frame. It does not, and it cannot: that ratio reads exactly one on every window ever measured, because the loop is driven one draw per frame by construction. The repaint that actually matters happens in the browser's compositor, one level below anything this harness observes, and it is a property of drawing each frame on the main thread instead of in a worker against a transferred canvas. That is a separate piece of work with its own decision to make, and no metric here should imply it has been measured. So all three go. The presented-frame rate was a window average of a number the cadence scenario already gates three sharper ways, including against the engine's own ledger. The paused render count asserted the loop is idle while stopped, which is a cost question wearing a fidelity name; whether a stopped transport holds its clock is already gated at zero drift. Swept the residue with them. A source contract pinned a demo component's playhead geometry on the grounds of main-thread paint load, which this harness prices at 0.106ms per frame. A comment justified pinning the Demo view with a paint census, and now names the reason that still stands: the Debug view's readouts land inside every frame time and long task sampled. The one number worth keeping is the price of the deferred work, so it moves to the document that describes the presentation boundary, as a recorded measurement rather than a gate's justification. It remains disputed: an independent pass measured the same handler an order lower, and both readings were taken on a machine running many jobs at once. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
In the region effects sample the big-head lens jumped off a player's head and
snapped back, frame after frame. It was most obvious when almost nothing was
moving, and some lenses floated over the crowd with no player under them at all.
Every head the model actually saw was fine. The wrong ones were the frames where
the model saw nothing and the head position had to be invented.
Inventing it used the wrong reference. A missing head was moved by however far
the player's bounding box had moved, and that box is set by whichever limb
reaches furthest, usually a raised arm. Vertically the two barely relate. So the
lens tracked an arm instead of a head, and because each invented frame copied
from the single nearest real one, four frames of invention drifted five times as
far as one.
A missing head now sits between the two real observations on either side of it,
which is where it was. And a frame with no player detection at all no longer
invents one by averaging the players before and after, which is what put lenses
in the crowd.
invented heads before 7.2px off, 15.7% badly placed
after 2.8px off, 2.8% badly placed
real heads 2.8px off, 3.5% badly placed
Invented heads are now placed slightly better than observed ones, which is the
point at which they stop being visible as a defect.
Long gaps are no longer filled. Through four frames the fills are
indistinguishable from real observations; at five and beyond the head travels
five or six of its own widths during the camera pan, and no placement rule
recovers that. Four estimators were compared on the same frames and every one of
them was wrong about half the time at seven frames, so those fills are dropped
rather than guessed. Frames keep at least two heads throughout.
Only invented heads changed. Every other detection in the fixture, and every
head the model saw, is byte-identical.
The rebuild runs from the committed fixture, needs no model and no API key, and
is idempotent.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dragging a detection drew it twice. Resizing one drew the old size and the new size at the same time, so a resize showed two boxes with only one of them under the pointer. Upstream solved this by hiding the detection from the base layers for the length of the gesture and letting the preview overlay draw it alone. Half of that arrived here and half did not. The overlay half is present, and its own comment still explains that it draws the box fill because the source is hidden. Nothing hid the source. A move looked less broken only by accident: it takes a shortcut that slides the existing box, so the two landed on top of each other and read as one box with a doubled outline. A resize has no such shortcut, so both were visible. The hide is back, and the layers that have no preview to draw stay out of it: labels and region badges keep drawing and follow a move as they did. Four other call sites had to stay on the unhidden state or the fix would have worked against itself. Two feed the focus layer, which would otherwise have filtered out the very detection it was asked to follow. One decides what can be picked, and hiding there would have dropped the gesture's own selection halfway through the drag. With the detection hidden, the focus cut-out now follows the gesture instead of staying at the position the drag started from, and a hidden detection is no longer pickable. A keypoint style set through a presentation update now reaches the overlay. It was accepted at construction and ignored afterwards, so nothing a host set after the first frame ever arrived, and the setter that was meant to deliver it had no callers at all. One more thing, found while checking the above: mask preparation is invalidated on a visibility change only if the scene names one of a hand-written list of style kinds. The list was written when there was one source of prepared masks and never grew when two more arrived, so a halo-only or region-coverage-only scene kept a stale raster and its hidden detections kept claiming pixels. The condition now asks the resolver instead of restating what it knows. A regression test covers the hide, and a second covers a mask preview not triggering it, since masks have no overlay to draw and must stay visible. Also restores an upstream test for the label surviving a gesture, lost when a file was resolved wholesale during the merge. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The number input and the toggle in the quality controls had square corners while everything around them was rounded. They asked for a radius token the stylesheet never defined, so the browser resolved it to zero. They now use the token that exists. Also removes a playback store that was never wired to anything. It was added in the same change that rewrote the control bar, superseded before it shipped by the live-readout writer that has eight call sites, and never imported once. And a pose tool's usage example pointed at a fixture directory that no longer exists. It names the surviving one. That example lives in a Python docstring, which is why the documentation gate, which reads Markdown, could not see it. Six unused custom properties come out of the root block. A custom property that no rule and no script reads has no computed effect, so this cannot move a pixel; the extracted class set is byte-identical before and after. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Written after the fact, the branch had grown a second way to do work upstream already did, because the new engine feeds the same machinery from a different direction and each seam got its own answer. **Refilling the detection buffer.** Two rules, fifty lines apart in one file, computing the same threshold from the same option, neither mentioning the other. One came with upstream and fired from the pull path; the other was written here because the push path never reached it. The push path now uses the original, which also gets it coalescing and supersede handling it never had. Below two seconds of lookahead the old rule refilled on essentially every playhead move; that is now one refill at half the lookahead, like everywhere else. **A timeline the layers read.** A twenty-eight line wrapper claimed to withhold frames the prepared window did not cover. It withheld nothing: composed, its one non-delegating method was the identity. Its own test asserted the two were equal. The comment above its consumer described a filter that never existed, and that comment is gone rather than reworded. **Decoding an uploaded file.** The demo opened every upload twice, in two demuxers, concurrently, on a branch whose whole premise is that one engine owns decode. It now reads frames through the session it already has. The second mediabunny use stays, because encoding a still image into a one frame clip is a real thing no library entry offers. **Reporting a failed play.** Three mechanisms, one string. One is enough. **Recording that the playhead moved.** Two setters and two near-identical recorders, one per playback path. The legacy path now emits state on a time change, so a loop reset or a seek moves the readout before the frame lands, which is what the other path already did. **Formatting a playback rate.** The same value rendered as `8.0x` in one place and `8x` two lines later. The measured rate keeps its decimal, because it is a float and needs one; the commanded rate does not. Also removes a third frame-selection rule with no callers, a transport method with no callers, an option inert in both of its own branches, a duplicated session block, and a second component sharing a name with one in the same directory. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Changing a marker, ellipse, box-corner or mask-halo style through `setPresentation` drew nothing. The style was stored, the vector layer was updated, a render was requested, and the scheduler declined it. The scheduler decides by comparing a signature of the presentation fields that matter. That signature was a hand-written list, missing exactly the four styles upstream added while this branch was forked. For those four the signature never moved, so the scheduler concluded nothing had changed. The demo hid it completely, which is why it survived: the demo rebuilds its renderer array on every call, so the array's identity always changes and every render lands. An application that keeps its renderer list and swaps one style sees a still picture. The signature now derives its style half from the renderer registry, which is where the mapping from renderer kind to style field already lives and which already carried a helper for exactly this, with a note saying consumers should read it instead of repeating the mapping. A renderer kind added later joins the signature by existing. One hand-written entry stays and one goes. Mask opacity stays, because it is the one value a host is invited to change inside a style object it keeps, and comparing objects by identity cannot see that. A visibility version goes, because it only ever moves when the visibility object itself has already moved. Also folds the two copies of the annotation draw order this branch had added into one declaration. There were five copies in all, and they had already drifted: two of upstream's disagree about whether focus draws before or after the interaction presentation. Resolving the merge required hand-adding a layer to one of them. The remaining three are upstream's and are a separate job, because collapsing them means picking a side in that drift and reshaping a public diagnostics type. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Dragging the timeline, most often backwards, turned the picture solid black
while the timeline, the readouts and the buffer lanes all kept working. The
console carried one line:
Destroyed texture used in a submit. While calling Queue.Submit
A texture was freed while the GPU still held a command that referenced it. The
driver rejects the whole command buffer, so nothing in the scene reaches the
canvas: black picture, live interface.
The scene composites decoded frames into one GPU texture and swaps that texture
whenever the decode changes size, which is exactly what scrubbing does as it
alternates between a low resolution preview and a full resolution frame. On each
swap it restated the texture's size to the renderer. It read the old size back
from the object it had already overwritten, so instead of restating the
resolution it multiplied it: one, then a sixth, then a fortieth, on down.
Once that number is wrong the stated size can happen to match what was stated
before. The renderer reuses a texture binding as long as the stated size does not
change, so those swaps were completely silent: no new binding, no invalidation,
and the next draw ran against a binding pointing at the texture just freed.
The compositor now owns the statement of its own texture's size, and frees the
retired texture only after nothing can still point at it. The invariant is that
the stated size always equals the size of the texture it describes. A swap only
happens when the size actually differs, so with the statement truthful the
binding is always renewed and a stale one cannot survive. No guard and no
deferral.
Measured on the nine second clip, thirty backward drags each: one black frame and
one validation error before, none after. None forwards, and none on the seventy
second portrait clip either way. Canvas brightness across the thirty runs stays
between 107 and 111 where the failure read 2.
The reason it took so long to see: the two suspects were both wrong. Destroying a
texture with a queued copy still unsubmitted produces no error at all, which the
demo does hundreds of times a run, and the video engine creates no textures on
this path. It is a stale binding, not a stale copy.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The scene decides whether to repaint by comparing a list of presentation fields. The list was checked for membership, so it was complete today and nothing said it had to stay that way: a seventeenth field would compile, and the picture would simply never update when it changed. The list is now derived so that any field no renderer kind owns has to be named explicitly, and a field that is neither fails the build. A case per field asserts a single change repaints. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…seeks The playback controller feeds its own next decision, so what it does over a sequence is not what any single step does. The suite drove it at one rate per run, which left three moves untested: changing speed while it plays, a stall shorter than its recovery window, and a burst of seeks. The harness now takes a rate that varies over a run, anchored at each change so a constant rate stays exactly as it was. Three runs cover stepping up to 8x and back, a 400 ms decode blackout, and seeking repeatedly while the transport keeps running. This pins a claim that had no test: a machine with room opens a new rate at the full frame offer, with no ramp. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Closing the engine does not wait for a foreground decode that is already in flight, so a step or seek can land after teardown. The scheduler already refuses that frame and closes its sample; what was covered was only that the sample got closed, not that the frame stayed off screen. The test parks a step decode, closes mid-decode, then releases it, and checks all three ways such a frame could still surface: the step answers with nothing, the cache is not refilled, and a listener that subscribes afterwards gets no replay. Awaiting the step before asserting is what makes it bite. The cache checks run synchronously, so on their own they pass while the late decode is still settling. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The recovery ceilings nest: the first-frame seed is bounded tighter than a random-access decode, and every seed attempt has to fit inside the main thread's backstop or the worker is given up on while it is still doing what it was told to do. Only the outermost of those held by construction, being derived from the decode ceiling. The rest were prose. Raising the seed timeout to 20 seconds spends 60 across three attempts against a 45 second backstop, and nothing said so. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…millisecond The exact tier keyed frames on their timestamp rounded to a whole millisecond, so two frames closer than that shared a slot and the second overwrote the first. Nothing in the engine stops a source arriving at a rate where that happens: it was a bound that held in practice, not one the code guaranteed, in a change whose whole purpose is to stop naming frames by time arithmetic. The tier now keys on the frame's own tick count, which the timeline already derives from every packet in the track and which collapses same-instant pictures when it is built. Two distinct frames cannot share a slot at any rate a container can express. The preview tier still rounds, because its answer is declared approximate and never claims to be the frame at the target. What each tier learns as the source frame interval is now bounded by the grid its own keys round onto, so the same frame arriving twice under float slop cannot be mistaken for a frame gap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…shed The background walk yields to whatever the viewer is waiting on. It counted that read as finished the moment the request resolved, but the demuxer reads the body after that, so the walk went back to competing for the connection while the frames on screen were still arriving. A read now holds the link until its body ends, is cancelled, or errors, and every one of those paths releases it exactly once. A held link cannot be held forever by a reader that walks away, so a chunk left unclaimed past a ceiling releases it. That ceiling covers only the gap between a delivered chunk and the consumer's next pull: it used to span the wait for the server too, which made a slow link indistinguishable from an abandoned read, and standing down matters most on exactly the links that are slow. Disposing releases whatever is outstanding rather than leaving a timer per read behind. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Banking a block allocates, so it can throw. On the two paths that end a read, it ran before the claim on the link was given back, and the claim was already past the ceiling that would otherwise have expired it. An allocation failure there held the link for the rest of the session: the background walk then wakes ten times a second to find the link busy and never prefetches again. The claim is given back first on both paths, and armed before a block is banked mid-body for the same reason. The claim is a closure removed from a set by identity, so a read whose cancel races its own end releases twice and the second is a no-op. Both interleavings are covered, along with a cancel arriving while a pull is still in flight, which is the case with no ceiling left to catch it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The exact tier will not answer with a frame further from the target than the shortest gap it has seen between two frames. On a variable-rate source that bound can be learned from a dense stretch and then govern a sparse one: a frame that owns 990 ms of the clip answers for the first 10 of them, so across that second the cache serves 2% of lookups and sends 98% back to the decoder. Nothing widens the bound again, not even evicting every frame that taught it, so a tier holding two frames a second apart can still be bounded by a gap of ten milliseconds it no longer holds anything to justify. A second tier with the same residents and a different history answers differently, which is what the test pins. This is a miss and never a wrong frame: the at-or-before filter runs first, so the bound can only remove candidates. That is worth keeping true, which is why it is now pinned rather than described. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Playing a clip whose masks fall behind stopped the picture on "Drawing ahead of the video" and left it there. The video only moved again if you paused and pressed play, and then it stopped at the next hard frame. Nothing was slow. Preparation had stopped entirely: no frame pending, no frame in flight, the drawn lead a fraction under the second it was waiting for, and no work left that could ever close the gap. Dragging the playhead suppresses preparation so a drag does not queue work for frames nobody will see, and that suppression also covered the frames a waiting gate was asking for. It now yields to a gate that is holding. The wait a superseded run walks away from is cancelled rather than left outstanding. An abandoned wait used to read as a gate still holding, which disabled that same drag suppression for the life of the window and also left its promise pending forever. Playback, seeking and teardown all cancel on both the pull and the push transport. A required lead longer than the frame cache could ever hold was unsatisfiable: the requirement was bounded by the window it prepares into but never by the cache it keeps, so a small cache waited for a lead that could not arrive. The cache now bounds it too, and a cache size that is not a whole number of frames is floored rather than reaching past the end of the window. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every preview build since 4bb7bdf has failed, so the running preview is 19 hours behind the branch and serves a bundle none of the fixes are in. The build passes locally at this commit, so this asks the service for a fresh attempt rather than changing anything. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The gate that waits for masks only ever held a reader the renderer pulls frames from. On the engine, which paces itself, it held the very first frame and nothing after, so a machine that could not keep up played on with no masks at all and said nothing about it. Switching every gate on did not change that: the per-frame hold was wired to detections alone. Masks can now stop a running engine too, so a clip whose masks fall behind slows down and keeps its annotations, which is what the reader that pulls has always done. A hold that waits forever would be worse than the fault it fixes, so it gives up after two seconds and lets the picture go on without masks, saying so rather than going quiet. It gives up only on preparation that has finished nothing at all: a slow one re-arms the gate every time it finishes a frame, however far behind it still is. Preparation that only gets a frame out while the picture is stopped counts too, which is what happens when drawing and decoding share a busy processor. Pausing during a hold no longer leaves the reader frozen. A wait that fails, and a pause that lands while a play is still waiting, both give the reader back, so the next drag moves the playhead instead of restarting a video that was paused. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Playback could stop with nothing on screen to explain it. The only sign was a small ring on the play button, which is what the overlay falls back to when no notice could be built, so the viewer was left guessing. Three ways that happened, all now named. Fetching the video was invisible. The notices only ever described masks and detections, so a stop waiting on the clip's own bytes had nothing to report: on a real recording, two seconds after a scrub went entirely unexplained while the file was still arriving. A notice needed a quarter second of unbroken waiting, and the count started again every time a wait cleared for a frame. The reader that pulls holds many short waits rather than one long one, so the count never matured and no notice could appear however long the stutter ran, while the engine's single long hold showed one immediately. The same library, the same gate, opposite behaviour. A wait that clears for a moment and returns is now one wait. A hold on the frame about to be shown, while the frame on screen was ready, produced no notice at all. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Splitting the previous two commits dropped the code that reports a stop waiting on the clip's own bytes, while keeping the tests that cover it. A stop on a source read fell back to the generic buffering notice, which is what left the wait unexplained in the first place. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…f it Every detection carried by a streaming model was measured by painting its mask into a frame-sized buffer and scanning every pixel of it for the edges. At 1920x1080 that is a two-megabyte allocation and two million reads to produce four numbers the run lengths already carry. The runs are walked instead. A frame-filling mask goes from 5.69 ms to 0.03 ms; a heavily fragmented one from 6.57 ms to 2.38 ms. The answer is the same in both, checked against the old path. This ran on the thread that draws, so a model streaming its results took a third to two thirds of every tenth of a second away from the work that keeps the picture moving. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…p ahead How much drawn mask sits in front of the playhead was counted as an unbroken run, stopping at the first frame nobody had finished. A window that was almost entirely drawn reported nothing ahead when one frame near the playhead was outstanding, so the video stopped with a full bar of drawn masks on screen behind it. A model that streams its results puts a fresh undrawn frame into that window several times a second, so the count could never climb back to what the gate asked for. Measured on a clip driven that way: the picture was stopped for 15.6% of the time in five freezes, the longest most of a second. It is now stopped between nothing and 6% in freezes of 80ms, and where enough is drawn ahead it never stops at all. The run now steps over a frame something is already drawing, since that one arrives on its own, and still ends at a gap nobody is working on. A frame the viewer is about to see still stops the picture, as before. The two edges of the wait were also far apart and in the wrong unit. Stopping cost a quarter second of clip and starting again asked for a whole second of it, so every stop had to bank about twenty-three more drawn frames than the one that triggered it. Both are now wall clock and the second is the first plus a margin, scaled by how fast the clip is playing: a stop buys about six frames instead of twenty-three, and asking for a deeper bank no longer buys a longer stop. The two are held apart at every speed and bank, so the pair can never meet and flap. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The play button kept its play triangle while the video was stopped waiting for something, with only a thin line orbiting the square as a hint. It read as a video that could be played rather than one already trying, and it was the only sign at all whenever no notice named the wait. The button now shows a turning ring in place of the triangle while the picture is waiting, and stands still for anyone who has asked for less motion. The gate's two edges are separate controls, since they are now separate numbers, and the ceiling says what it does: it buys no drawn frames, it only shortens a stop. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Users can now seek, scrub, step, pause, and recover without the picture, playhead, or annotations diverging.\n\nOwn presentation completion through actual paint, bound decoder and source work, and keep package and docs contracts aligned. The regression suite covers stale generations, transfer failures, terminal states, variable-frame-rate identity, residency, and both pull and push backends.
People debugging playback could see that the picture paused, but not whether navigation, media reads, detection coverage, or mask preparation caused it. The demo also hid effective wait bounds and collapsed independent detection and mask policies into one pipeline branch. - put mask readiness targets on the timeline and live blockers in Status - record detection and mask waits independently in Pipeline - expose effective wait ceilings and mask stop/resume thresholds - align defaults, docs, fixture labels, and tests with per-frame behavior Co-Authored-By: Codex Opus 4.6 (1M context) <noreply@anthropic.com>
When playback falls behind, stopping now settles the hidden clock on the last painted frame. Resuming or stepping no longer skips catch-up frames users never saw.
Under load, playback or a paused frame could keep the previous frame mask while the current raster was still preparing. This displayed a plausible but incorrect annotation over the picture. A pending mask now leaves its frame bare until its own raster lands. Tests cover adjacent frames while playing and paused, and the stale-state readout remains as an invariant tripwire.
At high playback rates on Android, dropped presentation work could leave the visible pixels paired with another frame’s annotations and make pause or the next step jump. Materialize the selected frame before transfer, coalesce before the display refresh, and advance playback only after the scene renders that frame. Navigation generations prevent stale acknowledgements from reviving an older position.
Keep the diagnostics tap test aligned with the host presentation contract so the full workspace verification can typecheck the demo. The fixture now exposes an acknowledgement spy and proves the tap only forwards it without claiming the frame reached the screen.
Dragging on Android could repeatedly start neighbor decode walks during brief gaps, burning CPU and causing seconds of tail jank. Wait for 100 ms of quiet before speculative scrub prefetch. Exact foreground landing and playback stay unchanged, and diagnostics distinguish a parked timer from active decoding.
Android H.264 playback could pair a decoder buffer that had already been reused with annotations for its earlier timestamp. Snapshot affected decoder output before queueing it, preserve that ownership through presentation, and avoid a redundant transfer copy so playback can drop whole compositions under pressure without showing false ones.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The problem
supervisionalready plays a video. It opens the file through mediabunny, and the renderer asks thesource for a sample at a time the renderer picked.
What it does not have is exact frame identity. The frame index it reports is
round((mediaTime - firstTimestamp) * estimatedFrameRate), and its own type documents that as anestimate. That arithmetic names the wrong frame on:
The failure is quiet. The pixels stay correct and only the annotations move. It survives pausing and it
looks like bad model output.
This pull request adds a browser video engine. The engine, not the renderer, decides which frame is
on screen. It publishes that frame with its own media time and its identity in the container's
timebase, and every annotation layer draws against that one value.
Who this is for
renderer sources and the renderer keeps picking the time. What does reach them is the
maxDevicePixelRatiodefault in What breaks, which caps every presentation surface at 2.The pipeline
Which path a source takes
flowchart TD A["Source opened"] --> B{"Which renderer source<br/>did the caller pass?"} B -->|"the default one"| PULL["Pull path"] B -->|"createWebVideoEngineMediaRendererSource"| C{"Engine chunk loaded?"} C -->|"no"| ERR["Throws, and names supervision/web-video-engine"] C -->|"yes"| D{"H.264 with an avcC record?"} D -->|"yes"| S["One decode session, held across seeks"] D -->|"no"| K["mediabunny sinks, re-positioned per request"] S --> PUSH["Push path"] K --> PUSH PULL --> P1["The renderer picks the time.<br/>It reads the sample timestamp."] PUSH --> P2["The engine picks the time.<br/>It publishes the presented frame."]The library does not read the media and choose. The caller chooses by which source it passes. A
video file passed as a URL opens through mediabunny and takes the pull path.
The pull path is what existing consumers use. Its shape is unchanged: the renderer picks the time
and reads the sample timestamp, as it does on
main. Its behaviour is not. A container that openswith no parsed track now fails as
UnsupportedFormat, wheremainfailed it asNoVideoTrack.errorKindis the field to branch on, so a consumer that reads it sees this. The default changes inWhat breaks reach the pull path as well.
What the engine does
packages/video-engineis a new workspace package. It is private, so npm never publishes it. Itsbuild is staged into
supervision, and consumers import it fromsupervision/web-video-engine. Itdoes not depend on the renderer.
avcCrecord. Other codecs decode through mediabunny's sinks, which re-position on each request.armTrace(windowMs)sizes the ring from the broadcast rate.analysisentry point opens a source and pulls frames without a player.Presented-frame identity
The frame table is built from each packet's own timestamp. That is what makes identity survive a
fractional rate or an unusual timebase.
The position published for a frame comes from the packet that was submitted. It does not come from
the timestamp the platform decoder returns. A decoder that counts from its own origin, or reorders,
therefore cannot slide the annotations off the picture.
A tripwire in the present throws if any layer is handed a media time other than the presented one.
It is armed in every build, production included. It costs about eleven comparisons per presented
frame.
Two rate-derived indexes remain. Both are documented as estimates, not identity:
estimatedFrameIndexin the renderer state, and theNearestFrameIndexdetection selection mode.Temporal detections
Detections are temporal data, independent of decoding. They can be precomputed, appended while
playback runs, or composed from several sources. Overlapping results update the active range without
rebuilding the whole annotation state.
Two producers exist in this repository: precomputed fixtures, and a remote model that pulls frames
from the engine's sample sink.
Prepared annotation rendering
The renderer prepares annotation artifacts ahead of the playhead. It keeps prepared frames on both
sides of it in a bounded cache, so a reversing scrub finds work already done.
On the push path, rendering is event-driven. Pixi's ticker is unused. The scene draws only on a
change: a new presented frame, a detection change, a prepared artifact landing, a hover or selection
change, or a presentation change. A paused scene nobody touches submits no frames. The pull path
still repaints on the ticker.
Masks
A worker builds one byte per pixel holding a detection id. A shader colours those ids from a palette
on the GPU. Where that raster cannot be built, the same worker produces an RGBA composite.
Boxes, labels and vectors draw as before; the hover silhouette is the one thing lost, because
the ids it needs are what the raster carries. The mask layer reports that state rather than
leaving it silent.
The palette holds 80 entries. One entry is the background, so a raster can name 79 detections. It is
keyed on detection index: a mask writes its detection's index plus one. A frame past the ceiling
falls back to the RGBA composite. That path walks each mask's runs rather than the whole plane
once per detection, which on 81 masks over 1920x1080 costs 19 ms for fills and 69 ms with
outlines, where walking the plane cost 151 and 337. The raster path is 1.5 ms. On the 2113-frame horse trail clip, 75
frames used to fall back and no longer do.
A host can declare the box it paints masks into, through
renderPreparation.maskFrame.display. The raster is then built at the size that box can show. Leftunset, masks are built at the detections' own resolution.
The push path runs on WebGPU where images stay on WebGL. Every shader therefore carries a WGSL
variant, and a test requires each shader to carry a program for both backends.
What is new in the public API
The root
supervisionsurface grows from 405 to 425 exported names: 20 added and none removed.The separate
supervision/web-video-engineentrypoint now publishes an explicit list of 48 names.Five internal names formerly leaked by its wildcard barrel were removed before release. These are
different surfaces and should not be combined into one addition count.
WebVideoEngineErrorCodeDecodeUnsupportedtoRateUnsupported. Reached atsupervision/web-video-engine.createWebVideoEngineMediaRendererSource,openWebVideoEngineMediaSource,WebVideoEngineMediaSourcePresentedFrameChannel,PresentedFrameSource,PresentedFramePlayheadand their signal typesPreparedAnnotationWindowSnapshot,PreparedAnnotationWindowFrame,PlaybackGateReachresolveMediaSessionDefaults,ResolvedMediaSessionDefaultsDecodedMediaSourcedeclares both drive modes, and both are public.sampleSinkanswersgetSample(timestamp)for a time the renderer picked.engineis aPresentedFrameChannel: thesource hands each selected frame to the host, which atomically composites matching annotation layers
and acknowledges the frame once displayed. A host with its own decoder can implement the push path
rather than only consume this engine's.
sampleSinkstays required either way, and the engine suppliesa real one over its batch analysis path, which serves thumbnails and one-off frame grabs.
Out of scope
own pull request.
Type of Change
Validation
How to run it
Current inventory and validation scope
npx vitest runnpx vitest runpackages/video-engine/src/*.test.tsnpm run docs:checkMETRICSintools/demo-eval/baseline.mjsFRAME_TIMELINE.MAX_FRAMESMAX_ID_MASK_PALETTE_ENTRIESPLAYBACK_RATEPLAYHEAD_QUANTIZATION_TOLERANCE_SECONDSThe engine's 39 test files cover decoding, timelines, cache behaviour, scrub trajectories, playback
scheduling, frame ownership, worker communication and presentation. They run in Node against fake
browser APIs and a recorded packet table.
Focused regressions cover exact seek landing, frame ownership through cache and teardown, playback
cadence through rate changes and hitches, and both gates on pull and push sources.
Where the numbers come from
Every performance number in this description comes from one Apple M3 Max, 16 cores, 64 GB, in
Chrome, against the 70-second horse trail clip at 30fps through the WebGPU renderer. The clip
carries 2,113 frames and 98,115 detections, which is 46 a frame. These are not cross-device
baselines.
Android testing established correctness, not a performance baseline. On Galaxy S23 Chrome, the
accepted Android H.264 ownership route produced exact raw-pixel/frame identity in 24/24 samples at
1x and 8x, in 14/14 pause and step checks, and in a post-restart 24/24 confirmation while retaining
one playback decoder. Its paired 8x playback slope was 6.588x versus 6.655x for the control. Galaxy
S26 testing helped reproduce the original failure. Other Android browsers, rotated H.264, canvas
presentation, and wide-gamut paths remain unverified. Android HEVC scrub performance remains a
separate unresolved performance lane.
The browser evaluation harness
npm run eval:demodrives the running player and gates 30 metrics against a recorded baseline. Itexits non-zero on a regression. The baseline records the machine, the commit, the clip and whether
the tree was dirty. The repository ships no baseline file and gitignores it: one recorded on a
given processor is only meaningful on that processor, so it stays local.
Eleven families:
sync,latency,layers,cadence,throttle,battery,blanking,drag,playhead,backscrub,focus.The
layersfamily carries a hard budget: zero frames over 34 ms, in every layer combination.Playback rate, presented-frame identity, cache behaviour and cache memory ceilings are covered by
engine unit tests instead. The harness only measures playback at 1x.
Reviewer checklist
npm ci && npm run verifyfrom a clean clone.open the HEVC fixture.
annotations match the video pixels.
setPresentationwhile keeping the same detection array.MediaRendereror interaction-stylecode.
Notes For Reviewers
Answers to review feedback
1. "Polyline rendering broke on a docs page."
You were right, and it is fixed. It was the fixture, not the polyline renderer.
The polylines page embeds the demo with the
basketball_sam3fixture. The page filters toclassName === "basketball"andmetadata.trajectoryTrackId === "basketball-track:0".SAM3 returns a whole-scene answer for that prompt alongside the ball. The fixture's trajectory
step accepted the whole-scene mask as the tracked ball and stamped the track id on it. The page's
filter then kept it faithfully. The precise shape of the defect:
basketball-track:0detections with rect exactly1920x1080af35486rebuilt the trace and refuses any candidate covering 50% or more of the frame.demo/src/fixtures/demo-fixtures.test.tsnow gates it:widestFrameCoveragemust stay under 0.5.That assertion evaluates to 1.0 on the old data, so it is the regression gate for exactly this.
At HEAD the trail is a ball trail: 216 polylines, all on the ball track, footprint 0.013% to 0.221%
of the frame.
The polyline renderer itself is untouched by this branch apart from the new
shadowStrokedefault,which landed after your report.
2. "Make 'buffered by detections' part of the
createMediaSessionAPI."Done.
createMediaSessiontakesplaybackGate, a plain boolean. You either want that playbackmode or you do not, which is the shape you asked for.
It is an umbrella switch over two gates. Either gate can still be set on its own, through
detections.playbackGateandrenderer.renderPreparation.playbackGate.playbackGate: trueNeither default changed. Both were already resolved this way on
main. What was missing was a wayto say yes or no to the whole thing in one place, and a gate that reached a source presenting its
own frames at all: on
mainthe wait lived in the renderer's sample pump, which such a source neverenters.
The docs page you saw playing bare now waits. The masks page embeds the demo. The demo opens a
sample on the Mediabunny media path, which the renderer pulls samples from. A pull source is held at
every frame whenever any gate is on, so
playbackGateReachreportsEveryFrame. The sample passesno session gate, so render preparation is the gate holding it. The detection gate stays off, because
a sample ships its annotations with it.
Both gates reach every frame on both paths:
flowchart TD P["play()"] --> G{"playbackGate"} G -->|"off"| RUN["Frames arrive at once"] G -->|"on, pull path"| PULL["Hold each decoded sample before draw"] G -->|"on, push path"| PUSH["Stop the producer when coverage or artifacts are missing"] PULL --> READY["Present when the wait settles"] PUSH --> READYThe pull path holds each decoded sample between reading and drawing it. The push path stops and
starts the producer, so the detection and render-preparation gates both cover ongoing playback.
Each gate's
maxWaitSecondsbounds its own wait. A pause or a scrub supersedes an active wait, soreadiness landing later does not start a picture the viewer stopped.
What breaks
This takes
supervisionto 0.2.0-next.0, published on thenexttag.lateststays on 0.1.7 until0.2.0 goes out from
main. The pinned public surface goes from 405 exported names onmainto 425:20 added, none removed. The engine's own names are not among them: they reach consumers at the
supervision/web-video-enginesubpath. Every break below is a change to the shape of a type, or to what adefault does. Rows are ordered by how easily each slips past a consumer.
BaseInteractionStyleOptions:shape,cornerRadius,stroke,fill. All four were already@deprecatedonmain.hovered.boxStyleandselected.boxStyle, which reach mask, label, keypoint, polygon and polyline highlights too.requiredForPlaybackis nowrequiredForCoverage.falsereverts to the defaulttrue, so the composed source waits on that entry again.protectedresolvers are gone fromBaseInteractionStyle:resolveBoxInstruction,resolveShape,resolveCornerRadius,resolveStroke,resolveFill.noImplicitOverride. A subclass that overrode one keeps compiling and stops being called.hovered.boxStyleandselected.boxStyle.UnsupportedFormat, wheremainfailed it asNoVideoTrack.NoVideoTrackstops matching that file and falls through to your generic handler.UnsupportedFormatas well. A container whose tracks read and carry no video still fails asNoVideoTrack.maxCachedChunksfor a fixed cap.MediaRenderergains four required members:togglePlayback(),scrub(),getRenderCount(),getPreparedAnnotationWindow().Pick<MediaRenderer, ...>. Anyone who only callscreateMediaRenderer()is untouched.maxDevicePixelRatioleft unset now caps the presentation surface at 2, wheremainrasterized at the display's own ratio.window.devicePixelRatioexplicitly for the old behaviour. The cap is what puts the surface, the mask rasters and the decode on one grid. A mask raster can only be sampled nearest, so a grid it did not share showed as stair-stepped edges.shadowStroke: nulltoBasePolylineStyleto draw the path bare.mainre-derived every 0.5 s.detections.buffer.refreshIntervalSecondsfor the old cadence. Streams are unchanged at 0.25 s.mainbuffered ten ahead and half a second behind. Core's own defaults move the same way, from five and half a second.main's; what changed is how much ground behind the playhead stays buffered. A narrower lookahead was measured and rejected: over 48 runs six seconds ahead lost to ten in 11 of the 12 backward cells and tied in all 12 forward ones, so the window was widened rather than shifted. If you measured memory, the window is 15 seconds against 10.5.detections: { buffer: { bufferAheadSeconds: 10, bufferBehindSeconds: 0.5 } }.VideoSource.idis removed fromUrlVideoSource,BlobVideoSourceandStreamVideoSource.supervision, and reach consumers only atsupervision/web-video-engine, so no released consumer can be holding it.playbackGateis not on this list, and that is deliberate. The render-preparation gate alreadydefaulted to enabled on
main, and the detection gate already defaulted on for appendable sessions.Both are unchanged. What is new is the
playbackGateboolean itself: an off switch, and a way toturn the detection half on for a session that is not appendable. Nothing an existing consumer does
starts behaving differently.
Two more are changes in output rather than removals.
maincompares theplayhead against the frame's media time exactly. On a source whose frame timestamps are not whole
milliseconds, a playhead that rounds down selected the previous detection frame. Sources on
exact-millisecond timestamps are unchanged.
NearestFrameIndexmode the grid step is measured from the buffered frames' own mediatimes.
frameRateis the fallback when the buffered indexes cannot give a step. With no indexedframe at all the mode does not apply, and selection matches by interval instead. A caller whose
rate matched the clip sees no change. A caller who passed a nominal rate the clip does not run at
was previously walked off the grid by the accumulating difference.
MediaRendererStategains five optional fields, so an existing renderer still satisfies the type.drawnMaskFrameTimemaskHeldStaleplaybackGateReachOfforEveryFrame.seekingplaybackStatecannot say so.scrubbingseekinganswers for the transport. The transport settles one message before the landed framereaches the main thread. A host that needs "is the right picture up" must compare the presented
frame's own media time instead. A scrub sets
seekingon every tick, so a host that draws a waitindicator must read
scrubbingfirst.Deprecated
MediaRendererOptions.mutedMediaSessionRendererOptions.mutedDetectionFrameSelectionOptions.frameIndexOriginTimeNeither
mutedoption was ever read, so nothing sounded different before or after. Audio playbackis deferred.
The main-thread cost
Every annotation is drawn on the page's own thread. The engine decodes off it. The picture and the
boxes, masks, labels, polygons, keypoints and focus over it are composited by Pixi on the main
thread, in one synchronous block per presented frame.
Measured on the reference machine and clip, playing from
t=5s, three runs of a 6.0-second windowholding 180 presented frames:
presentVideoFrame, entry to returnThe frame period is 33 ms. Annotations cost 0.03 to 0.07 ms of the block.
What you see when the budget runs out is the picture falling behind. You never see annotations from
the wrong moment: the frame and every layer over it are drawn from one media time, in one block
nothing can interrupt.
A host application shares this thread with its own work. The direction is to make the block smaller
rather than move it to a worker, and the ceiling on what moving it would buy is known: the block is
1.08 to 1.14 ms of the 7.23 to 7.50 ms the thread is busy, so the rest of the thread bounds the win.
docs/internal/video-engine-presentation.mddocuments the mechanism. The figures above come froma CDP profiling run over the demo, which is not committed.
Tradeoffs
supervision, on its own import path.import()keeps it out of the bundle, so an app that only creates a media session emits no engine asset.<video>element are refused here.WebVideoEngineErrorCodenames which limit was hit.demo/fixturesis 306 MB tracked over 107 files. Every clone and every CI run pays it.Known limitations
<video>elementplays the file, but
VideoDecoderreports the testedhvc1andhev1configurations unsupported.The engine refuses the file at load with
DecodeUnsupported, before any frame is presented. Thebuilt-in URL/File source has no software fallback. This is scoped evidence, not a claim about every
HEVC profile or non-Chromium browser: Safari 18.6 reports both tested configurations supported and
plays the same file. The 9-second basketball fixture is H.264 and plays in Firefox.
Safari 18.6 reaches this fallback because it has no WebGPU. Firefox reaches it because its WebGPU
queue rejects a decoded frame. On the tested Safari 18.6/reference-Mac route, staging dominated the
recorded playback wall time; this is a scoped measurement, not a cross-device browser guarantee.
Eligible Android H.264 decoder-session output may instead materialize owned pixels before WebGPU
performs the final upload.
Passing
renderPreparation.maskFrame.displayis what makes the raster follow what the screen canshow. The demo passes one. The presentation numbers above are optimistic for an integration that
has not opted in.
draws another frame's mask: an unprepared mask is cleared, preparation is scheduled, and atomic
presentation keeps every drawn layer on the presented frame's identity.
StreamVideoSourcevariant is declared but no test or demo exercises it. A stream cannotbe re-opened, so the decoder-recovery path degrades instead of rebuilding on one.
Three defects that ship on
maintodayAll three are fixed here, and none of the fixes is on
main. The code each one lives in was therefirst: region effects and their fixture landed on
mainbefore this branch, the prepared-windowtimeline has been there since the shape-primitives work, and the interaction layer has followed a
selected detection across frames since before this branch opened.
The region-effects lens jumped off a player's head, frame after frame. Some lenses floated over
the crowd with nobody under them. A head the model did not see was moved by however far the player's
whole bounding box moved, and that box is set by whichever limb reaches furthest, usually a raised
arm. An invented head now sits between the two real observations on either side of it.
A detection selected while scrubbing vanished for good the first time its annotations were late.
Scrubbing backward is where they are most often late, so the selection usually died within a frame or
two of the first drag, and picking the detection again was the only way back. An absent frame and a
detection that had genuinely left the video both rebased to nothing, and the caller wrote that empty
result over the selection. The follow step now leaves a selection alone while data is missing and
adjudicates on the next frame that has any.
On a looping clip the prepared render window ranked a frame from the previous lap as the furthest
thing prepared. Seventy seconds of footage reported 66.86 seconds of readiness for 211 frames
covering seven. That number is not a readout: it is compared against the lookahead a session asks
for before playback is considered ready, so a wrong value can hold or release the gate for the wrong
reason.
Reading the diff
The pull path is unchanged. The push path, the transport, the frame-present walk and the
prepared-annotation window are new files, reached only through a presented-frame channel. Today only
the video engine drives that channel. The pull path keeps its three ticker callbacks and its draw
order. That is the split worth holding in mind while reading the renderer diff.
Almost every deletion is fixture data. 2,191,256 of 2,197,601 deleted lines sit under
demo/fixtures, because the detection payloads are no longer pretty-printed. Outside thosefixtures the diff is 425 files, 88,464 insertions against 6,345 deletions. That is the code to
review.
The fixture data itself differs from
main. The SAM3 fixtures are generated against the sourcevideos at their native frame rate rather than a resampled proxy. The clearest case is the basketball
sample. On
mainits manifest reads 270 frames at 30fps againstbasketball_sample.normalized.webm.Here it reads 225 frames at 25fps against
basketball_sample.mp4, the clip's own rate. Loading everyfixture in the demo picker covers this better than reading the diff does.
What the fixtures cost a clone.
demo/fixturesis 306 MB tracked over 107 files, in arepository whose
.gitis 772 MB.horse_trail1min-horse-video.mov, 128 MB, the media the demo playsbasketball_sam3raw-sam3.jsonl, 11 MBbasketball_samplebasketball_sample.mp4, 22 MBbasketball_regionshead-detections.json, 9 MBhorse_trail/raw-sam3.jsonlis 44 MB of raw model output kept for provenance beside the 59 MB ofchunked detections derived from it. Nothing loads it at runtime. It is worth deciding deliberately,
since it is what every reviewer and every CI run pays to clone.
Packaging and release
The engine does not publish on its own.
packages/video-engineis a private workspace, and itsbrowser build is staged into
supervisionunderdist/web-video-engine. Consumers reach it byimport path:
The subpaths are
supervision/web-video-engine,supervision/web-video-engine/analysisandsupervision/web-video-engine/worker.createWebVideoEngineMediaRendererSourceandopenWebVideoEngineMediaSourceare exported from the package root as well, and are the same functionin both places.
There is no second install and no optional peer dependency. Installing
supervisioninstalls theengine, because the staged build is inside the tarball. The tarball grows from 654,101 to 1,732,755
bytes, and every consumer pays that download even if it never imports the engine. The bundle cost
stays conditional. The engine is reached by a dynamic
import(), so an app that imports onlycreateMediaSessionemits 1,750,666 bytes and no engine asset, while adding the engine adapteremits 3,278,684 bytes with the engine in its own 1,503,131-byte chunk. Still images and camera input
never load it. Opening a video file does. If that chunk does not load, the video path throws an
error naming
supervision/web-video-engineand saying the engine is a lazily loaded chunk ofsupervision, rather than a bundler stack trace naming a hashed asset.The release workflow publishes one package. It builds the video-engine workspace, stages that build
into
dist/web-video-engine, and deletes the engine'sfile:devDependency from the packedmanifest. It then builds the portable tarball, smoke-tests it in a clean consumer, and publishes
supervision. A releasedsupervisiontherefore names no engine package and no engine version.After the upload the workflow polls
npm view supervision@<dist_tag>up to twelve times atfive-second intervals, until the dist-tag resolves to the version it just published. The workflow
publishes from
main, or from arelease/*branch whendist_tagisnext.No release step needs a person.
supervisionis already on npm, so its trusted publisher isalready attached. The workflow publishes the generated tarball with
npm publishand authenticatesthrough OIDC. It needs no
npm loginand noNPM_TOKEN. The engine is private and is neverpublished, so there is no second name to register.
Two things that will not warn you
A custom
workerFactorymust match the host's version. The mask preparation protocol changed.The artifact kind is
idMaskrather thanpngIdMask, the payload field israsterrather thanpng, and the job carries amaxRasterWidth. None of those types is exported, so nothing warns.Point the factory at
supervision/render-preparation-workerand this cannot happen.Content Security Policy is unaffected. This package already spawns classic blob workers for mask
preparation and for tracking. The engine's worker needs the same directive and no new one.
Documentation status
docs/publicis the published documentation and it is checked against the code.npm run docs:checkruns 31 checks: every path a document names exists, every npm script it runs is declared, every flag
matches the script that reads it, every checksum matches the file beside it, every version matches
the manifest, every symbol it imports is exported, and every copyable integration example
typechecks. All 31 pass.
Eighteen files under
docs/publicchange here:guides/browser-support.mdapi/video-engine.tsguides/media-sessions.md,guides/detections-and-rendering.md,guides/media-preparation.md,recipes/streaming-detections.md,recipes/multiple-detection-sources.mdguides/application-integration.mdguides/public-api.md,concepts.md,annotation-renderers/polylines.mdapi/media-preparation.ts,api/rendering.ts,api/sessions.tsguides/presentation-styles.md,recipes/interactive-picking.md,recipes/progressive-upload-normalization.mdtypedoc-icons.jsWhat this pull request does not have
the artifact worth recording: load a fixture, scrub backwards with masks on, and watch every
annotation stay on its frame.
at
1a4db5erecorded 91 ms p95 backward-scrub settlement, 3.7 ms p95 seek, 53.5 ms p95 step,zero reported drops, and zero reported engine stalls. It had no retained comparison baseline and
predates the accepted scrub-scheduling and Android-ownership changes, so it is not performance
evidence for the current head.
.gitignoreexcludestools/demo-eval/report.jsonandtools/demo-eval/baseline.json, because those numbers only mean anything on the machine thatrecorded them. Reproduce with
npm run eval:demo -- --url 'http://localhost:5173/?mediaPath=engine'.Android testing covers frame identity and playback cadence, not comparative scrub or composition
cost on slower hardware.
on one side only. The other side needs a harness story that lives in the engine repository.