Clip Extractor

Live: https://clip-extractor.brain-bbqs.org

A TypeScript + Vite video player built on sleap-io.js for selecting a clip range or a single frame from a video and then downloading it or uploading it to EMBER. Drop a video (and optionally a SLEAP .slp or ndx-pose .nwb for a pose overlay) into the top file picker, choose Snippet or Frame selector mode, scrub to the range or frame you want, and pick Save or Upload in the bottom card. Playback streams directly from the source with no re-encoding; only the extracted snippet is re-encoded, frame-exactly, on the way out.

Features

• Top-loading source picker with a centered Load local file / Stream from EMBER toggle:
  • Load local file — a drag-and-drop dropzone (mirroring bbqs-uploader's picker); drop a video, click to browse, or load the bundled sample (MediaBunnyVideoBackend.fromBlob, with an mp4box fallback).
  • Stream from EMBER — paste an EMBER/DANDI asset URL and stream it directly (MediaBunnyVideoBackend.fromUrl, with a full-download fallback); also reachable via the ?url= param.
• Optional pose annotations step — an on/off switch at the top right of the load card (default off) reveals a second card for loading a SLEAP .slp (loadSlp({ openVideos: false })) or an ndx-pose .nwb (loadNwb, both the PoseEstimation predictions and PoseTraining annotations flavors); dropping either anywhere enables it automatically. Both end up in the same skeleton/tracks/per-frame model, so everything downstream — the overlay, the annotations sidecar, the rendered overlay copy — is format-blind.
  • The pose file has to describe the loaded video. It records the video it was labeled against, so the card compares the two on load — in either order, since the video can be swapped underneath a loaded pose file. The answer comes in two tiers, because the recorded facts are not equally telling:
    • Refused (red) — a frame count, frame size, or labeled frame range that does not fit is proof of a different recording. The card names each field that disagrees and asks for another file, and nothing is kept: no overlay, no annotations in a delivery. A pose file that cannot be parsed at all is reported the same way.
    • Flagged (amber) — a differently named video, an fps that differs by more than 2%, or more than one video in the file. The pose file still loads and the pose is drawn; the card says what looked off so it can be checked before anything is extracted. The name is only a warning because neither format records a checksum of the video and copies get renamed and re-encoded between machines, so a name is a prompt to look rather than grounds to refuse.
  • An .nwb carries less to check against, so the pose series stands in for what it does not record. An ndx-pose PoseEstimation file names its original video and nothing else about it — no frame count, no frame size, no fps — but its PoseEstimationSeries is sampled once per video frame, so the length of that data array is the video's frame count. The app reads it straight out of the HDF5 (src/lib/nwb.ts, a shallow walk that decodes no values) and uses it twice: more samples than the video has frames is a refusal, fewer is an amber note that says how much of the video the file covers. It also settles where the samples belong — ndx-pose stores a sample's position as a timestamp, and a series slower than about 1 fps rounds into frame indices that are spread out rather than consecutive, so a series exactly as long as the video is re-indexed by sample order instead. A PoseTraining file needs none of this: it names an ImageSeries with the frame count and frame size in it, and is checked like a .slp.
• Snippet / Frame selector toggle centered above the player:
  • Snippet — bound the range with the two In / Out markers on the timeline; the clip streams directly, no re-encoding.
  • Frame — move the playhead to select a single frame.
  • Switching between the two keeps whatever was marked, so a look at a single frame does not cost you a snippet range you had already trimmed.
• One timeline, three markers — In and Out point down onto the track from above; the playhead points up at it from below, in its own colour, since three markers sharing a lane are indistinguishable wherever they meet. Each tapers to a point, so its tip names one frame on the ruler rather than covering a dozen of them. Drag a marker to move it, drag the band between In and Out to slide the whole range at its current length, or press the bare track to send the playhead there. All three are keyboard-operable (←/→ by one frame, Shift by ten, Home/End to the bounds), and [ / ] (or I / O) still mark either end at the playhead. A marker dragged past its partner stops there rather than crossing it.
• Time ruler — hash marks under the timeline, subdivided and labelled from the loaded video's own duration (aiming for roughly six labelled divisions, whether the clip runs seconds or an hour), so a selection reads as a stretch of time and not only as two frame indices. Present in both modes.
• Typed frame indices — the In, Current and Out readouts under the transport are entry fields: type an exact frame and press Enter instead of hunting for it. Out-of-range entries are clamped rather than rejected.
• Frame-accurate player — play/pause, step, scrub, a 0.5× / 1× / 2× speed toggle, and a B-frame decode→display reorder (from getFrameTimes) so playback never jumps backwards. Playback loops the marked range in Snippet mode. Keyboard: Space play/pause, ←/→ step, Shift+scrub extends the range.
• Pose overlay — skeleton edges + nodes drawn per track when a pose file is loaded.
• Light/dark theme with an OS-preference default and a header toggle, styled after bbqs-uploader.
• Sign in with EMBER — the same browser-side OAuth2 (Authorization Code + PKCE) flow as bbqs-uploader, with the signed-in account behind the header avatar.
• Save / Upload toggle centered on the bottom card, which starts on whichever route is actually usable: Upload when you are signed in with at least one incoming dataset, otherwise Save. Whichever side you pick is remembered across refreshes.
  • Description — a required free-text box above both panes: what event the selection showcases, what went wrong in it, or anything else worth passing on with it. Neither button is available until it is filled in, and what it says travels either way, written into the provenance JSON described below.
  • Save — writes the selection to your computer as a single .tar.gz holding exactly what an upload would have sent: the extracted snippet (a frame-exact MP4 trimmed by ffmpeg.wasm) or frame (a PNG), the pose overlay, the original content, and the provenance JSON. It unpacks into one dated folder, date-<YYYYMMDD>_time-<HHMMSS>_type-snippet/ (or _type-frame), with the original content in an original/ subdirectory of it — the same directory an upload writes, minus the archive's sourcedata/raw/ prefix, which means nothing outside a dandiset. Every file is checksummed on the way in, so the bundle's provenance quotes the same dandi:dandi-etag digests the archive would have registered.
  • Upload — sends the same set of files to the EMBER dataset picked below the toggle, into its own directory under sourcedata/raw/clip-extractor/date-<YYYYMMDD>_time-<HHMMSS>_type-snippet/ (or _type-frame), following the same sourcedata/raw/ convention as bbqs-uploader. The extracted selection always goes up first. Transfers use DANDI's own multipart flow: dandi-etag checksum, presigned part PUTs straight to S3, then asset registration. When it finishes, the button is replaced by a status line linking straight into the archive's file browser at that upload's own directory, ready to view or share — so the same selection is not sent twice. Changing the selection, the source or the destination brings the button back.
  • Include the original content is recommended and pre-selected, but optional, and applies to both routes: it covers the source video and any loaded pose file.
• BIDS-style names — every file this app writes is named in entity style, key-value pairs joined by underscores and closed by a type- entity, so one glance says what a file is:
  • name-mice_range-120+300_type-snippet_video.mp4 — the extracted snippet, with its inclusive frame range
  • name-mice_range-120+300_type-snippet_provenance.json — its sidecar, sharing every entity and differing only in the suffix
  • name-mice_index-42_type-frame_image.png and name-mice_index-42_type-frame_provenance.json — a single extracted frame and its sidecar
  • name-mice_range-120+300_type-snippet_overlay.mp4 — the same selection with the pose drawn in
  • name-mice_range-120+300_type-snippet_bundle.tar.gz — a saved bundle, holding all of the above
  • The name- label is reduced to alphanumerics so the entities stay unambiguous to parse, with every word separator (space, _, -, punctuation) marked as +: mice_new becomes name-mice+new. The unabridged original file name is preserved in the provenance record.
  • Original content is never renamed — the source video and any pose file are untouched, so they keep the names they arrived with (minus spaces), and they sit together in an original/ subdirectory of the delivery, apart from the files this app produced.
• Pose overlay version — whenever a pose file is loaded, a delivery also carries a rendered copy of the selection with the skeleton drawn into the pixels, for looking at without a viewer that understands .slp or ndx-pose: a PNG in Frame mode, an H.264 MP4 in Snippet mode (drawn frame by frame, then encoded by the same ffmpeg.wasm that trims the plain snippet, so it does not depend on which codecs the browser can encode). It is not governed by the "include the original content" toggle — it is a view of the selection, not a copy of a source.
• Human-subjects warning and blur tool — when the upload destination is a dataset admins have flagged as holding recordings of people (the phrase CONTAINS HUMAN SUBJECTS in its draft description, the same convention bbqs-uploader reads), the upload pane raises the same red banner that tool raises and holds the Upload button until you confirm both that the files are de-identified and that the work is covered by your institution's IRB approval. The banner is about a destination, so it only appears on the Upload side, and a confirmation lasts the session per dataset.
  • The warning brings out a blur tool under the player: Add blur area places a circle where you click, which you then drag over a face, a badge or anything else identifying, at whatever radius you set (a slider and an exact pixel entry, bounded by the frame). Each circle is a focusable ring on the picture — arrow keys nudge it, Shift by more, +/− resize it, Delete removes it — and Remove area / Clear all sit beside the controls.
  • What you see is what leaves. The blurred pixels are painted into the player itself, into the extracted frame's PNG, into every frame of the rendered pose overlay, and into the snippet by ffmpeg (one gaussian pass blended back inside the circles, in luma coordinates so colour is blurred over the same circle as detail). A blur always forces a re-encode, since a stream copy would hand the source's own frames over untouched.
  • The original stops travelling while anything is blurred: it still holds the pixels you covered, so Include the original content is switched off and disabled with a note saying why. Clear the blur areas and the choice comes back.
  • Blur areas are dropped when a different video is loaded — their coordinates point at pixels in the recording they were drawn on — and the tool stays on screen for as long as any area exists, so one placed under the flag can always be found and removed.
• Provenance sidecar — every delivery also writes a provenance JSON into the same directory: the description typed above the buttons, who uploaded it, the destination dataset (null for a saved bundle, which has no archive behind it), the source video's name and dandi:dandi-etag checksum (recorded even when the original is not included, so the clip can always be traced back to it), the video's fps/dimensions/frame count, the exact frame range, the extracted file's own size and checksum, the literal ffmpeg command that produced it, the rendered overlay's own size, checksum and command, any blur areas burned in (their centres and radii in source pixels, and the strength they were blurred at, so a reader can see which parts of the frame carry no data), and any loaded pose file (its name, checksum and counts, again whether or not it rode along).
• Upload destination — the upload pane lists the signed-in user's Incoming: datasets (the BBQS staging convention), narrowed by the same server-side check that a BBQS/EMBER admin co-owns the dataset, and blocks a destination that is not embargoed.

Usage

1. Pick a source at the top: drop a local video into the picker (or click to browse; Load the sample (mice) works too), or switch to Stream from EMBER and paste an asset URL.
2. Pick Snippet or Frame mode above the player.
3. Pick your selection: in Snippet mode drag the In and Out markers on the timeline (or type frame indices into the In / Out boxes, or press I / O to mark either end at the playhead); in Frame mode move the playhead to the frame, or type its index into the Current box.
4. Optionally flip the Pose annotations (.slp / .nwb) switch on the load card and drop a pose file into the card that appears.
5. In the bottom card, describe the selection — what event it showcases, or what went wrong in it — then pick Save to write it to your computer as a .tar.gz, or Upload to send it to EMBER. Both carry the same files and the same description, and neither is available until the description is filled in.
6. For an upload, Sign in with EMBER in the header first; only Incoming: datasets you own that a BBQS/EMBER admin also owns are offered. Leave Include the original content on unless you know the source is already archived — either way, its name and checksum are recorded in the provenance JSON written alongside the clip.
7. If the destination is flagged as holding human subjects, a red banner appears above the button: cover anything identifying with Add blur area under the player, then confirm the de-identification and IRB notice to enable the upload.

URL params: ?url=<video>&pose=<labels> auto-load on open (?slp= is the older spelling of ?pose= and still works; either takes a .slp or an .nwb).

Development

A standard TypeScript + Vite app (structure and CI mirror bbqs-uploader) — see docs/README.md for the project layout and dev commands.

Notes

• Remote video/SLP URLs must be CORS-accessible.
• Sign-in, token storage, and the admin-check service's trust boundary are documented in SECURITY.md, mirroring bbqs-uploader's.
• Extraction and transfer live in src/lib/: extract.ts (ffmpeg.wasm trimming and frame encoding), blur.ts (the blur areas' geometry, their canvas painting and their ffmpeg filter graph), humanSubjects.ts (the flagged-dataset check, shared verbatim with bbqs-uploader), etag.ts/s3.ts/upload.ts (the DANDI upload pipeline, ported from bbqs-uploader), delivery.ts (the destination path and the default toggle side), bundle.ts (the tar + CompressionStream gzip behind a saved bundle, no dependency), and provenance.ts (the clip-extractor-provenance/v1 sidecar). The clip-relative annotation JSON and payload-packaging helpers are also there, for a future annotations sidecar upload.
• Local content travels from the bytes already in the browser; a range-streamed URL is treated as already archived, so the "include the original content" option is not offered for it (and its checksum cannot be recorded, which the provenance file says explicitly).
• Blur areas are drawn with the browser's own gaussian blur on canvas (ctx.filter), matching ffmpeg's gblur so the player previews the pixels the extraction produces. Where canvas filters are missing (older Safari) it falls back to shrink-and-magnify, so the detail is destroyed either way rather than quietly left in.
• The source video and any pose file are checksummed even when they are not being uploaded, since that checksum is what ties a clip back to its source. For a multi-gigabyte source that hashing takes a while; it is chunked and reported in the status line.

Initial prompt

let's start a /new-vibe in a new PR. use sleap-io.js extensively (look at the other open PR and other vibes that have video players -- though careful, some of them are out of date). make a video player that supports both remote web endpoints + local file system access api reading (this is all handled by sleap-io.js) and is optimized for selecting a clip that we will extract with ffmpeg wasm (see PR 67 and related issue) to transmit to a ember backend (details on the handoff TBD). right now it should just be able to pull up a video, optionally with an SLP file (also sleap-io.js) and pull out the frames (+ annotations, encoded out as json for payload transmission, no SLP dependency), and get it ready for transmission to a POST request to a REST API backend (again, protocol TBD) for upload

Follow-ups locked the name (clip-extractor) and redesigned the interface: top-loading drag-and-drop file picker, a Video/Frame selector toggle (direct streaming, no re-encoding), an optional pose file loader above the player, and a bottom Save/Upload card, with layout and styling based on bbqs-uploader.