Skip to content

feat(desktop-gpui): add captions at the playhead and on the empty caption lane - #4246

Merged
richiemcilroy merged 7 commits into
mainfrom
feat/gpui-add-caption-at-playhead
Nov 16, 2026
Merged

richiemcilroy merged 7 commits into
mainfrom
feat/gpui-add-caption-at-playhead

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Add a caption by hand: the + button in the Transcript tab adds one at the playhead, and clicking the empty caption lane adds one where you click, like the other lanes do.

Screenshots

Linux under Xvfb at 1600x1000, debug build of #4245, #4246 and #4247 combined, on a 52 s recording with a word-timed transcript. The real editor is driven with xdotool and recorded with ffmpeg (probe run). Screen recordings: light, dark. Each add logged a reason="add-caption" timeline edit.

Light Dark
+ in the Transcript header adds Caption 4 at the playhead, 9.84 to 11.84 s, selected for typing Caption added at the playhead, light Caption added at the playhead, dark
A click on the empty caption lane adds Caption 5 centred on the click, 12.35 to 13.35 s, playhead moved there Caption added on the empty lane, light Caption added on the empty lane, dark

Both hand-added captions come out in the saved SRT as cues 4 and 5.

macOS still (the + button (add caption at playhead) in the Transcript header):

Transcript tab, light, macOS

How it works

flowchart LR
  A[+ in Transcript tab] -->|playhead, 2 s| C
  B[click empty caption lane] -->|click time, lane default length| C
  C[find_placement on the caption track] -->|free gap| D[add_caption_segment]
  C -->|no room| N[notice: No room for another caption here]
  D --> E[captions.segments, recording time, no word timings]
  E --> F[synchronize_captions] --> G[caption track, selected for typing]
Loading
  • The 0.6.0 editor had "Add" in its captions panel (TranscriptPage.tsx addCaptionAtPlayhead). The GPUI editor had insert functions for the other segment kinds but not captions, and a click on the empty caption lane did nothing (editor_window.rs, the Hit::Empty arm).
  • add_caption_at (src/editor_window.rs) places the caption with the same find_placement the text, mask and shape lanes use, so it never overlaps an existing caption and takes the nearest free gap. It then selects it, which opens the caption panel so its text can be typed straight away.
    • The + button starts it at the playhead and makes it 2 s long.
    • A lane click centres it on the click, with the lane's default length, and moves the playhead there.
  • add_caption_segment (src/editor_edits.rs) maps the placement back to recording time through the clip under it, like every other caption:
    • it follows clip speed and fullscreen text holds, and stays on its moment as clips are trimmed, cut or moved
    • it ends where its clip ends; past that point the recording jumps, and the caption would cover footage it was never placed over
    • starting inside a fullscreen text hold there is no recording time to attach to, so it is refused with a notice instead of landing after the title
    • captions are switched on and the caption lane is shown. Projects with no transcript and no captions get a fresh source-time captions block.
    • legacy output-time captions (0.6.0 projects that were never opened since) are moved to recording time first, so the new caption cannot be mis-migrated later.

Decision: hand-added captions carry no word timings

  • A caption typed in by hand is text only (words: []), with a caption- id, the prefix the 0.6.0 editor also used for hand-added captions (is_manual_caption, src/transcription.rs).
  • That keeps it out of the transcript: it never shows up as spoken words, and filler removal and pause shortening never treat it as speech.
  • Editing such a caption's text in the caption panel keeps it text only (write_caption_edit_to_source). It used to invent evenly spaced word timings for any wordless caption.
  • Hand-added captions now survive a new transcription (apply_caption_result), including one added while transcription was still running. Generated captions are still replaced as before, and a refinement already keeps wordless captions (keep_caption_identities).
  • Splitting a hand-added caption gives its second half a caption- id too, so both halves survive.
  • Where a kept caption overlaps the new speech captions, it stays on screen there: it was placed by hand, and this is the same state as dragging a caption over another one today. The download in feat(desktop-gpui): download captions as SRT or WebVTT #4247 follows the same rule.

Callers checked

  • fresh_caption_source_id gained a prefix argument; its only other caller, split_caption_segment, still passes cap-split.
  • write_caption_edit_to_source: edit_caption_segment (caption panel) and the caption drag release. Captions with word timings behave exactly as before.
  • apply_caption_result: caption generation (editor_tabs.rs) and tests. Only caption- captions with no words are carried over.
  • The Hit::Empty arm: only the caption lane is new; every other lane keeps its branch.

What could regress

  • A wordless caption edited in the panel no longer gains word timings, so it shows without word-by-word highlighting. Only hand-added and old wordless captions are affected.
  • A regenerated transcript keeps caption- captions with no words. Generated captions never use that prefix, so they cannot be duplicated.
  • Projects saved by 0.6.0 and 0.6.1 to 0.6.3 load unchanged; nothing new is stored.

Tests added

  • src/editor_edits.rs add_caption_tests
    • a_caption_with_no_transcript_turns_captions_on_in_recording_time
    • a_caption_follows_clip_speed_and_stops_at_its_clip_edge
    • a_caption_under_a_fullscreen_hold_is_refused
    • a_caption_inside_a_later_cut_leaves_with_it_and_comes_back_on_restore
    • legacy_output_time_captions_move_to_recording_time_first
  • src/editor_window.rs add_caption_ui_tests (Linux, like the other editor window tests)
    • the_playhead_caption_takes_the_next_free_gap
    • a_full_caption_track_refuses_with_a_notice
  • src/transcription.rs
    • a_new_transcript_keeps_captions_typed_in_by_hand
    • editing_a_caption_typed_in_by_hand_keeps_it_out_of_the_transcript
  • src/editor_window.rs: both_halves_of_a_split_hand_added_caption_survive_a_new_transcript

RetriggerView in GreptileConfidence Score: 5/5

The PR appears safe to merge; no new actionable issues remain.

Summary

Adds captions from the Transcript header and empty caption lane.

  • The Transcript tab and caption lane can add a caption where you want.
  • Hand-written captions stay separate from speech and survive new transcripts.

Reviews (14) · Last reviewed commit: "style(desktop-gpui): format the retyped ..." · Reviewed by Greptile

@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 16, 2026 12:38pm UTC

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 14, 2026 05:05 Inactive
@sandchest-com

sandchest-com Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed c6ea2f9: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 1 credit · Review details

Comment thread apps/desktop-gpui/src/editor_window.rs Resolved
Comment thread apps/desktop-gpui/src/transcription.rs Resolved
Comment thread apps/desktop-gpui/src/editor_edits.rs Resolved
Comment thread apps/desktop-gpui/src/editor_edits.rs Resolved

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

View 2 other findings in Capy.

Open in Capy

Comment thread apps/desktop-gpui/src/editor_window.rs Resolved
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 14, 2026 06:41 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

Comment thread apps/desktop-gpui/src/transcription.rs Resolved
@thedavidweng

Copy link
Copy Markdown

I found another case where a manual caption edit can be lost:

  1. Start a word-timing refinement with an existing manual caption.
  2. Edit that caption's text while refinement is running.
  3. When refinement completes, the old text is restored.

words_signature doesn't include the text or timing of wordless captions, so the edit doesn't change the signature. apply_refined_captions then replaces the current segments with the refinement snapshot, which contains the old manual caption. A regression test against 94dcffd reproduced the replacement ("edited" became "original"). Could refinement preserve the current manual caption edits when it lands?

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 16, 2026 10:05 Inactive
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 16, 2026 10:05 Inactive
@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@richiemcilroy
richiemcilroy merged commit 4fb2fa7 into main Nov 16, 2026
53 checks passed

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
Preview – cap-web — c6ea2f90 Deployed Nov 16, 2026 by vercel[bot]
Preview – cap-status — 6dff73e3 Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants