Skip to content

feat(desktop-gpui): correct transcript words in place - #4245

Open
richiemcilroy wants to merge 6 commits into
mainfrom
feat/gpui-transcript-word-edit
Open

richiemcilroy wants to merge 6 commits into
mainfrom
feat/gpui-transcript-word-edit

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Double-click a word in the Transcript tab to fix its spelling. The fix lands in the captions, the exported video and every later word cut, and is one undo step.

Screenshots

Linux under Xvfb at 1600x1000, debug build of #4245, #4246 and #4247 combined, on a 52 s recording with a word-timed transcript. The real editor is driven with xdotool and recorded with ffmpeg (probe run). Screen recordings: light, dark. The app logged transcript word edited in both themes.

Light Dark
Double-click "mishurd." opens the field Word field open, light Word field open, dark
Fixed to "misheard.", the caption follows Tab moved to the next word, light Word fixed with Enter, dark
Ctrl+Z brings back "mishurd." Undo, light
Ctrl+Shift+Z restores "misheard." Redo, light

Light sequence: double-click, type, Tab moves the field to "Thanks", Esc closes it, play shows the fixed caption in the preview, then undo and redo.

A word that starts a soft-wrapped line ("mishurd." above) opens its field on its own line (probe run, light, recording):

Field open Fixed with Enter
Field on the wrapped line, light Word fixed on the wrapped line, light

macOS still (the transcript hint now mentions double-click to fix a word):

Transcript tab, light, macOS

How it works

flowchart LR
  A[double-click word] --> B[field over the word]
  B -- Enter / click away --> C[replace_word]
  B -- Tab / Shift-Tab --> C --> D[next or previous word that plays]
  B -- Esc --> E[discard]
  C --> F[captions.segments, source time]
  F --> G[synchronize_captions]
  G --> H[caption track: preview, burn-in export]
  F --> I[TranscriptDoc rebuilt, same word spans]
Loading
  • The 0.6.0 editor let you fix a word in place (TranscriptPage.tsx updateWordText). The GPUI Transcript tab could only cut, restore and search words, so a misheard word stayed wrong in the captions and the export.
  • A transcript word can be built from several caption words (hello + ,, it + 's). word_sources (src/transcript/model.rs) records which caption words each transcript word came from, using the same fold flatten_words already did.
  • replace_word (src/transcript/word_edit.rs:23) swaps those caption words for the typed text, keeping the time the word is spoken at:
    • one spoken token keeps the word's span, so the word stays cut or kept exactly as before and word cuts, refined timings and the export line up with no other change
    • as many spoken tokens as the word had (it's to it is) reuse each one's own timing
    • any other count shares the span evenly; punctuation and emoji tokens share the span of the word they sit beside, so the transcript folds them back in
    • empty text removes the word from the captions only. An edit never cuts video.
  • The field (word_field, src/transcript/view.rs:399) is the existing ui::TextInput, laid over the word at the paragraph's own type size and line height, on the panel surface with an accent ring and the popover shadow. Its position is measured from the paragraph's text layout when it paints, so it stays on the word through scrolling and panel resizes.
  • Enter saves, Esc cancels, clicking away saves, Tab and Shift-Tab save and open the next or previous word that plays (cut words are skipped). The footer shows the keys while the field is open. Undo and redo inside the field edit the text; once saved, the whole correction is one editor undo step.
  • If the transcript is rebuilt while the field is open (a refinement landing), the field moves with its word, found again by its text and start (locate, src/transcript/word_edit.rs), so it stays over the right word and Tab still moves from it.
  • Each word opens with a fresh undo history in the field, so undo never brings back another word's spelling.
  • A transcript rebuild now keeps the scroll position instead of jumping to the top, so saving a correction does not lose your place.

Refined timings

  • A refinement that lands after a correction kept the corrected word on its first-pass timing, because the refined times were matched by text. A filler cut next to that word could then not follow the refined filler fully and left a sliver of it audible.
  • RefinedCaptions (src/transcription/refine.rs) now also matches a word by its boundaries when its text no longer matches: a word that starts where an input word started and ends where an input word ended takes the refined start and end of those words. That covers a respelled word and several caption words corrected into one (it + 's into its). Boundaries two input words share are ignored, and a word whose times were changed by hand (in the sidebar or on the timeline) does not sit on input boundaries, so it stays where it was put.
  • a_refinement_keeps_caption_edits_made_while_it_ran now expects the corrected yeah to take the refined 2.1..2.6: the text edit is still kept, and the word now sits where it is spoken.

Decision: double-click edits

  • Double-click on a word opens it for editing, as in the 0.6.0 editor. Triple-click now selects a sentence (was double-click) and four clicks select a paragraph (was triple-click). The third click lands on the field the second click opened; while nothing has been typed it closes the field and selects the sentence (transcript_select_from_field). Clicking a paragraph's timestamp still selects the paragraph.
  • A double-click on a cut word only restores it: the first click restores the cut, and the second click does not also open the field.
  • The hint under the transcript now mentions double-click in every locale.

Callers checked

  • flatten_words: TranscriptDoc::build, transcript tests. Same output; the source tracking is only collected by word_sources.
  • transcript_mouse_down: the paragraph's mouse handler only. Shift-click and drag selection are unchanged.
  • TranscriptState::refresh: transcript_refresh and the editor's render path. Only the scroll restore is new.
  • RefinedCaptions::retime_of replaces the text-keyed lookup in apply_refined_captions, its only use. RefinementLanding and undo-entry amending go through it unchanged.

What could regress

  • Sentence selection by double-click moved to triple-click (decision above).
  • Refinement now moves a corrected word with an unchanged timing. Words whose timing was changed by hand, and words that share their exact times with another input word, are left alone.
  • Transcript projects saved by 0.6.0 and 0.6.1 to 0.6.3 load unchanged: nothing new is stored, and corrections are plain caption word edits in the existing format.

Tests added

  • src/transcript/tests.rs
    • correcting_a_word_keeps_its_timing_and_reaches_the_rendered_captions
    • a_word_with_attached_punctuation_is_replaced_whole_and_keeps_its_span
    • splitting_a_word_shares_its_span_and_cut_words_stay_cut
    • a_corrected_cut_word_comes_back_with_its_new_text_when_restored
    • clearing_a_word_drops_it_from_the_captions_but_not_the_video
    • punctuation_in_the_next_caption_is_replaced_with_its_word
    • right_to_left_cjk_and_emoji_corrections_keep_their_text
    • an_edited_word_is_found_again_after_its_transcript_moves
    • a_correction_survives_a_refinement_and_the_word_cut_still_follows
    • correcting_a_word_in_a_two_hour_transcript_is_fast (20,000 words)
    • word_edit_ui (Linux, like the other editor window tests): Tab and Shift-Tab, Tab skipping cut words, Esc, a split word landing on the next word, undo and redo, the field following its word through a rebuild, triple-click from the field
  • src/transcription.rs: a_refinement_moves_caption_words_corrected_into_one, a_refinement_leaves_a_word_retimed_by_hand_where_it_was_put

RetriggerView in GreptileConfidence Score: 5/5

The reviewed changes appear safe to merge.

Summary

Adds in-place transcript corrections that reach captions and exports without cutting video. The latest change fixes field placement for wrapped words and adds focused layout tests.

  • Double-clicking a transcript word opens an editor for its caption text.
  • Refinement keeps edited caption words aligned with their original boundaries.

Reviews (12) · Last reviewed commit: "fix(desktop-gpui): open the word field o..." · Reviewed by Greptile

@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 16, 2026 8:54pm UTC

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 14, 2026 05:04 Inactive
@vercel
vercel Bot temporarily deployed to Preview – cap-status November 14, 2026 05:47 Inactive
@sandchest-com

sandchest-com Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 61d3806: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 2 credits · Review details

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 3 potential issues (3 medium).

View 1 other finding in Capy.

Open in Capy

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 14, 2026 06:40 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

hey @greptileai, please re-review the PR

@vercel
vercel Bot temporarily deployed to Preview – cap-status November 16, 2026 10:11 Inactive

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

This branch was successfully deployed

1 active and 1 inactive (outdated) deployments
Preview – cap-web — 61d3806c Deployed Nov 16, 2026 by vercel[bot]
Preview – cap-status — cd5bdfec Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant