Replace realtime Scribe streaming with post-recording batch transcription - #14
Merged
Conversation
…tion The realtime WebSocket session died during long silences (idle/session limits), killing the transcript for the rest of the take. Transcription is now a batch pass over the finalized recording, ordered so footage durability never depends on the network: - Stop flow saves the recovery checkpoint before any transcription call; failure or timeout falls back to full-duration sections with a visible warning instead of blocking or losing the take. - New main-process transcription-service extracts audio from the finalized file via ffmpeg stream copy and calls batch Scribe (scribe_v2, word timestamps); the API key never reaches the renderer. - Word timestamps map into recording time via per-file recorder start offsets instead of the old wall-clock estimate. - Record is blocked while the previous take is still post-processing so overlapping take flows cannot race on the shared recovery checkpoint. - Removed the WebSocket/AudioWorklet/PCM pipeline, live segment editing UI, scribe token IPC, and scribe-status module (net -215 lines). Tests: segment builder + timeout helper (15), transcription service temp-file cleanup and failure paths (10), IPC handler validation (3); media-cleanup and register-handlers suites updated. pnpm run check passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Recording UI overhaul: the transcript panel and "keep silences" checkbox are gone from the recording screen. Stopping a recording now puts the take on the timeline immediately as one full-length section (no network in the stop flow), and a new "Transcribe & Cut" button in the timeline toolbar runs the batch transcription and silence cutting only when asked. - Transcribe & Cut targets the selected section's take (fallback: latest take on the timeline), replaces its sections with speech-cut ones as a single undo step, and syncs the persisted take snapshot in take-local coordinates. - Failure, timeout, no-speech, or a timeline edit made while transcription was in flight leaves the timeline unchanged and reports a visible status next to the button. - System-audio "keep" regions captured during recording are stashed per take for the session and merged into the cut so audible screen sound is not trimmed. - Recorder failures now surface on a compact notice line in the recording sidebar (replacing the transcript-panel status). - Retired the settings.autoCutSilences project setting; normalization drops it from existing project files. Tests: transcribe-cut pure helpers (15) covering source routing, target-take resolution, and section replacement; project-domain tests updated for the retired setting. pnpm run check passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
Real-world testing of Transcribe & Cut found playback continuing on the old uncut sections after the timeline visually updated. Root cause: the draw loop's boundary check looked up the active section by OBJECT IDENTITY (sections.indexOf(activePlaybackSection)); when the section list is wholesale-replaced (Transcribe & Cut, undo/redo restoring copies) a stale reference returns -1 and sections[-1 + 1] silently redirected playback to the first section, replaying old content. - Extract the per-frame advance decision into a pure module (renderer/features/timeline/playback-advance.ts): id-based lookup, geometry always read from the current sections array, and an explicit 'stale' result when the active id is gone so the draw loop re-resolves from timeline time instead of guessing an index. - Refresh activePlaybackSection to the current array's object every frame, and drop it before the post-cut re-seek in transcribeAndCutTake. - Fix the "Cannot close a closed AudioContext" unhandled rejection: cleanupAllMedia closed a context that stopAudioMeter had already closed, and close() rejects asynchronously past a sync try/catch. Both paths now guard on state === 'closed' and swallow the close promise rejection. Tests: playback-advance unit suite (9) including the stale-reference regression and undo-copy geometry case; media-cleanup tests for the already-closed guard and rejected close() promise. pnpm run check passes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Bugbot is not enabled for your account, so this pull request was not reviewed. Enable Bugbot in the Cursor dashboard to get automatic reviews on future PRs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Two commits that together rearchitect transcription end-to-end:
1. Realtime → batch transcription (main process)
transcription-service: extracts audio from the finalized take via ffmpeg stream copy (source file untouched; remux also fixes MediaRecorder's incomplete WebM headers), uploads to ElevenLabs batch Scribe (scribe_v2, word-level timestamps). The API key stays in the main process — the renderer no longer receives a Scribe token.batch-transcriptrenderer helper: groups word timestamps into speech segments using the same 1.5s silence threshold the old VAD used.audio-processor.ts,scribe-status.ts, scribe token IPC.2. On-demand "Transcribe & Cut" UI (recording screen overhaul)
.pending-recording.json) land on disk, then the take enters the timeline immediately as one full-length section. No network call anywhere in the stop flow.settings.autoCutSilences; normalization drops it from existing project files.Why
The realtime Scribe session was killed server-side during long pauses, silently ending transcription mid-take. Batch transcription of the whole file removes that failure mode; making it an explicit editor action keeps the stop flow instant, keeps footage durability fully offline, and makes cutting a deliberate, reversible choice.
Behavior notes for review
audioStartOffsetMs/cameraStartOffsetMs) instead of the old wall-clock estimate.Tests
tests/unit/transcribe-cut.test.ts— source routing, target-take resolution, section replacement (15)tests/unit/batch-transcript.test.ts— segment grouping, offsets, timeout bounds (15)tests/unit/transcription-service.test.ts— ffmpeg args, temp-file cleanup on all failure paths (10)tests/unit/register-handlers.test.ts—transcription:transcribehandler (3)pnpm run checkpasses (lint, typecheck, 454 unit/integration tests, Electron smoke, packaging smoke). Docs updated: feature-inventory, runbook, target-architecture,.env.example, AGENTS.md.Residual risk: the real-network Transcribe & Cut path is exercised only manually — suggested spot-check: record a short take with a deliberate 30s pause, confirm it lands on the timeline instantly, then click Transcribe & Cut and verify the cut boundaries and that Cmd+Z restores the full take.
🤖 Generated with Claude Code