Repository navigation
fix(project): stop a late microphone from freezing a segment's video - #4308
richiemcilroy wants to merge 1 commit into
Conversation
|
Sandchest reviewed Sandchest reviews every change for vulnerabilities an attacker could use · 3 credits · Review details |
| } | ||
| let Some(clip) = clips | ||
| .iter_mut() | ||
| .find(|clip| clip.index as usize == index && clip.offsets_auto_calculated) |
There was a problem hiding this comment.
Manual clips retain audio offsets on the old segment clock
Medium Severity
For an existing segment with video starting at 0 and mic at 5 seconds, the saved automatic audio offsets are mic=0 and system_audio=5. Editing only the camera offset clears the entire clip's offsets_auto_calculated flag in set_clip_offset, so this filter skips both audio offsets. Video now starts at 0 through latest_start_time, but playback and export still use those saved audio offsets directly, playing both tracks five seconds early. Migrate existing audio offsets by the origin delta once while preserving the user's adjustment, rather than excluding the whole clip.
| } else if repair_late_audio_clip_offsets( | ||
| &recording_meta.project_path, | ||
| meta, | ||
| &mut project.clips, |
There was a problem hiding this comment.
Saved captions remain on the old audio clock after repair
Medium Severity
Existing caption timing is retained by synchronize_captions in crates/project/src/caption_timing.rs, which only projects saved timestamps through the edit list. For a segment whose mic starts five seconds late, a word previously transcribed at source time 0 now plays at time 5 after this repair, while its saved caption and word timings remain at 0. Both editor open and export perform the repair without migrating captions, so previously synchronized subtitles become seconds early. Rebase existing caption and word timestamps for affected segments once alongside the offset migration.
| let store = calibration_store.get_or_insert_with(|| load_calibration_store(project_path)); | ||
| let calibration_offset = | ||
| get_calibration_offset(segment.camera_device_id(), segment.mic_device_id(), store); | ||
| let offsets = segment.calculate_audio_offsets_with_calibration(calibration_offset); |
There was a problem hiding this comment.
Repeated repair overwrites persisted device calibration
Medium Severity
This runs on every editor open and export for late-audio segments, including projects already repaired or recorded with the new alignment. It rebuilds offsets from the current shared CalibrationStore rather than preserving the calibration embedded in saved offsets. Moving a repaired project to a machine without sync_calibrations.json, or changing that calibration later, therefore changes its audio synchronization on reopen/export; previously nonempty clip configurations retained their saved values. Make this a versioned, one-time origin migration that preserves the saved calibration contribution instead of recalculating it each time.
| for audio in self.aligned_audio() { | ||
| let start = audio.start_time?; | ||
| if !is_late_audio_start(start, video_start) { | ||
| value = value.max(start); |
There was a problem hiding this comment.
Existing markers keep the old segment clock when latest_start_time() changes. With video starting at 0 s and mic at 4 s, a marker captured at 6 s was saved at 2 s. This PR restores video from 0 s but leaves that marker four seconds before its footage.
place_markers() uses the saved time directly, so repairing audio offsets does not move it. Migrate saved markers when changing an existing segment's start.
Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/project/src/meta.rs
Line: 849-852
Comment:
**Saved markers point elsewhere**
Existing markers keep the old segment clock when `latest_start_time()` changes. With video starting at 0 s and mic at 4 s, a marker captured at 6 s was saved at 2 s. This PR restores video from 0 s but leaves that marker four seconds before its footage.
`place_markers()` uses the saved time directly, so repairing audio offsets does not move it. Migrate saved markers when changing an existing segment's start.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
A user reported a 12 minute Studio recording (macOS, 0.6.3, mic + camera, 13 segments from pause/resume) where several clips showed a frozen screen and camera with no audio, while the real footage was intact on disk. Their microphone dropped at each pause and on resume delivered nothing for seconds to minutes.
Root cause
latest_start_time()(crates/project/src/meta.rs) is the wall time of segment time zero for every consumer: screen and camera decoders (crates/rendering/src/segment_timing.rs), cursor/keyboard mapping, markers, and the audio clip offsets (calculate_audio_offsets*).Fix
LATE_AUDIO_START_SECS(1.0 s) after the later of screen and camera no longer sets the segment start. Its clip offset turns negative, so it plays at the time it was recorded with silence before it. Screen and camera play from their real start.repair_late_audio_clip_offsets(crates/editor/src/editor_instance.rs) recomputes the automatic offsets of segments with late audio only, on editor open (written back) and on export (fill_clip_offsets, in memory). User-set offsets (offsetsAutoCalculated: false) are kept. Normal segments are never touched, so their saved offsets stay bit-identical even if a calibration changed since.calculate_audio_offsets.Noneinstead would makelatest_start_timereturnNoneand drop alignment for the whole segment, and the stop time now does no harm.crates/verify/src/model.rs) states the same rule, written independently of the production mapping.Why 1.0 s
STALL_SILENCE_KEEPALIVE_AFTER).Callers checked
rendering::segment_timing(editor preview, playback, export)calculate_audio_offsets*:initial_clip_configuration, recorder stop,fill_clip_offsets, GPUIadd_clip_configs, transcription, CLI selftest, verify fixturesexport_audioand transcription already handle negative offsets (leading silence)keyboard.rs, markers (studio_recording.rs,recovery.rs), cursorsegment_offsetrecoverystart-times.json,preparing_canonical_probelatest_start_timeEditorInstance's repaired config viaset_projectWaveforms in the GPUI timeline are drawn from raw samples without clip offsets (unchanged, as for any user-set offset).
Before / after
Real Studio recording on macOS (Mac mini): 4 segments with pause/resume, mic delivering nothing in segment 1 and coming back 4 s into segment 2, a full-screen flash at every even wall-clock second and a click written into the mic at the same instants. Each column is one exported frame's brightness, with the exported audio underneath. Same project on disk, exported by
cap exportfrom main and from this branch.A recording made by this branch persists clip mic offsets of -7.98 s and -3.91 s for segments 1 and 2; all other values match main's shape.
Regression check (main vs this PR)
cap exportof 11 real Studio recordings (single and multi segment; every camera / mic / system audio combination; fragmented; a long reference fixture), exported twice per build:project-config.json/recording-meta.jsonPerformance
latest_start_time, 13-segment meta, release, ns/call (3 runs)calculate_audio_offsets_with_calibration, ns/call (3 runs)cap export, 3-clip camera + mic + system audio recording, 4 interleaved rounds, median scap export, long reference recording, 4 interleaved rounds, median shas_late_audiocheck per segment; the calibration store is only read when a segment has late audio.Unit tests added:
crates/project/src/meta.rs(never-started mic, mid-segment recovery with calibration, late system audio with on-time mic, threshold boundary, camera-relative lateness, missing starts),crates/rendering/src/segment_timing.rs(late mic does not offset screen or camera),crates/export/src/lib.rs(saved auto offsets repaired, user offsets kept, normal saved offsets bit-identical). Six of these fail on main.The PR appears safe to merge, with a non-blocking issue affecting saved markers in older late-audio recordings.
Findings
Fix with agent prompt
Summary
Keeps audio that starts over one second late from moving a segment's video start.
Acknowledged intent from richiemcilroy: recompute automatic offsets only for late-audio segments, preserve user-set offsets, and leave normal segments unchanged despite later calibration changes.
Reviews (1) · Last reviewed commit: "fix(project): stop late audio from movin..." · Reviewed by Greptile