Skip to content

fix(project): stop a late microphone from freezing a segment's video - #4308

Open
richiemcilroy wants to merge 1 commit into
mainfrom
fix/late-audio-segment-alignment
Open

richiemcilroy wants to merge 1 commit into
mainfrom
fix/late-audio-segment-alignment

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 17, 2026 •

Copy link
Copy Markdown
Member

A user reported a 12 minute Studio recording (macOS, 0.6.3, mic + camera, 13 segments from pause/resume) where several clips showed a frozen screen and camera with no audio, while the real footage was intact on disk. Their microphone dropped at each pause and on resume delivered nothing for seconds to minutes.

Root cause

flowchart LR
  A[mic delivers no frame in a segment] --> B["stop_with_outcome: first_timestamp dropped,<br/>defaulting to now (core.rs)"]
  B --> C["SegmentStartTimes: mic.start_time = segment end<br/>(or the moment it came back)"]
  C --> D["MultipleSegment::latest_start_time =<br/>max(display, camera, mic, system_audio)"]
  D --> E["segment_video_timing aligns screen + camera to it:<br/>video skips forward, holds the last frame"]
Loading
  • latest_start_time() (crates/project/src/meta.rs) is the wall time of segment time zero for every consumer: screen and camera decoders (crates/rendering/src/segment_timing.rs), cursor/keyboard mapping, markers, and the audio clip offsets (calculate_audio_offsets*).
  • A mic that never delivered a frame records the stop time as its start, so the whole clip plays the last screen frame. A mic that comes back after 4 s skips the first 4 s of video and freezes the last 4 s.

Fix

  • An audio track that starts more than LATE_AUDIO_START_SECS (1.0 s) after the later of screen and camera no longer sets the segment start. Its clip offset turns negative, so it plays at the time it was recorded with silence before it. Screen and camera play from their real start.
pub fn latest_start_time(&self) -> Option<f64> {
    let video_start = self.video_start_time()?;          // max(display, camera)
    let mut value = video_start;
    for audio in self.aligned_audio() {                    // mic, usable system audio
        let start = audio.start_time?;
        if !is_late_audio_start(start, video_start) {      // start - video > 1.0 s
            value = value.max(start);
        }
    }
    Some(value)
}
  • Projects already on disk: repair_late_audio_clip_offsets (crates/editor/src/editor_instance.rs) recomputes the automatic offsets of segments with late audio only, on editor open (written back) and on export (fill_clip_offsets, in memory). User-set offsets (offsetsAutoCalculated: false) are kept. Normal segments are never touched, so their saved offsets stay bit-identical even if a calibration changed since.
  • New recordings get the right offsets at stop, because the recorder already uses calculate_audio_offsets.
  • The recorder still persists the stop time for a mic that never delivered. Persisting None instead would make latest_start_time return None and drop alignment for the whole segment, and the stop time now does no harm.
  • The verify oracle (crates/verify/src/model.rs) states the same rule, written independently of the production mapping.

Why 1.0 s

  • Across 195 segments of 185 real Studio recordings, the largest audio lag behind the video was 0.155 s (a legacy first-packet system audio start); the mic maximum was 0.057 s.
  • It matches the recorder's own outage threshold: a mic silent for 1 s is treated as stalled (STALL_SILENCE_KEEPALIVE_AFTER).

Callers checked

Caller Effect
rendering::segment_timing (editor preview, playback, export) screen + camera start at their own start
calculate_audio_offsets*: initial_clip_configuration, recorder stop, fill_clip_offsets, GPUI add_clip_configs, transcription, CLI selftest, verify fixtures late audio gets a negative offset; renderer, export_audio and transcription already handle negative offsets (leading silence)
keyboard.rs, markers (studio_recording.rs, recovery.rs), cursor segment_offset map to the video that now plays
recovery start-times.json, preparing_canonical_probe same rule via latest_start_time
GPUI editor takes EditorInstance's repaired config via set_project

Waveforms in the GPUI timeline are drawn from raw samples without clip offsets (unchanged, as for any user-set offset).

Before / after

Real Studio recording on macOS (Mac mini): 4 segments with pause/resume, mic delivering nothing in segment 1 and coming back 4 s into segment 2, a full-screen flash at every even wall-clock second and a click written into the mic at the same instants. Each column is one exported frame's brightness, with the exported audio underneath. Same project on disk, exported by cap export from main and from this branch.

Before (main) before
After after
Clip main: distinct frames / flashes this PR: distinct frames / flashes Flash vs click
0 (normal) 64 / 5 64 / 5 identical
1 (mic never delivered) 6 / 0 (frozen) 72 / 4 no audio, as recorded
2 (mic back after 4 s) 44 / 2, then frozen 75 / 4 +30 ms on both (same as main's surviving events)
3 (normal) 70 / 4 71 / 4 identical

A recording made by this branch persists clip mic offsets of -7.98 s and -3.91 s for segments 1 and 2; all other values match main's shape.

Regression check (main vs this PR)

  • Alignment and automatic offsets (exact bits, with and without calibration) for 195 segments of 185 real recordings: identical except the 3 segments with a mic 5 s, 20 s and 27 s late.
  • cap export of 11 real Studio recordings (single and multi segment; every camera / mic / system audio combination; fragmented; a long reference fixture), exported twice per build:
Check Result
Decoded audio md5 identical on all 11
Video framemd5, main vs PR 0 differing frames on every normal recording (main vs main control also 0)
Late-mic recording differs only in clips 1-2 plus the rest of the GOP they share with clip 3 (SSIM 0.998, PSNR 51 dB, identical from the next keyframe)
Exported project-config.json / recording-meta.json identical between builds
Two recordings that refuse to export (retained track failure) same error on both builds

Performance

Measurement main this PR
latest_start_time, 13-segment meta, release, ns/call (3 runs) 2.8 / 3.1 / 4.2 6.8 / 6.3 / 4.2
calculate_audio_offsets_with_calibration, ns/call (3 runs) 6.2 / 5.6 / 5.8 7.6 / 8.1 / 6.6
cap export, 3-clip camera + mic + system audio recording, 4 interleaved rounds, median s 24.8 21.7
cap export, long reference recording, 4 interleaved rounds, median s 58.4 60.9
  • The alignment runs a handful of times per rendered frame, so the extra few nanoseconds are far below a 16 ms frame.
  • Editor open adds one has_late_audio check per segment; the calibration store is only read when a segment has late audio.
  • Export times are debug builds on a shared Mac mini; rounds of the same build varied by up to 10x from other load, and paired rounds show no systematic difference.

Unit tests added: crates/project/src/meta.rs (never-started mic, mid-segment recovery with calibration, late system audio with on-time mic, threshold boundary, camera-relative lateness, missing starts), crates/rendering/src/segment_timing.rs (late mic does not offset screen or camera), crates/export/src/lib.rs (saved auto offsets repaired, user offsets kept, normal saved offsets bit-identical). Six of these fail on main.

RetriggerView in GreptileConfidence Score: 4/5

The PR appears safe to merge, with a non-blocking issue affecting saved markers in older late-audio recordings.

Findings

  1. P2 Saved markers point elsewhere ▶
Fix with agent prompt
### Issue 1
crates/project/src/meta.rs:849-852
Existing markers keep the old segment clock when `latest_start_time()` changes. With video starting at 0 s and mic at 4 s, a marker captured at 6 s was saved at 2 s. This PR restores video from 0 s but leaves that marker four seconds before its footage.

`place_markers()` uses the saved time directly, so repairing audio offsets does not move it. Migrate saved markers when changing an existing segment's start.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

Keeps audio that starts over one second late from moving a segment's video start.

  • Late audio no longer makes a segment’s screen and camera jump forward.
  • The editor and exports repair saved automatic offsets for late-audio segments.

Acknowledged intent from richiemcilroy: recompute automatic offsets only for late-audio segments, preserve user-set offsets, and leave normal segments unchanged despite later calibration changes.

Reviews (1) · Last reviewed commit: "fix(project): stop late audio from movin..." · Reviewed by Greptile

@sandchest-com

sandchest-com Bot commented Nov 17, 2026

Copy link
Copy Markdown

Sandchest

Sandchest reviewed 410a27c: nothing exploitable found in this change.

Sandchest reviews every change for vulnerabilities an attacker could use · 3 credits · Review details

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 3 potential issues (3 medium).

Open in Capy

}
let Some(clip) = clips
.iter_mut()
.find(|clip| clip.index as usize == index && clip.offsets_auto_calculated)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Manual clips retain audio offsets on the old segment clock

Medium Severity

For an existing segment with video starting at 0 and mic at 5 seconds, the saved automatic audio offsets are mic=0 and system_audio=5. Editing only the camera offset clears the entire clip's offsets_auto_calculated flag in set_clip_offset, so this filter skips both audio offsets. Video now starts at 0 through latest_start_time, but playback and export still use those saved audio offsets directly, playing both tracks five seconds early. Migrate existing audio offsets by the origin delta once while preserving the user's adjustment, rather than excluding the whole clip.

Open in Capy

} else if repair_late_audio_clip_offsets(
&recording_meta.project_path,
meta,
&mut project.clips,

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Saved captions remain on the old audio clock after repair

Medium Severity

Existing caption timing is retained by synchronize_captions in crates/project/src/caption_timing.rs, which only projects saved timestamps through the edit list. For a segment whose mic starts five seconds late, a word previously transcribed at source time 0 now plays at time 5 after this repair, while its saved caption and word timings remain at 0. Both editor open and export perform the repair without migrating captions, so previously synchronized subtitles become seconds early. Rebase existing caption and word timestamps for affected segments once alongside the offset migration.

Open in Capy

Comment on lines +2019 to +2022
let store = calibration_store.get_or_insert_with(|| load_calibration_store(project_path));
let calibration_offset =
get_calibration_offset(segment.camera_device_id(), segment.mic_device_id(), store);
let offsets = segment.calculate_audio_offsets_with_calibration(calibration_offset);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Repeated repair overwrites persisted device calibration

Medium Severity

This runs on every editor open and export for late-audio segments, including projects already repaired or recorded with the new alignment. It rebuilds offsets from the current shared CalibrationStore rather than preserving the calibration embedded in saved offsets. Moving a repaired project to a machine without sync_calibrations.json, or changing that calibration later, therefore changes its audio synchronization on reopen/export; previously nonempty clip configurations retained their saved values. Make this a versioned, one-time origin migration that preserves the saved calibration contribution instead of recalculating it each time.

Open in Capy

Comment on lines +849 to +852
for audio in self.aligned_audio() {
let start = audio.start_time?;
if !is_late_audio_start(start, video_start) {
value = value.max(start);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Saved markers point elsewhere

Existing markers keep the old segment clock when latest_start_time() changes. With video starting at 0 s and mic at 4 s, a marker captured at 6 s was saved at 2 s. This PR restores video from 0 s but leaves that marker four seconds before its footage.

place_markers() uses the saved time directly, so repairing audio offsets does not move it. Migrate saved markers when changing an existing segment's start.

Prompt To Fix With AI
This is a comment left during a code review.
Path: crates/project/src/meta.rs
Line: 849-852

Comment:
**Saved markers point elsewhere**

Existing markers keep the old segment clock when `latest_start_time()` changes. With video starting at 0 s and mic at 4 s, a marker captured at 6 s was saved at 2 s. This PR restores video from 0 s but leaves that marker four seconds before its footage.

`place_markers()` uses the saved time directly, so repairing audio offsets does not move it. Migrate saved markers when changing an existing segment's start.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant