Repository navigation
feat(ios): Studio Mode and the Studio editor - #4239
richiemcilroy wants to merge 33 commits into
Conversation
|
Sandchest found 1 issue in 1 issue found in This repository is public, so the details stay private: view them in the Sandchest dashboard → Only people who can push to this repository can see them. Reviewed · 24 credits. |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
… keep the waveform on volume changes
|
@greptileai review |
…ing upload can't race the check
Presets apply a whole look in one tap: twelve from Cap, plus your own, saved per account beside the drafts and one of them chosen for new recordings. Each card is the paused frame drawn in that look by a sibling renderer that shares the preview's Core Image context but not its cache. Glow backgrounds are soft blooms of colour in the style of the system wallpapers, drawn from a few numbers on the GPU at any size and cached apart from the frame so padding and corner drags don't redraw them.
Main now lets a recording turn half a turn mid-recording, keyed on recordingRotationAngle, which only the Instant path set. Studio camera recordings now set it when they start and go back to ready through becomeReady(), so a quarter turn waits until the recording ends and is applied then, as it is for Instant recordings.
|
@greptileai review the latest commits (merge of main, Studio rotation, and the two review fixes). |
|
@greptileai review 737a476: both findings from the last review are fixed (screen recordings wait for the extension's finished track, drafts save to the account the editor opened in). |
StudioExporter.cancel() runs on another thread and read the reader, writer and pumps while export() was assigning them. Thread Sanitizer flagged the race on every run and the cancel test crashed about 1 run in 40. They're now only touched under a lock: 0 crashes in 55 runs and Thread Sanitizer clean across the Studio suites.
POST /api/mobile/uploads had no length check, unlike the web, desktop and segmented mobile routes, so a free account could share a Studio edit (or any rendered video) longer than five minutes. The route now applies the same client-reported limit and tolerance as the segmented route, and the editor says so before rendering rather than after.
…ds, so leaving it out can't skip the limit
|
@greptileai please review the latest head (3c991bb). |
| Task { | ||
| if audioChanged, !(await loadMusic()) { return } | ||
| show(look: target.look, timelineFrom: current.timeline, forceRebuild: audioChanged, seekTo: target.timeline.outputTime(forSource: source)) |
There was a problem hiding this comment.
restore() waits for music before applying the saved look. If someone changes the background or preset while that music loads, the later show(look: target.look, ...) replaces their newer choice. Apply the restored look before waiting, and prevent an older restore from applying after another edit.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/Cap/Features/Edit/EditorModel.swift
Line: 703-705
Comment:
**Undo overwrites newer looks**
`restore()` waits for music before applying the saved look. If someone changes the background or preset while that music loads, the later `show(look: target.look, ...)` replaces their newer choice. Apply the restored look before waiting, and prevent an older restore from applying after another edit.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 4d3703c. restore() now puts the saved look back straight away, and the rebuild after the music loads uses the current look, so a background picked while the music loads stays. The editor debug script undoes a music change, picks a new background at once and checks it survives: it failed without the fix and passes with it on the simulator.
| guard let wanted, let url = musicURL(wanted.source) else { | ||
| musicSource = nil | ||
| musicWaveform = [] | ||
| return true |
There was a problem hiding this comment.
Reselected songs lose their waveform
Choosing None clears musicWaveform but leaves loadedMusic unchanged. Selecting that same song again skips the waveform load because its URL still matches, so the waveform stays empty. Clear the cached-song marker with the waveform, or keep and restore the cached peaks.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/Cap/Features/Edit/EditorModel.swift
Line: 983-986
Comment:
**Reselected songs lose their waveform**
Choosing None clears `musicWaveform` but leaves `loadedMusic` unchanged. Selecting that same song again skips the waveform load because its URL still matches, so the waveform stays empty. Clear the cached-song marker with the waveform, or keep and restore the cached peaks.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.There was a problem hiding this comment.
Fixed in 4d3703c. Choosing None now clears loadedMusic along with the waveform, so picking the same song again reads its peaks. The editor debug script picks None then the same song and checks the waveform is drawn: it failed without the fix and passes with it.
…raw a reselected song's waveform Undo put the saved look back only after the music loaded, so a background picked meanwhile was replaced. The look now goes back at once and the rebuild after loading keeps the current look. Choosing None cleared the waveform but not the loaded-song marker, so picking the same song again skipped reading it. None now clears both. The editor debug script checks both and leaves the editor open when a check fails, so the scripted UI test fails.
|
@greptileai please review the latest head (4d3703c). |
… itself cancel() called cancelReading and cancelWriting from another thread while export() was still adding outputs and starting to read, and while the pumps were copying and appending. AVAssetReader and AVAssetWriter aren't safe to use from two threads, and the lock only guarded reading the references. With the CPU saturated the cancel test crashed 89 times in 150, in addOutput and startReading (SIGTRAP, SIGSEGV, SIGABRT); CI hit the same as signal 11. cancel() now only raises the flag and ends the pumps. A pump ends on its own queue, after any pass it is in the middle of, so once the pumps are done nothing else touches the reader or writer, and export() cancels both itself. Same loop: 0 crashes in 150; Thread Sanitizer clean over the Studio suites.
|
@greptileai please review the latest head (2a00f13). |
| lock.lock() | ||
| let pumps = pumps | ||
| lock.unlock() | ||
| pumps?.finishAll() |
There was a problem hiding this comment.
Cancelling Share while writer.finishWriting() is running no longer stops the upload. cancel() now only raises the flag and ends the pumps, but export() clears the pumps and checks that flag before awaiting finishWriting(). A cancel during that wait is never checked again, so the completed file still reaches share() and gets uploaded. Save to Photos has the same problem.
Check cancellation again after finishWriting() and delete the file before returning a result.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/CapKit/Sources/CapKit/StudioExporter.swift
Line: 48
Comment:
**Cancelled videos still upload**
Cancelling Share while `writer.finishWriting()` is running no longer stops the upload. `cancel()` now only raises the flag and ends the pumps, but `export()` clears the pumps and checks that flag before awaiting `finishWriting()`. A cancel during that wait is never checked again, so the completed file still reaches `share()` and gets uploaded. Save to Photos has the same problem.
Check cancellation again after `finishWriting()` and delete the file before returning a result.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Adds Studio Mode to the iOS app, and turns the editor (the "Edit" screen) into the Studio editor: backgrounds, frame shape, padding, corners, shadow, camera layouts, music and voice volume, alongside the existing Photos-style trim, split and delete.
What changes for people
AVCaptureSession.isMultitaskingCameraAccessSupported). On iPhone that needs thevoipbackground mode, which this PR does not add, so on the App Store build it stays hidden. See "Not in this PR".POST /api/mobile/uploads(the single-file route used by Share, Save as New and videos added from Photos), which until now had no length check, unlike the web, desktop and segmented mobile routes. It uses the client-reported length, as those routes do, and the same 5 s tolerance. On the free plan a video upload has to say how long it is, so leaving the length out can't skip the check (the app always sends it for a video it can read).How it works
Recording.
StudioTrackWriter(CapKit) writes one HEVC file per source, with a keyframe every second so the editor can land on any frame fast. Sound goes to its own file beside it: in one file AVAssetWriter keeps picture and sound interleaved, so when sound stops arriving (mic muted from Control Center, a route change, an interruption) it stops taking frames about a second later and the recording silently freezes. Apart, each keeps its own start time, and gaps in the sound are filled with silence so speech after a dropout stays where it was said. The camera records from the app (camera.mp4). The screen records from the broadcast extension (screen.mp4): iOS only lets its own button start a broadcast, so while Studio's screen option is showing the app leaves a short-livedStudioScreenRequestin the app group, and the extension takes it when it starts. Instant Mode disarms it.Sync. Screen and camera come from different processes and clocks. Each writer notes the host clock time of its first frame. Camera frames are converted from the capture session's clock with
CMSyncConvertTime. Every frame keeps its own timestamp in the file, so dropped frames and frame-rate differences can't shift anything. Tracks are lined up by first-frame host time, and each track's sound by its own first-sample host time. A pause in a track longer than 0.5 s (a camera iOS interrupted) is recorded as a gap and left empty in the composition, so the bubble disappears instead of freezing.Editing. The preview is an
AVCompositionof references to the files (a rebuild takes under a millisecond), drawn byStudioCompositorwith Core Image on the GPU. Background, shadow and masks are drawn once per look into a cached layer, so each frame is a handful of texture samples. A look change while paused redraws the last decoded frame instead of asking the decoder again, and Core Image's kernels are compiled while the video loads. An untouched Cap with no camera still plays straight from the decoder, as before. The export goes through the same compositor, so preview and output can't differ.Presets.
StudioPresetis a namedStudioLook. Saved presets and the default live inUsers/<user id>.presets/presets.json, beside the drafts folder rather than in it, because recovery treats every folder among the drafts as a recording and clears empty ones. Thumbnails are drawn by a siblingStudioRendererthat shares the preview's Core Image context and compiled kernels but has its own cache. Drawing twelve looks therefore never evicts the background the live preview is using. They're drawn from the source frames the compositor last composed, or from one small decode when nothing has been composed yet (an untouched Cap plays without the compositor). They're drawn off the main thread, one at a time, and each replaces the previous picture only when it's ready, so the strip never blinks.Glows.
StudioGlowlays four radial blooms over a base colour at an eighth of the frame size, blurs them and scales them up, with Core Image's dither so slow gradients don't band in 8-bit video. The result is drawn once per glow and size and kept apart from the frame cache, so dragging padding or corners doesn't redraw it.Export.
StudioExporterrenders H.264 at about 0.1 bits per pixel per frame with AAC.AVAssetExportSessionhas no bitrate control, and its high-quality presets make files several times larger. Rendering happens with Cap in front, because iOS refuses GPU work in the background; leaving Cap stops the render with a message.Failure handling.
Studio/Users/<user id>), and sign-out clears what the app holds in memory.StudioExporter.cancel()ran on another thread and calledcancelReadingandcancelWritingwhileexport()was still adding outputs and starting to read, and while the pumps were copying and appending.AVAssetReaderandAVAssetWriteraren't safe to use from two threads. With every core busy, the cancel test crashed 89 times in 150 (inaddOutputandstartReading); CI hit the same as signal 11.cancel()now only asks the export to stop: each pump ends on its own queue after any pass it is in the middle of, andexport()cancels the reader and writer itself once the pumps are done. The same loop: 0 crashes in 150, and Thread Sanitizer is clean across all 42 Studio tests.PHPhotoLibrary.performChangesblock was inferred as main-actor code. Photos runs it on its own queue, and Swift's runtime isolation check stopped the app (_dispatch_assert_queue_failunder-[PHPhotoLibrary _performCancellableChanges:...]). The block now lives in anonisolatedhelper,PhotoLibrary.addVideo(at:). The Cap screen's Save to Photos on main has the same shape and gets the same fix; that one is also being fixed on its own in a separate PR, so it isn't held up by this one.Measurements
CapKit on the Mac mini (M4). These tests are new in this PR, and all CapKit tests pass:
StudioSyncTests): screen and camera flash, and the screen's voice beeps, at the same host-clock moments over 30 s. The camera starts 0.73 s early, runs at 29.97 fps against 30, with timestamp jitter, 3 to 4% dropped frames, a 4 s gap and three cuts, and the microphone goes silent for 2.5 s part way. Every flash frame from both tracks lands on exactly the output frame its timestamp says, and every beep on exactly its flash (0.0 ms at all five, first to last, including after the dropout). The camera is gone during its gap and back after it.StudioPresetsTests), 360 px on the long edge from a 1179×2556 screen and a 720p camera: about 2.5 to 3 ms each at p50 with the mini quiet (6 to 9 ms under load averages of 70 to 100). The first costs about 50 ms while Core Image compiles; that's moved into the editor's load. The preview's cached look and size are unchanged after drawing all twelve.Simulator (iPhone 17, iOS 27.2; the simulator decodes and encodes in software, so these are pessimistic):
UI tests (simulator):
StudioEditorUITests: presets (apply, save, pin for new recordings, and the save button refusing a look that's already a preset), every tool, the camera dragged between corners, the split layout, music, a cut and undo, a camera recording in Studio Mode through the real encoder into the editor and drafts, and a scripted measurement session.EditorUITests,testCaptureModes(updated: Studio Mode is selectable now) andtestFreeLimitNotepass.Not in this PR
voipbackground mode. Apple's docs: the camera keeps running while multitasking when the app "links against iOS 18 or later and uses voip", or has the multitasking camera entitlement, which is itself restricted to voip apps. No request is pending in the developer portal. Addingvoipis an App Review risk for an app that isn't a calling app.Fix late cancellation before merging so a cancelled Share cannot upload the video.
Findings
Fix with agent prompt
Summary
This PR adds local Studio recordings, drafts, presets, styling, music, and export to the iOS editor.
Reviews (14) · Last reviewed commit: "fix(mobile-ios): leave cancelling the re..." · Reviewed by Greptile