Skip to content

feat(ios): Studio Mode and the Studio editor - #4239

Open
richiemcilroy wants to merge 33 commits into
mainfrom
feat/mobile-studio-mode
Open

richiemcilroy wants to merge 33 commits into
mainfrom
feat/mobile-studio-mode

Conversation

@richiemcilroy

@richiemcilroy richiemcilroy commented Nov 14, 2026 •

Copy link
Copy Markdown
Member

Adds Studio Mode to the iOS app, and turns the editor (the "Edit" screen) into the Studio editor: backgrounds, frame shape, padding, corners, shadow, camera layouts, music and voice volume, alongside the existing Photos-style trim, split and delete.

What changes for people

  • The recorder's mode switch now has a working Studio Mode. It records Camera or Screen and keeps the recording on the phone as separate tracks instead of uploading as it goes. When recording stops, the editor opens on it.
  • Screen + Camera records the screen and the floating camera as two tracks at once. It only appears where iOS keeps the camera running behind other apps (AVCaptureSession.isMultitaskingCameraAccessSupported). On iPhone that needs the voip background mode, which this PR does not add, so on the App Store build it stays hidden. See "Not in this PR".
  • Studio recordings stay as drafts in the recorder until shared. Edits save as they're made.
  • Presets apply a whole look in one tap. Each card shows the paused frame drawn in that look, so you see your own recording before you choose. Cap includes twelve, and Save keeps the current look as your own. Holding a card offers Use for New Recordings, so every Studio recording starts in it, plus Rename, Update to Current Look and Delete for your own presets. Presets belong to the account, like drafts. A photo background is copied in beside its preset, so it outlives the recording it came from.
  • Glow backgrounds: ten soft colour fields in the style of the system wallpapers (Sky, Mint, Peach, Lavender, Sunset, Dusk, Aurora, Ocean, Midnight, Graphite). They're drawn from a few numbers each on the GPU, so they're sharp at any size and add nothing to the download.
  • Share renders the edit on the phone and uploads it as a Cap through the existing background upload. Save to Photos renders to the photo library instead.
  • Free plan limit. Studio recordings stay on the phone, so recording isn't limited. Sharing one as a Cap is: on the free plan an edit over 5 minutes is refused before it renders, with a message to trim it, save it to Photos or upgrade. The server now enforces the same limit on POST /api/mobile/uploads (the single-file route used by Share, Save as New and videos added from Photos), which until now had no length check, unlike the web, desktop and segmented mobile routes. It uses the client-reported length, as those routes do, and the same 5 s tolerance. On the free plan a video upload has to say how long it is, so leaving the length out can't skip the check (the app always sends it for a video it can read).
  • The same editor opens on an existing Cap. Cuts alone still save on Cap's servers, exactly as before. A background, frame or music renders on the phone and uploads as a new Cap (the button reads "Save as New"). The original is never touched, because the server has no way to replace a Cap's video with an uploaded file.
Wallpaper 1:1 frame Camera bubble Dragged to a corner Split Music
Presets Sky Dusk Vertical Square Widescreen
Save a preset Use for new recordings Glow: Sky Glow: Mint Saving to Photos In Photos
Camera recorded in Studio Mode Kept as a draft

How it works

Recording. StudioTrackWriter (CapKit) writes one HEVC file per source, with a keyframe every second so the editor can land on any frame fast. Sound goes to its own file beside it: in one file AVAssetWriter keeps picture and sound interleaved, so when sound stops arriving (mic muted from Control Center, a route change, an interruption) it stops taking frames about a second later and the recording silently freezes. Apart, each keeps its own start time, and gaps in the sound are filled with silence so speech after a dropout stays where it was said. The camera records from the app (camera.mp4). The screen records from the broadcast extension (screen.mp4): iOS only lets its own button start a broadcast, so while Studio's screen option is showing the app leaves a short-lived StudioScreenRequest in the app group, and the extension takes it when it starts. Instant Mode disarms it.

Sync. Screen and camera come from different processes and clocks. Each writer notes the host clock time of its first frame. Camera frames are converted from the capture session's clock with CMSyncConvertTime. Every frame keeps its own timestamp in the file, so dropped frames and frame-rate differences can't shift anything. Tracks are lined up by first-frame host time, and each track's sound by its own first-sample host time. A pause in a track longer than 0.5 s (a camera iOS interrupted) is recorded as a gap and left empty in the composition, so the bubble disappears instead of freezing.

Editing. The preview is an AVComposition of references to the files (a rebuild takes under a millisecond), drawn by StudioCompositor with Core Image on the GPU. Background, shadow and masks are drawn once per look into a cached layer, so each frame is a handful of texture samples. A look change while paused redraws the last decoded frame instead of asking the decoder again, and Core Image's kernels are compiled while the video loads. An untouched Cap with no camera still plays straight from the decoder, as before. The export goes through the same compositor, so preview and output can't differ.

Presets. StudioPreset is a named StudioLook. Saved presets and the default live in Users/<user id>.presets/presets.json, beside the drafts folder rather than in it, because recovery treats every folder among the drafts as a recording and clears empty ones. Thumbnails are drawn by a sibling StudioRenderer that shares the preview's Core Image context and compiled kernels but has its own cache. Drawing twelve looks therefore never evicts the background the live preview is using. They're drawn from the source frames the compositor last composed, or from one small decode when nothing has been composed yet (an untouched Cap plays without the compositor). They're drawn off the main thread, one at a time, and each replaces the previous picture only when it's ready, so the strip never blinks.

Glows. StudioGlow lays four radial blooms over a base colour at an eighth of the frame size, blurs them and scales them up, with Core Image's dither so slow gradients don't band in 8-bit video. The result is drawn once per glow and size and kept apart from the frame cache, so dragging padding or corners doesn't redraw it.

Export. StudioExporter renders H.264 at about 0.1 bits per pixel per frame with AAC. AVAssetExportSession has no bitrate control, and its high-quality presets make files several times larger. Rendering happens with Cap in front, because iOS refuses GPU work in the background; leaving Cap stops the render with a message.

Failure handling.

  • Writers fragment every second, so an app or extension killed mid-recording keeps all but its last second. Start times are written on the first sample, and the folder comes back as a draft the next time drafts are shown.
  • Drafts are kept per account (Studio/Users/<user id>), and sign-out clears what the app holds in memory.
  • A render waits for a wallpaper or song picked a moment earlier, and cancelling it (or leaving Cap) always ends it.
  • Recording refuses to start with under 1 GB free. A writer that fails mid-recording (full disk) stops the recording and keeps what was written.
  • Too-short recordings are discarded.
  • Cancelling a render could crash the app. StudioExporter.cancel() ran on another thread and called cancelReading and cancelWriting while export() was still adding outputs and starting to read, and while the pumps were copying and appending. AVAssetReader and AVAssetWriter aren't safe to use from two threads. With every core busy, the cancel test crashed 89 times in 150 (in addOutput and startReading); CI hit the same as signal 11. cancel() now only asks the export to stop: each pump ends on its own queue after any pass it is in the middle of, and export() cancels the reader and writer itself once the pumps are done. The same loop: 0 crashes in 150, and Thread Sanitizer is clean across all 42 Studio tests.
  • Save to Photos crashed the app. The app builds with main-actor default isolation, so the PHPhotoLibrary.performChanges block was inferred as main-actor code. Photos runs it on its own queue, and Swift's runtime isolation check stopped the app (_dispatch_assert_queue_fail under -[PHPhotoLibrary _performCancellableChanges:...]). The block now lives in a nonisolated helper, PhotoLibrary.addVideo(at:). The Cap screen's Save to Photos on main has the same shape and gets the same fix; that one is also being fixed on its own in a separate PR, so it isn't held up by this one.
  • Upload limits stay server-side, as for any upload.

Measurements

CapKit on the Mac mini (M4). These tests are new in this PR, and all CapKit tests pass:

  • Sync (StudioSyncTests): screen and camera flash, and the screen's voice beeps, at the same host-clock moments over 30 s. The camera starts 0.73 s early, runs at 29.97 fps against 30, with timestamp jitter, 3 to 4% dropped frames, a 4 s gap and three cuts, and the microphone goes silent for 2.5 s part way. Every flash frame from both tracks lands on exactly the output frame its timestamp says, and every beep on exactly its flash (0.0 ms at all five, first to last, including after the dropout). The camera is gone during its gap and back after it.
  • Sound stops, picture doesn't: sound ending at 1.5 s of a 10 s recording leaves a 10.0 s picture.
  • Cancel: cancelling a render at 0, 50 and 300 ms returns within 0.03 to 0.23 s and leaves no file.
  • Composite one frame, 884×1920 with gradient, padding, rounded screen, shadow and camera bubble: 1.2 ms p50 / 1.5 ms p95 with the mini idle. Asserted under 33 ms p95 off CI; on CI's shared virtual GPUs (18 to 136 ms) it's only reported.
  • Full-size export, 884×1920 from a 1178×2556 screen and a 720p camera: about 8x realtime, adding 133 to 163 MB while it runs. The mini was under heavy load from other jobs, so treat speeds as a floor.
  • Rebuild with 600 clips: 2 to 3 ms.
  • Preset thumbnails (StudioPresetsTests), 360 px on the long edge from a 1179×2556 screen and a 720p camera: about 2.5 to 3 ms each at p50 with the mini quiet (6 to 9 ms under load averages of 70 to 100). The first costs about 50 ms while Core Image compiles; that's moved into the editor's load. The preview's cached look and size are unchanged after drawing all twelve.
  • Glow, 1080×1920: 2 to 5 ms to draw, once per glow and size. A padding drag over a glow costs 5.4 ms per step at p50 under load, the same work as over a gradient.
  • Unfinished track: a writer dropped without finishing after 7 s still plays 6 s.

Simulator (iPhone 17, iOS 27.2; the simulator decodes and encodes in software, so these are pessimistic):

Measure Result
Frame compose 5.7 ms p50, 8.6 ms p95
Scrub frame 13 ms p50, 30 ms p95
Trim frame 14 ms p50, 31 ms p95
Paused look redraw 19 ms p50
Open to first frame 0.8 s (was 1.5 s before pre-warming kernels)
Memory while editing 100 to 125 MB

UI tests (simulator):

  • StudioEditorUITests: presets (apply, save, pin for new recordings, and the save button refusing a look that's already a preset), every tool, the camera dragged between corners, the split layout, music, a cut and undo, a camera recording in Studio Mode through the real encoder into the editor and drafts, and a scripted measurement session.
  • Existing EditorUITests, testCaptureModes (updated: Studio Mode is selectable now) and testFreeLimitNote pass.

Not in this PR

  • Device proof. Neither phone was reachable, and the simulator has no camera or broadcast. Studio screen recording through the extension, Screen + Camera, and on-device speed and memory are not yet measured on a phone.
  • Screen + Camera on iPhone needs the voip background mode. Apple's docs: the camera keeps running while multitasking when the app "links against iOS 18 or later and uses voip", or has the multitasking camera entitlement, which is itself restricted to voip apps. No request is pending in the developer portal. Adding voip is an App Review risk for an app that isn't a calling app.
  • Landscape screen recordings keep the portrait frame ReplayKit delivers, as Instant Mode does.

RetriggerView in GreptileConfidence Score: 4/5

Fix late cancellation before merging so a cancelled Share cannot upload the video.

Findings

  1. P1 Cancelled videos still upload ▶
Fix with agent prompt
### Issue 1
apps/mobile-ios/CapKit/Sources/CapKit/StudioExporter.swift:48
Cancelling Share while `writer.finishWriting()` is running no longer stops the upload. `cancel()` now only raises the flag and ends the pumps, but `export()` clears the pumps and checks that flag before awaiting `finishWriting()`. A cancel during that wait is never checked again, so the completed file still reaches `share()` and gets uploaded. Save to Photos has the same problem.

Check cancellation again after `finishWriting()` and delete the file before returning a result.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Summary

This PR adds local Studio recordings, drafts, presets, styling, music, and export to the iOS editor.

  • Studio recordings stay on the phone as drafts for editing.
  • The editor adds visual styles, camera layouts, and music alongside its trim tools.
  • Whole-video mobile uploads now follow the free-plan duration limit.

Reviews (14) · Last reviewed commit: "fix(mobile-ios): leave cancelling the re..." · Reviewed by Greptile

@sandchest-com

sandchest-com Bot commented Nov 14, 2026 •

Copy link
Copy Markdown

Sandchest

Sandchest found 1 issue in 2a00f13 (1 high).

1 issue found in 984ea6e is still open (1 high).

This repository is public, so the details stay private: view them in the Sandchest dashboard →

Only people who can push to this repository can see them. Reviewed · 24 credits.

@vercel

vercel Bot commented Nov 14, 2026 •

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
cap-status Ignored Ignored Preview Nov 16, 2026 9:23pm UTC

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 5 potential issues (5 medium).

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found 1 potential issue (1 medium).

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

Open in Capy

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 1 other finding in Capy.

Open in Capy

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 3 other findings in Capy.

Open in Capy

Presets apply a whole look in one tap: twelve from Cap, plus your own,
saved per account beside the drafts and one of them chosen for new
recordings. Each card is the paused frame drawn in that look by a sibling
renderer that shares the preview's Core Image context but not its cache.

Glow backgrounds are soft blooms of colour in the style of the system
wallpapers, drawn from a few numbers on the GPU at any size and cached
apart from the frame so padding and corner drags don't redraw them.
Main now lets a recording turn half a turn mid-recording, keyed on
recordingRotationAngle, which only the Instant path set. Studio camera
recordings now set it when they start and go back to ready through
becomeReady(), so a quarter turn waits until the recording ends and is
applied then, as it is for Instant recordings.
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review the latest commits (merge of main, Studio rotation, and the two review fixes).

@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai review 737a476: both findings from the last review are fixed (screen recordings wait for the extension's finished track, drafts save to the account the editor opened in).

StudioExporter.cancel() runs on another thread and read the reader,
writer and pumps while export() was assigning them. Thread Sanitizer
flagged the race on every run and the cancel test crashed about 1 run in
40. They're now only touched under a lock: 0 crashes in 55 runs and
Thread Sanitizer clean across the Studio suites.
POST /api/mobile/uploads had no length check, unlike the web, desktop
and segmented mobile routes, so a free account could share a Studio edit
(or any rendered video) longer than five minutes. The route now applies
the same client-reported limit and tolerance as the segmented route, and
the editor says so before rendering rather than after.
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai please review the latest head (3c991bb).

Comment on lines +703 to +705
Task {
if audioChanged, !(await loadMusic()) { return }
show(look: target.look, timelineFrom: current.timeline, forceRebuild: audioChanged, seekTo: target.timeline.outputTime(forSource: source))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Undo overwrites newer looks

restore() waits for music before applying the saved look. If someone changes the background or preset while that music loads, the later show(look: target.look, ...) replaces their newer choice. Apply the restored look before waiting, and prevent an older restore from applying after another edit.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/Cap/Features/Edit/EditorModel.swift
Line: 703-705

Comment:
**Undo overwrites newer looks**

`restore()` waits for music before applying the saved look. If someone changes the background or preset while that music loads, the later `show(look: target.look, ...)` replaces their newer choice. Apply the restored look before waiting, and prevent an older restore from applying after another edit.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d3703c. restore() now puts the saved look back straight away, and the rebuild after the music loads uses the current look, so a background picked while the music loads stays. The editor debug script undoes a music change, picks a new background at once and checks it survives: it failed without the fix and passes with it on the simulator.

Comment on lines +983 to +986
guard let wanted, let url = musicURL(wanted.source) else {
musicSource = nil
musicWaveform = []
return true

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Reselected songs lose their waveform

Choosing None clears musicWaveform but leaves loadedMusic unchanged. Selecting that same song again skips the waveform load because its URL still matches, so the waveform stays empty. Clear the cached-song marker with the waveform, or keep and restore the cached peaks.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/Cap/Features/Edit/EditorModel.swift
Line: 983-986

Comment:
**Reselected songs lose their waveform**

Choosing None clears `musicWaveform` but leaves `loadedMusic` unchanged. Selecting that same song again skips the waveform load because its URL still matches, so the waveform stays empty. Clear the cached-song marker with the waveform, or keep and restore the cached peaks.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 4d3703c. Choosing None now clears loadedMusic along with the waveform, so picking the same song again reads its peaks. The editor debug script picks None then the same song and checks the waveform is drawn: it failed without the fix and passes with it.

…raw a reselected song's waveform

Undo put the saved look back only after the music loaded, so a background picked meanwhile was replaced. The look now goes back at once and the rebuild after loading keeps the current look.

Choosing None cleared the waveform but not the loaded-song marker, so picking the same song again skipped reading it. None now clears both.

The editor debug script checks both and leaves the editor open when a check fails, so the scripted UI test fails.
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai please review the latest head (4d3703c).

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 7 other findings in Capy.

Open in Capy

… itself

cancel() called cancelReading and cancelWriting from another thread while export() was still adding outputs and starting to read, and while the pumps were copying and appending. AVAssetReader and AVAssetWriter aren't safe to use from two threads, and the lock only guarded reading the references. With the CPU saturated the cancel test crashed 89 times in 150, in addOutput and startReading (SIGTRAP, SIGSEGV, SIGABRT); CI hit the same as signal 11.

cancel() now only raises the flag and ends the pumps. A pump ends on its own queue, after any pass it is in the middle of, so once the pumps are done nothing else touches the reader or writer, and export() cancels both itself. Same loop: 0 crashes in 150; Thread Sanitizer clean over the Studio suites.
@richiemcilroy

Copy link
Copy Markdown
Member Author

@greptileai please review the latest head (2a00f13).

@capy-ai capy-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Capy found no issues.

View 4 other findings in Capy.

Open in Capy

lock.lock()
let pumps = pumps
lock.unlock()
pumps?.finishAll()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Cancelled videos still upload

Cancelling Share while writer.finishWriting() is running no longer stops the upload. cancel() now only raises the flag and ends the pumps, but export() clears the pumps and checks that flag before awaiting finishWriting(). A cancel during that wait is never checked again, so the completed file still reaches share() and gets uploaded. Save to Photos has the same problem.

Check cancellation again after finishWriting() and delete the file before returning a result.

Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/mobile-ios/CapKit/Sources/CapKit/StudioExporter.swift
Line: 48

Comment:
**Cancelled videos still upload**

Cancelling Share while `writer.finishWriting()` is running no longer stops the upload. `cancel()` now only raises the flag and ends the pumps, but `export()` clears the pumps and checks that flag before awaiting `finishWriting()`. A cancel during that wait is never checked again, so the completed file still reaches `share()` and gets uploaded. Save to Photos has the same problem.

Check cancellation again after `finishWriting()` and delete the file before returning a result.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

This branch was successfully deployed

1 active (outdated) and 1 inactive (outdated) deployments
Preview – cap-web — 3c991bbf Deployed Nov 16, 2026 by vercel[bot]
Preview – cap-status — 3c991bbf Deployed Nov 16, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant