Repository navigation
perf(infra): encode long 1080p-and-below Infra videos' adaptive stream in slices - #4238
Conversation
Ports perf/infra-sliced-hls (ee2d6ef..8400435) onto current main. Videos of two minutes or more encode their HLS ladder as time slices on several workers: a 20 s first slice so the video plays in seconds, then slices growing six times, then the tail split in three. A growing playlist is served from the contiguous finished slices, the whole ladder is written once every slice is in, and a slice that fails twice falls back to one whole-file job. Reconciled with main: - Slice jobs carry infra: true, so they get the 1920 Infra poster and the Infra admission rule. A slice reserves max(reserved slots, half the worker) inside canAcceptNewInfraVideoProcess and still counts as a heavy ladder at 1440p and 2160p. canAcceptNewVideoProcess is main's. - The first dispatch after prepare starts the MP4 and the slices and never a whole HLS job beside them. - A slice turned away busy stamps busyAt, and is not reported busy while other slices of the video encode, so the busy-wait deadline reset only applies when nothing of the video is encoding. - A slice job whose start lands after a delete is cancelled, and video and app deletes cancel running slice jobs. - Growing playlists go through main's subtitles, short segment and text/plain media playlist handling and carry the cache tags. - Delivery metering counts slice media files under their rendition with exact seconds, from the whole ladder's playlist or, while it grows, the slice's own playlist.
…parity test A static import reached the media server's Bun-only input policy and brought Bun's global types into the web app's typecheck, which changed fetch for every file and added 44 errors.
The first slice of a sliced Infra video carries the whole video's audio, so it finished only when one AAC encode of the full length did: about 90 s for an hour, which held the first playable slice back by that much. Stereo AAC-LC at 44.1 or 48 kHz and up to 192 kbps is now copied, which reads the file once and takes a few seconds. Every other source is encoded as before. Sliced master playlists allow 192 kbps of audio in BANDWIDTH.
…h together Thirds of a long video are long slices, and while one is still encoding the other workers sit idle. Past 450 s a slice, the rest of the video goes out in four rounds of even slices per worker, at least 300 s each, after the first slice and the ramp. Videos up to about 25 minutes keep the plan they had. On three L4 workers an hour of 4K finished in 206 s instead of 242 s with even slices.
… MP4 A slice that lands beside an Infra MP4 slows the MP4 and pulls the same source at the same time, so the MP4 finished later than it does beside a whole-video ladder and the first slice could wait half a minute for its download. Infra MP4 jobs are now marked, and a slice turns a worker making one away. Whole-video ladders and every other job are admitted as before.
The MP4 now reaches a worker first, so the slices sent after it find that worker taken and land on the others.
The first slice makes the video playable, and sent together with the others it could reach the MP4's worker and then a worker already taking another slice, and wait for the next pass. It now goes out on its own first and asks up to four workers before it waits.
|
Sandchest reviewed Sandchest reviews every change for vulnerabilities an attacker could use · 12 credits · Review details |
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
hey @greptileai, please re-review the PR |
|
hey @greptileai, please re-review the PR |
|
hey @greptileai, please re-review the PR |
|
hey @greptileai, please re-review the PR |
|
hey @greptileai, please re-review the PR |
| !canAcceptNewInfraVideoProcess( | ||
| infraHeavyLadder, | ||
| result.data.range !== undefined, | ||
| )) |
There was a problem hiding this comment.
Slices bypass the worker limit
The route sets infraHeavyLadder only for 1440p and 2160p, but the new slicing path selects ladders at 1080p or below. On a four-slot worker, two slices can therefore run together, and neither blocks a heavy whole-video ladder. This breaks the promised worker limit and lets several ladder encodes compete on one worker. Count Infra slices against that limit regardless of their resolution.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/media-server/src/routes/video.ts
Line: 1066-1069
Comment:
**Slices bypass the worker limit**
The route sets `infraHeavyLadder` only for 1440p and 2160p, but the new slicing path selects ladders at 1080p or below. On a four-slot worker, two slices can therefore run together, and neither blocks a heavy whole-video ladder. This breaks the promised worker limit and lets several ladder encodes compete on one worker. Count Infra slices against that limit regardless of their resolution.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.| part: DeveloperVideoPart, | ||
| ): Promise<DeveloperVideoDispatch> { | ||
| if (part === "hls") { |
There was a problem hiding this comment.
Workflow bypasses MP4-first ordering
dispatchDeveloperVideoPart starts slices without waiting for the MP4 to be accepted. The workflow calls this function for both pending parts concurrently, bypassing the MP4-first ordering in dispatchDeveloperVideoParts. If a slice reaches a worker first, that worker can then accept the MP4 too: admission blocks slices beside an existing MP4, but not the reverse. The jobs then compete on the same worker instead of keeping the MP4 separate. Preserve the MP4-first ordering in the workflow path as well.
Prompt To Fix With AI
This is a comment left during a code review.
Path: apps/web/lib/developer-video-processing.ts
Line: 900-902
Comment:
**Workflow bypasses MP4-first ordering**
`dispatchDeveloperVideoPart` starts slices without waiting for the MP4 to be accepted. The workflow calls this function for both pending parts concurrently, bypassing the MP4-first ordering in `dispatchDeveloperVideoParts`. If a slice reaches a worker first, that worker can then accept the MP4 too: admission blocks slices beside an existing MP4, but not the reverse. The jobs then compete on the same worker instead of keeping the MP4 separate. Preserve the MP4-first ordering in the workflow path as well.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.
Cap Infra videos of 20 minutes or more whose ladder tops out at 1080p or lower now have their adaptive stream encoded in time slices across the farm. Such a video is playable from its first slice, and the whole ladder finishes in about half the single-job time. Every other video keeps the single ladder job exactly as before: shorter ones, 1440p and 4K ladders, and everything while
INFRA_SLICED_HLS=off.sequenceDiagram participant W as web (workflow) participant LB as load balancer participant A as worker A participant B as worker B participant C as worker C W->>LB: POST /video/process (MP4, infra) LB->>A: MP4 runs on its own W->>LB: POST /video/ladder range 0 (audio + poster), up to 4 tries LB-->>A: 503, A is making an Infra MP4 LB->>B: slice 0 W->>LB: slices 1, 2 (2 tries each) LB->>C: slice 1 Note over W: each finished slice webhook sends the next one,<br/>the last one writes the whole ladderHow it works
apps/web/lib/developer-video-slices.ts): a 20 s first slice, a ramp slice, then even slices sized so three workers finish together in four rounds (at least 300 s each, at most 14 slices). Only videos of 20 minutes or more are sliced (SLICED_HLS.minDurationSeconds), and only when the ladder tops out at 1080p or lower (SLICED_LADDER_TIERS). Both limits come from the measurements below.apps/web/lib/developer-video-processing.ts):apps/media-server/src/lib/job-manager.ts):sourceAudioCopiesToHls) instead of encoded, which takes seconds instead of about 90 s for an hour. Anything else is encoded as before.apps/web/lib/developer-video-playback.ts):video.readystill fires once the whole ladder is in.apps/web/lib/infra-delivery-log.ts): slice media files (<rendition>-<slice>-<attempt>.m4s) resolve to exact HLS seconds through the joined playlist, or through the slice's own playlist before that.INFRA_SLICED_HLS=offkeeps every video on the single ladder job. Videos already slicing finish their slices.Benchmarks
Three L4 workers (g6.4xlarge, the farm's image, tuning and admission), with production's dispatch order and retry timing. Each video was run on main's path and on this branch back to back. Only the rows marked sliced are sliced by this PR as merged. The others were measured with slicing forced on, and are why the limits exist.
Cap's own jobs beside Infra work
A regular Cap job (a 1 min 4K iPhone recording) arrives every 8 s, spread over the 3 workers, while one Infra video processes. On an idle farm each regular job takes 5.32 to 5.38 s. Each pair was run back to back; repeats are separated by
/, and the 1 h 4K pairs were repeated in reverse order (sliced first). No regular job failed in any run.Retained slices after a fallback
When viewers were already playing from slices and the video then falls back to one whole-video job, the slices' media and their own playlists are kept. This way:
supersededSlicedHlsin playback;The kept files are left out of the video's
storedByteson purpose. They exist only because our encode fell back, so the customer is not charged storage for them.bytes.hlsis the whole job's ladder, as for any other single-job video.Fix the slice worker limit and workflow dispatch ordering before merging.
Findings
Fix with agent prompt
Summary
Adds sliced adaptive-stream encoding for Infra videos of at least 20 minutes whose ladder tops out at 1080p.
Reviews (9) · Last reviewed commit: "Merge branch 'main' into perf/infra-slic..." · Reviewed by Greptile