Repository navigation
Round the remuxed audio tfdt up and absorb the residual in the audio timestampOffset - #8080
stevan-borus wants to merge 3 commits into
Conversation
robwalch
added
the
Needs Triage
| // Round the base media decode time up so that the first sample never starts before the fragment. | ||
| // WebKit drops a sample whose start precedes the append window, even by a fraction of a tick. | ||
| const baseMediaDecodeTime = firstPTS! / scaleFactor; | ||
| const roundedBaseMediaDecodeTime = Math.ceil(baseMediaDecodeTime); |
There was a problem hiding this comment.
Couldn't this add up as drift over time?
If this issue is only dealing with rounding errors at the append window of 0, then shouldn't it be possible to simply adjust timestampOffset for audio based on initPTS in base-stream-controller?
There was a problem hiding this comment.
Note
claude-fable-5-1 responding on behalf of Stevan
No, it can't accumulate. The ceil only changes the value written into tfdt; nextAudioTs, _initPTS and the track's startPTS/endPTS still come from the raw PTS, and each fragment's tfdt is computed from its own first PTS. On the controller side the offset is recomputed from initPTS on every append and set as an absolute value with that fragment's residual subtracted, so nothing carries over. I've added a test that remuxes 600 contiguous fragments (about an hour of 44.1 kHz AAC) and asserts tfdt_N - tfdt_0 == N * frames * 1024 exactly, plus one that seeks 50 times and checks each residual stays under a tick while varying per fragment. The branch is also rebased on current master.
You're right that the only place this bites is the append window at 0. I measured the mid-stream cases on WebKit (both MediaSource and ManagedMediaSource): a fragment whose first audio frame starts up to a tick early against an already-buffered edge keeps both frames and trims the new one's head, with no stall and no hole. Seeks into empty regions and discontinuities behave the same on master and on this branch. So mid-stream the change is a few microseconds of placement and nothing visible.
The reason I kept the residual per fragment rather than deriving it from initPTS in the controller is that it's in audio ticks: it depends on the samplerate and the fragment's first PTS, both of which the remuxer has exactly and the controller doesn't. Reporting the one number seemed a smaller wire than plumbing the samplerate out, and it covers the first fragment after a seek or discontinuity without special-casing. If you'd rather it lived in base-stream-controller based on initPTS, I'm happy to rework it that way.
While measuring this I found a separate issue: with B-frames and audio starting after the video DTS, the silence fill can place its first frame before initPTS (the nextPts < 0 guard compares an absolute PTS), and WebKit drops it regardless of rounding. That's in #8110.
There was a problem hiding this comment.
Thanks. #8110 was easy to verify - merged and marked it for v1.7.4.
remuxAudio passed a fractional baseMediaDecodeTime to MP4.moof and traf floored it, so a first PTS off the audio sample grid put the first AAC frame up to one tick before the fragment start. WebKit drops that frame as preceding the append window, the audio buffered range starts one frame late, and playback from 0 stalls until the gap controller seeks. Rounding up leaves a sub-tick residual that makes buffered.start(0) a microsecond past 0, which WebKit also refuses to start from. The remuxer now reports that residual as timestampOffsetAdjustment on the audio track and bufferFragmentData subtracts it from the audio append offset.
The rounding is recomputed from each fragment's own first PTS and the residual is reported per fragment, so neither can accumulate. The tests pin that: over 600 contiguous fragments the tfdt advances by exactly the samples remuxed, and after 50 far seeks every residual stays under one audio tick while varying from fragment to fragment.
2ef8e02 to
9caf78d
Compare
…-round-up # Conflicts: # tests/unit/remux/mp4-remuxer.ts
This PR will...
Round the remuxed audio fragment's
tfdtbase media decode time up instead of down, and absorb the sub-tick rounding residual in the audio SourceBuffertimestampOffsetfor that fragment.Why is this Pull Request needed?
remuxAudiopassesfirstPTS / scaleFactortoMP4.moof, andMP4.traffloors it intotfdt. When the first audio PTS is not on the audio sample grid, flooring places the first AAC frame slightly before the fragment start. For example, a first PTS of 132006 (90 kHz) at 44.1 kHz gives 64682.94 ticks, floored to 64682, which is about 21 µs early. Safari/WebKit (MSE and ManagedMediaSource) drops that frame as preceding the append window. The audio buffered range then starts one AAC frame (about 23 ms) late, and playback from 0 stalls until the gap controller seeks over the hole. MPEG-TS sources at 29.97/59.94/23.976 fps or with variable frame rate hit this routinely.Rounding up alone is not enough: the first frame then lands about 1 µs late,
buffered.start(0)is 0.000001, and WebKit will not start playback whilecurrentTime < buffered.start(0). The remuxer now reports the residual(ceil(raw) - raw) / samplerateon the audioRemuxedTrackastimestampOffsetAdjustment.bufferFragmentDatasubtracts it from the audio append offset, so the buffered range starts at the intended time. The video path andMP4.trafare unchanged. Raw MPEG audio has no moof and reports 0.Are there any points in the code the reviewer needs to double check?
updateTimestampOffsetonly applies changes of at least 1e-6 s. A residual of 1.36e-6 s, as in the case above, is applied, and a test covers it. A residual under 1 µs is skipped when the SourceBuffer offset already equals the base offset, which for a first append only happens when initPTS gives an offset of about 0. Lowering that tolerance would close the gap; this PR leaves it unchanged.Math.ceilturns floating-point noise just above an integer into a full extra tick. The reported residual compensates, so the presentation time stays correct.RemuxedTrackgains an optional field; the api-extractor report is updated.Resolves issues:
Checklist