Adding support to provide 3D stereo information from the SEI via FPA (Frame Packing Arrangement)

[Ste-RH]

[REQUIRED] Use case description

It is great that there is support for reading 3D stereo information (stereoscopic-3d-video-box-st3d), but is it planned to support the SEI FPA (Frame Packing Arrangement) method of extracting this information as well?

Proposed solution

Adding support to extractors to pull FPA information from the SEI data would be simply wonderful :)

Alternatives considered

Not applicable

Links:
https://www.htcdev.com/devcenter/s3d
https://stackoverflow.com/questions/11747278/how-does-android-mediaplayer-decide-3d-display-by-looking-sei-fpa
https://en.wikipedia.org/wiki/Advanced_Video_Coding (Version 12 (Edition 5): (March 9, 2010))

Test Media:
https://rh-testmedia.s3-eu-west-1.amazonaws.com/StereoVideo/Stereo01-1080p-RedBlue.mp4

[kim-vde]

Thanks for suggesting this enhancement. As you are the first one to ask for it, we will probably not look into it in the near future. Feel free to make a pull request if you want the change to be added soon.

[danielcamposramos]

Five years on, you now have a second person asking, so I have filed this on the active tracker with the evidence that was missing in 2020: androidx/media#3419

Short version of what has changed since @kim-vde's reply here:

• GStreamer does the full loop already — gst_h264_parser_parse_frame_packing() decodes payload type 45, h264parse publishes it as GstVideoMultiviewMode on caps, and x264enc derives the encoder parameter back from those caps automatically.
• FFmpeg exposes the same signal as AV_FRAME_DATA_STEREO3D, and VLC parses payload type 45 into multiview_mode for stereo rendering.
• HandBrake merged a patch to write the SEI (PR #8100), and Universal Media Server has an approved one (PR #6330).
• Media3 meanwhile shipped MV-HEVC support for Apple spatial video, so the stereo plumbing on the Format side now exists — C.STEREO_MODE_LEFT_RIGHT and STEREO_MODE_TOP_BOTTOM are already there to be filled in.

The reason it matters more now than in 2020 is that the frame-packed SEI is the only stereoscopic signal that survives MPEG-TS delivery, where container tags do not exist at all, and it is what the existing body of 3D material actually carries. I have it measured on two televisions: an MP4 carrying only this SEI switches the set into 3D automatically, the same file minus 791 bytes of SEI plays flat.

Thanks for leaving this open rather than closing it — it made the second ask possible.