MJPEG over RTSP (RTP/JPEG, RFC 2435) — discussion #119

Goal: play a camera whose RTSP stream is MJPEG instead of H.264/H.265 (SDP m=video … RTP/AVP 26, a=rtpmap:26 JPEG/90000). To be discussed before implementation.

Is it possible? Yes

• Transport side. RTP/JPEG (RFC 2435) is a simple payload: every packet carries an 8-byte JPEG header (fragment offset, type, Q, width/8, height/8), optional restart-marker and quantization-table headers (Q ≥ 128), then a piece of the scan data. A frame ends with the RTP marker bit. The receiver rebuilds a complete JPEG (SOI, DQT from the tables in the stream or from Q via the standard tables, SOF0 from type/size, DHT standard tables, DRI if present, SOS, the data, EOI). No SPS/PPS, every frame is a key frame.
• Decoding side. The AG-216 has a hardware MJPEG decoder: OMX.allwinner.video.decoder.mjpeg (video/jpeg) in its codec list. Elsewhere a fallback is BitmapFactory / ImageDecoder (software, fine for small sizes and low fps, CPU-heavy for 1080p25).

Fit with what exists

• Single view and multi-view: RtspSession already passes RTP packets per channel; a JpegAssembler next to NalAssembler would produce whole frames; MediaCodecManager would be created for video/jpeg (or a Bitmap path draws to the Surface).
• Multi-view planner (Multi-view: adaptive decoding — full rate or key frames per cell, learnt device capacity #85): MJPEG has no "key frames only" mode — every frame is independent, so the cheap mode would be decode every N-th frame (frame skipping is trivial). Load per frame is higher than H.264 P-frames.
• Bandwidth: MJPEG 1080p is roughly 5–15× the bitrate of H.264 at similar quality; on Wi‑Fi this matters, on the lab Ethernet not.
• Audio is unaffected.
• udp:// slots (feat: receive plain RTP from Majestic outgoing over UDP, unicast and multicast (#68) #72) could take PT 26 the same way.

Questions to decide

1. Which cameras / why? Which devices only (or preferably) give MJPEG over RTSP — old IP cameras, USB-to-RTSP bridges, encoders without H.264, a substream? Is there one we can put on the bench? (OpenIPC/Majestic serves MJPEG over HTTP at /mjpeg, multipart/x-mixed-replace, checked on .133/.126; whether its RTSP can carry MJPEG needs checking.)
2. HTTP MJPEG too? Should a slot also accept http://<cam>/mjpeg (multipart)? It is even simpler (whole JPEGs, no RTP) and works with every OpenIPC camera today.
3. Decoder: MediaCodec video/jpeg first (hardware where present), software fallback — or software only to start?
4. Multi-view budget: how should MJPEG cells count in the Multi-view: adaptive decoding — full rate or key frames per cell, learnt device capacity #85 planner (frame skipping instead of key frames)?
5. Scope of the first step: single view only, or multi-view from the start?

Rough plan once agreed

1. RFC 2435 depacketizer + JPEG header reconstruction, unit tests with captured RTP (Q < 128 and Q ≥ 128 with in-band tables, restart markers).
2. SDP: accept PT 26 / JPEG/90000.
3. Decode via MediaCodec video/jpeg, fallback to Bitmap; device check on AG-216.
4. Multi-view + planner integration.
5. (Optional) HTTP MJPEG slots.