Skip to content

Multi-view: one shared decoder for key frames of several cameras (lift the per-codec instance limit) #88

Closed as not planned
Closed as not planned
Multi-view: one shared decoder for key frames of several cameras (lift the per-codec instance limit)#88
Labels
deferredPostponed, not planned for nowenhancementNew feature or request

Description

@compot-dev
compot-dev
opened on Oct 12, 2026

Follow-up to #85. Key-frame mode reduces the load but still needs one decoder instance per cell, and the AG-216 allows only 4 × H.265. A fifth H.265 camera cannot be shown at all (Decoder limit). Idea: key-frame cells over the limit share one decoder that decodes their IDR frames in turn (reconfigure or flush per camera) and renders each into its own surface. Needs research: the cost of reconfiguring, and whether a surface can be switched with setOutputSurface (API 23+).

Activity

  1. compot-dev
    added
    enhancementNew feature or request
    todayTasks for today
    on Oct 12, 2026

  2. compot-dev commented on Oct 13, 2026

    @compot-dev
    compot-dev
    on Oct 13, 2026
    CollaboratorAuthor

    Research notes (2026-10-13) — how one decoder could serve the key frames of several cells:

    Problem: the AG-216 allows 4 × H.265 decoder instances (H.264 more); with 8 H.265 cameras, cells 5–8 show Decoder limit: 4 × H.265 even on key frames only, where each instance idles > 90 % of the time (one frame per GOP).

    Approach A — one MediaCodec per codec, output surface switched per frame (preferred to try first):

    • A SharedKeyDecoder per mime owns one MediaCodec; cells in key-frame mode hand it complete access units (VPS/SPS/PPS + IDR) instead of decoding themselves.
    • For each queued key frame: setOutputSurface(cell's surface) (API 23+), queue the parameter sets + IDR, wait for the output buffer, releaseOutputBuffer(render=true). Different resolutions per camera rely on the codec's adaptive playback (FEATURE_AdaptivePlayback); otherwise flush and reconfigure on a size change.
    • Cost: 8 cameras × ~1 key frame/s = ~8 decodes/s — far below one instance's budget.
    • Unknowns to probe on the AG-216: does OMX.allwinner.video.decoder.hevc support setOutputSurface and adaptive playback; latency per switched frame; does a new IDR from another camera decode without a flush.

    Approach B — decode to buffers and draw (fallback): one decoder per codec outputs to an ImageReader; frames are converted and drawn into each cell's surface with lockCanvas/GL. Works everywhere but costs CPU/GPU per frame and adds a copy.

    Integration: the #85 planner keeps full-rate cells on their own decoders; key-frame cells (and slides, #108) beyond the instance limit join the shared decoder instead of showing Decoder limit. MJPEG cells (#132) are independent (JPEG decoder).

    Next step: a probe build that logs isFeatureSupported(FEATURE_AdaptivePlayback) for each decoder and tries setOutputSurface between two cells' surfaces on the AG-216.

  3. compot-dev
    added
    in progressBeing worked on right now (open PR)
    on Oct 13, 2026

  4. compot-dev commented on Oct 13, 2026

    @compot-dev
    compot-dev
    on Oct 13, 2026
    CollaboratorAuthor

    Probe result (AG-216, PR #137 /stats.json → decoders):

    decoder type hardware adaptive max instances 1080p fps
    OMX.allwinner.video.decoder.avc H.264 ✓ ✓ 16 60
    OMX.allwinner.video.decoder.hevc H.265 ✓ ✓ 4 60
    c2.android.hevc.decoder H.265 ✗ (software) ✓ 32 62
    c2.android.avc.decoder H.264 ✗ (software) ✓ 32 241
    • The hardware H.265 decoder supports adaptive playback → approach A (one shared instance, setOutputSurface per key frame) is possible in principle.
    • A simpler first step: cells over the hardware instance limit that are on key frames only (≈ 1 frame/s) could use the software decoder (c2.android.hevc.decoder, 32 instances, ~62 fps at 1080p on this CPU). Four extra H.265 cells at 1 fps cost about 4 / 62 of one core's decoding budget — far cheaper than the complexity of a shared decoder, and it works on any device. Promotion to full rate stays on hardware instances only.
    • Plan: (1) software fallback for key-frame cells beyond the hardware limit, measured on the AG-216 with 8 H.265 cameras; (2) the shared hardware decoder only if (1) costs too much CPU.
  5. compot-dev
    added 5 commits that reference this issue on Oct 13, 2026

    Merge pull request #137 from OpenIPC/feat/decoder-info

    1d4a179

    fix: a software cell's losses do not pause hardware cells (#88)

    4c5bc26

    fix: software decoders get their own instance limit, stay out of hard…

    d709c5d

    fix: read a software codec's limit before choosing it (#88, Qodo)

    ed0fde4

    Merge pull request #138 from OpenIPC/feat/sw-keyframes

    56016e6

  6. compot-dev commented on Oct 13, 2026

    @compot-dev
    compot-dev
    on Oct 13, 2026
    CollaboratorAuthor

    Step 1 merged in #138: key-frame cells beyond the hardware instance limit now run on a software decoder (c2.android.*) instead of showing "Decoder limit".

    Device test on AG-216 (Aceline-1), 9 cells, 6 of them H.265 (hardware HEVC limit 4):

    • the 5th H.265 cell plays key frames on c2.android.hevc.decoder (up to 32 instances), corner: "key frames · 1.0 fps · software";
    • a software cell is never promoted to full rate, and its losses/load stay out of the hardware planning (otherwise it paused a healthy hardware cell);
    • software decoders have their own instance budget; beyond it the usual decoder-limit message is shown;
    • app CPU about 65 % of one core with all 9 cells.

    Next (step 2): one shared decoder for the key frames of several cameras.

  7. compot-dev commented on Oct 13, 2026

    @compot-dev
    compot-dev
    on Oct 13, 2026
    CollaboratorAuthor

    Decision on step 2 (2026-10-13): deferred. The plan was to build the shared hardware decoder only if step 1 (software decoder for key-frame cells beyond the hardware limit, #138) costs too much CPU. Measured on AG-216 with 9 cells (6 × H.265, 2 of them software, 1 cell at full rate): the app uses ~83 % of one core on a 4-core CPU — no reason yet for the complexity of switching one codec between surfaces. Revisit when a device or layout runs out of CPU.

  8. compot-dev
    added
    deferredPostponed, not planned for now
    and removed
    in progressBeing worked on right now (open PR)
    todayTasks for today
    on Oct 13, 2026

  9. compot-dev commented on Oct 13, 2026

    @compot-dev
    compot-dev
    on Oct 13, 2026
    CollaboratorAuthor

    Closed for now: on the AG-216 the software decoder of step 1 (#138) covers the cells beyond the hardware limit at acceptable CPU (~83 % of one of four cores with 9 cells). Reopen when a device report shows a box that runs out of CPU or decoder instances in multi-view — the research notes and the probe results above stay valid.

  10. compot-dev
    closed this as not plannedon Oct 13, 2026
  11. compot-dev
    added 7 commits that reference this issue on Oct 15, 2026

    feat: /stats.json lists the video decoders with instance limits and a…

    dd0052f

    Merge pull request #137 from OpenIPC/feat/decoder-info

    89a5f41

    feat: key-frame cells beyond the hardware decoder limit use a softwar…

    3857686

    fix: a software cell's losses do not pause hardware cells (#88)

    36c9253

    fix: software decoders get their own instance limit, stay out of hard…

    514befe

    fix: read a software codec's limit before choosing it (#88, Qodo)

    67923f7

    Merge pull request #138 from OpenIPC/feat/sw-keyframes

    b1a1487
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    deferredPostponed, not planned for nowenhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions