Skip to content

Ensure enc_frm_finish() is called on error paths in oapve_encode() - #295

Merged
kpchoi merged 1 commit into
AcademySoftwareFoundation:mainfrom
fkyslov:fix-enc-frm-finish-on-error
Sep 29, 2026
Merged

kpchoi merged 1 commit into
AcademySoftwareFoundation:mainfrom
fkyslov:fix-enc-frm-finish-on-error

Conversation

@fkyslov

@fkyslov fkyslov commented Sep 25, 2026

Copy link
Copy Markdown
Contributor

Summary

Ensures enc_frm_finish() is called on error paths in oapve_encode() (src/oapv.c):

  • Release input and reconstruction imgb reference counts via enc_frm_finish(ctx, stat) when enc_frame() or MD5 payload insertion returns an error after enc_frm_prepare().

Testing

  • Verified all 24 ctest unit/conformance tests pass with AddressSanitizer and UndefinedBehaviorSanitizer.

@kpchoi kpchoi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks correct: enc_frm_prepare() takes the references at its end, and all three error paths between it and enc_frm_finish() now release them.

Two small refinements are suggested.

  1. The mid == NULL check is an argument check, so it can be done at the top of oapve_encode(), before any frame is prepared. Then no cleanup is needed for it:
    for(i = 0; i < ifrms->num_frms; i++) {
        frm = &ifrms->frm[i];
        if(ctx->use_frm_hash[i] &&
           (frm->pbu_type == OAPV_PBU_TYPE_PRIMARY_FRAME || frm->pbu_type == OAPV_PBU_TYPE_NON_PRIMARY_FRAME)) {
            oapv_assert_rv(mid != NULL, OAPV_ERR_INVALID_ARGUMENT);
        }
    }
  1. The remaining two error paths can use the oapv_assert_gv() / ERR: pattern that is used elsewhere in this file, instead of repeating enc_frm_finish() and return:
        ret = enc_frame(ctx, bs);
        oapv_assert_gv(OAPV_SUCCEEDED(ret), ret, ret, ERR);
        ...
        if(ctx->use_frm_hash[i]) {
            if(frm->pbu_type == OAPV_PBU_TYPE_PRIMARY_FRAME ||
               frm->pbu_type == OAPV_PBU_TYPE_NON_PRIMARY_FRAME) {
                ret = oapv_set_md5_pld(mid, frm->group_id, ctx->imgb_r);
                oapv_assert_gv(OAPV_SUCCEEDED(ret), ret, ret, ERR);
            }
        }
    ...
    return OAPV_OK;

ERR:
    enc_frm_finish(ctx, stat);
    return ret;
}

ERR: is reached only from inside the frame loop, after enc_frm_prepare() has succeeded, so enc_frm_finish() always releases exactly the references taken for the current frame.

Signed-off-by: Fyodor Kyslov <kyslov@google.com>
@fkyslov
fkyslov force-pushed the fix-enc-frm-finish-on-error branch from e85fdfa to cb71060 Compare September 28, 2026 17:58
@fkyslov

fkyslov commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor Author

Updated per review feedback: moved the mid != NULL validation when ctx->use_frm_hash[i] is enabled before enc_frm_prepare(), and routed post-prepare error paths to an ERR: label that calls enc_frm_finish(ctx, stat).

@kpchoi kpchoi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@kpchoi
kpchoi merged commit 05d1360 into AcademySoftwareFoundation:main Sep 29, 2026
10 checks passed

@kpchoi kpchoi left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants