RSA-OAEP cannot encrypt or decrypt an empty message (C_Encrypt: CKR_FUNCTION_FAILED, C_Decrypt: CKR_ENCRYPTED_DATA_INVALID) #270

Summary

CKM_RSA_PKCS_OAEP refuses a zero-length message on both sides. C_Encrypt of 0 bytes returns CKR_FUNCTION_FAILED. C_Decrypt of a valid OAEP ciphertext of the empty message (made by OpenSSL) returns CKR_ENCRYPTED_DATA_INVALID. Every other length from 1 to k - 2hLen - 2 works, for SHA-256, SHA-384 and SHA-512. A token that cannot decrypt what another RSA-OAEP implementation produces breaks interoperability, and the decrypt side reports a valid ciphertext as invalid.

The limit is in wolfCrypt, not in wolfPKCS11's own code. wolfSSL/wolfssl#10012 tried to lift it there and was closed without being merged (2026-09-29). I am filing it here because this is where PKCS#11 users meet it. If it belongs in wolfSSL instead, please move it.

Found with wolfPKCS11 master 979895f (and v2.2.0, dbf5bd4) on static wolfSSL v5.9.4-stable (3c5eead). The two wolfCrypt checks below are unchanged on wolfSSL master (816e922).

Expected (PKCS#11 v3.2 / RFC 8017)

§6.1.8, Table 42 (PKCS #1 RSA OAEP: Key And Data Length): C_Encrypt input "≤ k-2-2hLen". RFC 8017 §7.1.1 (RSAES-OAEP-ENCRYPT) requires only mLen ≤ k - 2hLen - 2, so mLen = 0 is a valid message, and §7.1.2 decrypts it back to an empty string.

Results

RSA-2048 key pair generated by the token, OAEP with SHA-256 and MGF1-SHA-256, no label.

CK_RSA_PKCS_OAEP_PARAMS p = { CKM_SHA256, CKG_MGF1_SHA256, CKZ_DATA_SPECIFIED, NULL, 0 };
CK_MECHANISM m = { CKM_RSA_PKCS_OAEP, &p, sizeof p };
CK_BYTE in[1], out[256];
CK_ULONG outLen = sizeof out;
fn->C_EncryptInit(hSession, &m, hPub);
fn->C_Encrypt(hSession, in, 0, out, &outLen);   /* CKR_FUNCTION_FAILED */

Cause

• Encrypt. WP11_RsaOaep_PublicEncrypt (src/internal.c) calls wc_RsaPublicEncrypt_ex. wolfCrypt's RsaPublicEncryptEx (wolfcrypt/src/rsa.c) starts with if (in == NULL || inLen == 0 || out == NULL || key == NULL) return BAD_FUNC_ARG;. wp11_C_Encrypt (src/crypto.c, line 3610) maps that to CKR_FUNCTION_FAILED.
• Decrypt. RsaUnPad_OAEP returns 0 both for a valid empty message and for a padding error. RsaPrivateDecryptEx therefore rejects every 0-length result unless wolfSSL is built with WOLFSSL_RSA_DECRYPT_TO_0_LEN (default off): ret = ctMaskSelInt(ctMaskNotEq(ret, 0), ret, RSA_BUFFER_E);. wp11_C_Decrypt (src/crypto.c, line 4826) maps every error to CKR_ENCRYPTED_DATA_INVALID.

Suggested fix

The clean fix is in wolfCrypt, along the lines of wolfssl#10012: allow inLen == 0 for WC_RSA_OAEP_PAD on encrypt, and give RsaUnPad_OAEP an error value that is distinct from a valid empty message on decrypt, in constant time. Defining WOLFSSL_RSA_DECRYPT_TO_0_LEN alone is not enough, because a padding error and an empty message would then both decrypt to 0 bytes.

Until then, a note in wolfPKCS11's documentation that RSA-OAEP needs a non-empty message would help users who meet the CKR_FUNCTION_FAILED.

Found while running the KerckhoffsLabs.Security.Cryptography.Pkcs11 conformance suite.