Add composite signature interface for RSA and ECDSA signatures

[mamckee]

OpenSSL 3.5 introduced hardcoded composite signature + digest interfaces for RSA and ECDSA. This PR adds the following composite signature interfaces to the SymCrypt provider, implementing the sign/verify message functions in place of sign/verify digest functions. The original RSA and ECDSA signature interfaces are still available.

[Copilot]

Pull request overview

This PR adds OpenSSL 3.5 “composite” signature algorithm entries (e.g., RSA-SHA256 / ECDSA-SHA256) to the SymCrypt provider and implements message-based sign/verify flows for those composite algorithms while keeping the existing RSA/ECDSA interfaces intact.

Changes:

• Introduces shared RSA/ECDSA signature context headers and refactors existing implementations to expose reusable internal sign/verify helpers.
• Adds new RSA/ECDSA composite (“sigalg”) signature implementations that use SIGN_MESSAGE / VERIFY_MESSAGE entry points.
• Registers the new composite algorithm names in the provider’s signature algorithm table and adds the new sources to the build.

Reviewed changes

Copilot reviewed 10 out of 10 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
SymCryptProvider/src/signature/p_scossl_rsa_signature.h	New shared RSA signature context + internal helper declarations.
SymCryptProvider/src/signature/p_scossl_rsa_signature.c	Refactor/expose RSA internal sign/verify; add state tracking for message APIs.
SymCryptProvider/src/signature/p_scossl_rsa_sigalg_signature.c	New RSA composite signature implementations (RSA-SHA* / RSA-SHA3-*).
SymCryptProvider/src/signature/p_scossl_ecdsa_signature.h	New shared ECDSA signature context + internal helper declarations.
SymCryptProvider/src/signature/p_scossl_ecdsa_signature.c	Refactor/expose ECDSA internal sign/verify; add state tracking for message APIs.
SymCryptProvider/src/signature/p_scossl_ecdsa_sigalg_signature.c	New ECDSA composite signature implementations (ECDSA-SHA* / ECDSA-SHA3-*).
SymCryptProvider/src/p_scossl_names.h	Adds composite algorithm name strings for RSA/ECDSA.
SymCryptProvider/src/p_scossl_base.c	Registers composite signature algorithms in the provider dispatch table.
SymCryptProvider/CMakeLists.txt	Adds new sigalg signature source files to the build.
ScosslCommon/src/scossl_rsa.c	Removes version guards around RSA_PSS_SALTLEN_AUTO_DIGEST_MAX usage (OpenSSL 3.5 baseline).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Comments suppressed due to low confidence (1)

SymCryptEngine/src/e_scossl_ecc.c:640

• The function comment now states it returns 1 (valid) / 0 (invalid) / -1 (error), but several error paths in this function still return SCOSSL_FAILURE (0). That makes internal errors indistinguishable from an invalid signature for OpenSSL callers. Either update the error returns here to -1, or adjust the comment to reflect the actual behavior.

// Return
// 1 (SCOSSL_SUCCESS) for valid signature
// 0 (SCOSSL_FAILURE) for invalid signature
// -1 for error
int e_scossl_eckey_verify(int type, _In_reads_bytes_(dgst_len) const unsigned char* dgst, int dgst_len,
                          _In_reads_bytes_(sig_len) const unsigned char* sigbuf, int sig_len, _In_ EC_KEY* eckey)
{
    const EC_KEY_METHOD* ossl_eckey_method = NULL;
    SCOSSL_ECC_KEY_CONTEXT *keyCtx = NULL;

    switch( e_scossl_get_ecc_context(eckey, &keyCtx) )
    {
    case SCOSSL_FAILURE:
        SCOSSL_LOG_ERROR(SCOSSL_ERR_F_ENG_ECKEY_VERIFY, ERR_R_OPERATION_FAIL,
            "e_scossl_get_ecc_context failed.");
        return SCOSSL_FAILURE;
    case SCOSSL_FALLBACK:
        ossl_eckey_method = EC_KEY_OpenSSL();
        PFN_eckey_verify pfn_eckey_verify = NULL;
        EC_KEY_METHOD_get_verify(ossl_eckey_method, &pfn_eckey_verify, NULL);
        if (!pfn_eckey_verify)
        {
            return SCOSSL_FAILURE;
        }
        return pfn_eckey_verify(type, dgst, dgst_len, sigbuf, sig_len, eckey);
    case SCOSSL_SUCCESS:
        break;
    default:
        SCOSSL_LOG_ERROR(SCOSSL_ERR_F_ENG_ECKEY_VERIFY, ERR_R_INTERNAL_ERROR,
            "Unexpected e_scossl_get_ecc_context value");
        return SCOSSL_FAILURE;
    }

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.