cli: verify signatures of side-loadable DLLs on Windows (#2411)

[ronaldtse]

Summary

• Addresses Fix rnpkeys dll sideloading vulnerability of rnp.dll #2411: the Windows loader resolves rnp.dll (and any other DLL) from the executable's directory before main() runs, so replacing a DLL next to a signed rnpkeys.exe — as distributed with Thunderbird — executes attacker code inside a process that appears trusted to EDR/AV.
• Scope note (important): the root cause is the distribution layout (signed exe + writable sibling DLL), which is owned by the distributor — rnp's code contains no untrusted-path loading of its own. This PR is defense-in-depth, not a complete fix: a main()-time check runs after the loader has already mapped the DLLs, so it detects and aborts but cannot prevent the replaced module's DllMain from executing. Complete closure for signed distributions is static linking (BUILD_SHARED_LIBS=OFF), documented in docs/packaging.adoc — Thunderbird has ruled that out due to size, hence this runtime check.
• Design (reworked after discussion with Thunderbird in Fix rnpkeys dll sideloading vulnerability of rnp.dll #2411 so that embedders need zero build changes):
  • the check lives in src/rnp/fficli.cpp, a file every embedder of the rnp CLI sources (Thunderbird's third_party/rnp moz.build) already compiles into both rnp and rnpkeys — a plain re-vendor picks it up;
  • WinVerifyTrust is resolved at runtime from System32 (LoadLibraryExW + GetProcAddress, action GUIDs instantiated locally via initguid), so no wintrust/crypt32 link-time dependency is added for anyone, and the verification API itself cannot be side-loaded;
  • only modules from the executable's directory that are part of its transitive import chain are verified (walking the in-memory PE import tables) — not every app-dir module. Thunderbird's install directory holds ~100 DLLs that are not rnp's to police, and verifying each would cost seconds per run; the import chain keeps it to rnp.dll (plus any sibling backend DLLs) at tens of milliseconds;
  • the check activates only when the executable itself is signed — unsigned dev builds are unaffected (also keeps CI green) — and fails open with a warning if the verification API is unavailable.
• Blast-radius analysis (why this hardening should not cause new problems): false positives are limited to the exe's own import chain, which a signed distributor ships signed by construction; revocation is not checked (offline-safe); the only hard-fail condition is signed exe + unsigned/invalid signature in its own import chain, which is precisely the attack signature. Failure modes are fail-open (unavailable API, empty path); the code parses only OS-provided module paths and already-mapped, loader-validated PE headers.

Test plan

• Windows legs compile and pass (msys2 shared/static + native MSVC) — unsigned binaries exercise the skip path
• rnp_tests.test_dll_verify_unsigned passes on Windows
• non-Windows builds unaffected (verified locally: macOS build + CLI smoke test; net diff has no build-system changes)
• signed-path behavior to be validated by a distributor (e.g. Mozilla) since CI has no signing infrastructure

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 85.46%. Comparing base (100cc8c) to head (e2e65d9).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #2478   +/-   ##
=======================================
  Coverage   85.46%   85.46%           
=======================================
  Files         125      125           
  Lines       22962    22962           
=======================================
  Hits        19625    19625           
  Misses       3337     3337           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ronaldtse]

@ni4 your approval here was auto-dismissed when I rebased the branch onto current main (to pick up the #2480 gpg-flake coverage) - the content is otherwise exactly what you reviewed: the fficli-based DLL verification plus the macOS library-validation docs. Could you re-approve when convenient? Everything is green except the fuzzers-openssl red that clears once #2477 lands.

[ni4]

LGTM x2 :)