librepgp: cap decompressed data and packet count during dump (fixes fuzz_dump OOM)

[ronaldtse]

Summary

• Fixes the fuzz_dump OOM that has made the fuzzing-openssl workflow red since its first run (all 13 runs on main failed; oss-fuzz's Botan-based fuzzing job has the separate known Botan 3.4.0 pin issue, [rnp] Update Botan to 3.6.0 google/oss-fuzz#15882).
• Root cause: a 4KB input with nested compressed data packets is a decompression bomb (each zlib layer expands ~1032x; two layers from 4KB already exceed 4GB). The dump recursed through the decompressed streams, and the JSON dump built an unbounded DOM from the resulting packet flood (measured 2.1GB RSS / 29s units in CI, reproduced locally with the exact same artifact hash). This is also a client-facing DoS: rnp --list-packets (or the FFI dump calls) on a crafted file exhausts memory. The existing MAXIMUM_NESTING_LEVEL / MAXIMUM_STREAM_PKTS checks bound neither the decompressed size nor the count of non-streaming packets (the observed bomb used pk-session-key packets).
• Fix, in the dump layer only (message decryption/parsing paths untouched):
  • each dump call gets a shared 256MB decompressed-data budget, enforced by a limiting source wrapper on top of every decompressed stream (shared across nesting levels, so nesting cannot multiply it);
  • a 32768-packet cap per dump, counting every packet (accumulated across armored messages as well);
  • hitting either limit truncates the dump gracefully (:too much decompressed data, stopping. / :too many packets, stopping.), consistent with the existing "too many layers" behavior.
• Regression: the OOM artifact is added to test_fuzz_dump corpus (outofmemory-12fd9b02d8465f2f, identical to the one libFuzzer saved on CI) with a 30s completion bound; it completes in ~1.5s.
• Verified locally: libFuzzer corpus + 500 mutation runs against the patched build complete in 12s with ~350MB peak RSS (766MB under ASAN) where the unpatched build OOMed at 2.1GB; all dump-related unit tests pass (test_fuzz_dump, test_stream_dumper, test_ffi_pkt_dump, test_cli_dump, test_ffi_dump_multiple_armored_messages, key dumps, etc.).

Security impact

Availability-only (CWE-409 / CWE-400): crafted input to --list-packets / dump FFI causes memory exhaustion. Happy to file it through the security channel instead if preferred - flagging here for visibility.

Test plan

• fuzzing-openssl workflow passes on this PR (first time ever)
• dump-related unit tests green
• no regressions in the full matrix

[codecov]

Codecov Report

❌ Patch coverage is 89.23077% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.45%. Comparing base (4ec426f) to head (6bf751c).

Files with missing lines	Patch %	Lines
src/librepgp/stream-dump.cpp	88.52%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2477      +/-   ##
==========================================
- Coverage   85.47%   85.45%   -0.02%     
==========================================
  Files         125      125              
  Lines       22964    23024      +60     
==========================================
+ Hits        19628    19676      +48     
- Misses       3336     3348      +12     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[ronaldtse]

@ni4 @antonsviridenko Ready for review -- all 137 checks green; the only red is the fuzzing job, which is the known oss-fuzz Botan 3.4.0 pin (google/oss-fuzz#15882), unrelated to this change.

Summary of the fix:

• fuzzers (openssl, libfuzzer) has been red since its first run on main: fuzz_dump OOMed at 2.1GB on a 4KB nested decompression bomb (each zlib layer expands ~1032x; the JSON dump built an unbounded DOM from the decompressed packet flood). Reproduced locally with the identical artifact hash. This is also a client-facing availability issue (rnp --list-packets / dump FFI on a crafted file).
• The fix is confined to the dump layer (decrypt/parse paths untouched): a shared 256MB decompressed-data budget enforced by a limiting source around each decompressed stream, plus a 32768-packet cap counting every packet (accumulated across armored messages too). Hitting a limit truncates the dump gracefully, like the existing "too many layers" cap.
• Regression: the exact OOM artifact is now a test_fuzz_dump corpus case.
• With this, fuzzers (openssl, libfuzzer) passes for the first time ever.

ni4: your approval covers the first review; anton: would you take the second? Happy to walk through the limiting-source details - one subtlety worth noting is that the budget is shared across nesting levels precisely so 32-deep nesting cannot multiply it.

[ni4]

@ronaldtse LGTM, thanks! Here we must have second approval as I understand.

[ronaldtse]

@antonsviridenko could you take the second review here when you have a moment? All 137 checks are green (only the known oss-fuzz fuzzing pin remains) and ni4 has approved.

The change is confined to the packet-dump layer: a shared 256MB decompressed-data budget (enforced by a limiting source around each decompressed stream, shared across nesting levels so 32-deep nesting cannot multiply it) plus a 32768-packet cap, so nested decompression bombs can no longer drive rnp_dump_packets_to_json / ..._to_output into unbounded memory/time. Hitting a limit truncates the dump gracefully, mirroring the existing MAXIMUM_NESTING_LEVEL behavior. Decrypt/parse paths are untouched, and the exact CI OOM artifact is now a test_fuzz_dump regression case.

Two review notes that may save you time:

• the limiter param deliberately holds a raw dump_budget_t * (not a shared_ptr) because source params are freed via plain free();
• armored-message walks accumulate the counters with targeted assignments instead of copy_params so the shared layers depth is not corrupted across sibling messages.