[unbalancedparentheses]

Motivation

Our suites check that ethrex computes the right answers. Our worst recent bugs came from how it behaves over time, under faults and between actors. Examples: the ~1.5 GB/day P2P leak (#6900), the ActorStopped → exit(2) teardown race (#7491), the request timeout that exited the node (#6823), the L2 import deadlock (#7429) and the duplicate-connection cleanup bug (#7435). This PR plans how to test for those failures with methods that database teams already use.

Description

Docs only. No code, CI or Cargo changes.

• docs/developers/l1/testing/roadmap.md (new, linked from SUMMARY.md and the testing README) covers:
  • what ethrex runs today (suite, what it catches, whether it is deterministic, where it runs);
  • gaps, each tied to the incident above that exposed it;
  • one short section per technique: failpoints/BUGGIFY, soak tests, keeping every fixed bug as a reproducible test, differential testing against geth/reth, spawned-test, Porcupine-style history checking, reference-model property tests (ShardStore), metamorphic storage tests (Pebble), Attacknet, PObserve-style invariant monitors, deterministic simulation (FoundationDB, TigerBeetle, Dropbox, madsim/turmoil), spawned actor exploration, ethshadow and Antithesis;
  • a three-phase plan with acceptance criteria, e.g. "reverting fix(l1): stop unbounded P2P connection memory growth (bounded mailbox, lifecycle timeouts, admission cap, map pruning) #6900's bounded mailbox fails the soak test";
  • sources.
• ROADMAP.md: a new ## Testing & Verification table. Each row links to its section of the new doc. The existing UX / DX testing rows are unchanged.

The priorities are proposals for the team to review. Cheap, high-value items come first (reproducible-bug rule and soak at 0; failpoints, differential testing and spawned-test adoption at 1). Simulation work comes later (p2p DST at 3; actor exploration, ethshadow and Antithesis at 4).

Notes:

• The CACM article on AWS correctness practices is cited without a link because cacm.acm.org and queue.acm.org return 403 to automated clients, which would fail the lychee link check.
• The ROADMAP rows use absolute docs.ethrex.xyz links, as the existing Based Rollups row does, because ROADMAP.md is also included into docs/roadmap.md and a relative path cannot resolve from both places. These links will work once the docs are deployed.

[github-actions]

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

rpc-compat log-bearing cases excluded

Where: KNOWN_EXCLUDED_TESTS in .github/scripts/check-hive-results.sh counts out
eight hive rpc-compat cases — the four eth_getLogs cases, eth_getBlockReceipts/get-block-receipts-latest,
and three eth_getTransactionReceipt cases. They are exactly the cases whose recorded
response contains at least one log object; every case with an empty log array still runs.
Note this leaves eth_getLogs with no rpc-compat coverage at all, since all four of its
cases are in the set.

Why: ethrex populates blockTimestamp on log objects, as geth, besu, nethermind, reth
and erigon all do. hive's rpc-compat compares responses byte-exactly (jsondiff.FullMatch;
the lenient checkJSONStructure path applies only to cases upstream marks speconly), and
the corpus is pinned to execution-apis d08382ae (2025-02-10), whose recordings predate the
field — it entered the schema in execution-apis#639 and the fixtures in #846 (2026-07-22).
So the extra key cannot match, and this is a property of the pin rather than of the response.

The pin cannot move, and this is not temporary. The pin sits one commit before
execution-apis#627, which moved the test chain to a pre-merge genesis: the current corpus has
~36 proof-of-work blocks before its terminal total difficulty. ethrex does not support
pre-merge chains and will not, so importing that chain.rlp fails at block 1 —
validate_block_header has no pre-London base-fee path. Every revision carrying
blockTimestamp in its fixtures also carries that chain, so there is no revision that
satisfies both. Nor can the corpus be patched locally: rpc-compat's Dockerfile clones
ethereum/execution-apis by hard-coded URL, so the branch buildarg cannot point at a fork.

Coverage: the field itself is pinned by
block_timestamp_is_on_the_log_and_not_on_the_receipt in
crates/networking/rpc/types/receipt.rs, which asserts it is present on each log and absent
from the receipt level.

Removal: delete the entries if ethrex ever gains pre-merge chain import, or if upstream
marks these cases speconly so they are type-checked instead of compared byte-for-byte.

---

rpc-compat malformed storage key cases excluded

Where: KNOWN_EXCLUDED_TESTS in .github/scripts/check-hive-results.sh counts out
eth_getStorageAt/get-storage-invalid-key and eth_getStorageAt/get-storage-invalid-key-too-large.
daily_hive_report.yaml runs the same pin without that list, so both show as failures there,
as the log-bearing cases above do.

Why: both cases send a storage key that does not decode (0xasdf, and a key of 65 hex
digits). ethrex answers -32602: Invalid params, the code JSON-RPC 2.0 reserves for malformed
params. The pinned recordings expect geth's -32000; execution-apis#712 (2025-12-04)
re-recorded both as -32602. hive strips error messages but compares the code, so the pinned
recordings cannot match. The pin cannot move, for the reason given in the section above.

Coverage: the key decode error surfaces as RpcErr::BadParams, and the BadParams →
-32602 mapping is pinned by eth_get_block_access_list_rejects_extra_params in
test/tests/rpc/block_access_list_tests.rs.

Removal: delete the entries if the pin ever moves past execution-apis#712.

---

The stateless schema id does not identify the encoding

Where: STATELESS_INPUT_SCHEMA_ID in crates/common/types/stateless_ssz.rs.

Upstream keeps the stateless input schema id at 0x1501
(fork_index 0x15 << 8 | revision 0x01) across incompatible body changes. Four
encodings have now shipped under it: tests-zkevm@v0.6.2, then #3248 + #3278,
then #3356, which moved state, codes and public_keys from SszList to
ProgressiveList, then #3652 in tests-zkevm@v21.0.1, which removed
public_keys outright without bumping the revision. ethrex speaks the last one.

The consequence is that the 2-byte prefix cannot be used to detect a stale or
mismatched bundle. A wrong-dialect input is accepted by the id check and then
fails later — in SSZ decode, or on a root that does not match — rather than being
rejected up front for what it is. only_amsterdam_schema_id_decodes therefore
proves less than its name suggests.

Worth raising upstream: a revision field that does not move across a body change
provides no version negotiation at all.

---

ZisK guest program hash changes with the unsync_cell gate

Where: crates/common/types/block.rs, transaction.rs.

The gate on the single-threaded unsync_cell::OnceCell moved from
all(feature = "eip-8025", target_arch = "riscv64") to
all(feature = "zisk", target_arch = "riscv64") when the eip-8025 feature was removed.

The guest ELFs were previously built --features "<zkvm>-build-elf,ci", which never enabled
eip-8025, so they compiled the atomic once_cell variant. bin/zisk/Cargo.toml does enable
ethrex-common/zisk, so the ZisK guest now compiles the unsafe impl Sync cell instead.
That changes the ELF bytes and therefore the program hash and verification key.

This is intended (the guest is single-threaded, so the unsync cell is sound and cheaper), but it
is a VK change rather than a no-op refactor, and the diffstat presents it as a file rename
(eip8025_cell.rs → unsync_cell.rs). Anyone pinning a ZisK VK across this change must
re-register it. The stateless-validator crate now forwards ethrex-common/zisk from its own
zisk feature so the two ZisK guests do not disagree on the cell type.

---

Release signing key is an unprotected repository secret

Where: .github/workflows/tag_release.yaml.

MINISIGN_SECRET_KEY is a plain repository secret. There is no environment: on
finalize-release or dry-run-release-assets, and gh api repos/lambdaclass/ethrex/rulesets
shows only branch-targeted rulesets, so the github.ref_type == 'tag' condition is a workflow
check rather than an enforced boundary: anyone who can push a tag can reach the signing key.

This is a repository-settings change, not a code change, so it is recorded here rather than
fixed in the tree. Recommended:

1. Move MINISIGN_SECRET_KEY / MINISIGN_PASSWORD into a GitHub Environment with required
   reviewers, and add environment: to the two jobs that sign.
2. Add a ruleset targeting refs/tags/v* restricting who may create release tags.

Until then, the compromise of that key is silent and durable: signatures would still verify
against the committed .github/minisign.pub.

[unbalancedparentheses]

Folded into #7518, which carries the spawned v0.7 adoption together with this testing roadmap (same commits).