Skip to content

feat(l1): let a discovery consumer edit the local ENR without locks - #7488

Draft
MegaRedHand wants to merge 2 commits into
feat/discv5-predicate-queriesfrom
feat/discv5-lock-free-local-enr
Draft

MegaRedHand wants to merge 2 commits into
feat/discv5-predicate-queriesfrom
feat/discv5-lock-free-local-enr

Conversation

@MegaRedHand

Copy link
Copy Markdown
Collaborator

Motivation

A consensus client that embeds ethrex discovery (ethlambda, lambdaclass/ethlambda#732) has to republish its eth2 ENR entry at every fork-digest boundary. On feat/discv5-predicate-queries it can read the live ENR but not write it:

  • DiscoveryServer serves its own local_node_record field and only copies it into shared_local_node after an IP or fork-id update. An edit written into the shared mirror never reaches the wire, and the next update overwrites it.
  • No message or method edits arbitrary entries; the only consumer-triggered edit is the fork-id watch, which writes the eth entry.

Stacked on feat/discv5-predicate-queries, which has no PR of its own yet. Retarget to main once it does or merges.

Description

SharedLocalNode is now the only copy of the local identity, and it has no locks.

Before After
Type Arc<RwLock<LocalNode>> newtype over Arc<ArcSwap<LocalNode>>
What discovery serves private local_node_record copy the slot, read on every PING, PONG, NODES (distance 0), handshake and discv4 ENRRESPONSE
IP voting, fork-id refresh edit the copy, then overwrite the mirror compare-and-swap on the slot, retried on conflict
Consumer edits none SharedLocalNode::edit_record(&signer, FnMut(&mut NodeRecordPairs))
  • Lost updates: every writer loads the slot, edits a clone through NodeRecord::edit (one seq bump, re-signed), and swaps it in only if the slot still holds what it loaded. Otherwise it re-applies its edit on top of the newer record, so a consumer edit and an IP update racing each other both land.
  • edit_record takes FnMut, not FnOnce, because a retry runs the edit again on the newer record. The closure must give the same result every time.
  • The caller passes the signer. This keeps the node key out of every holder of the slot (RPC, L2 RPC, shutdown). edit_record refuses a key that doesn't match the record's secp256k1 entry, and keeps id and secp256k1 as they were.
  • Endpoint entries (ip, ip6, tcp, udp) stay with discovery. IP voting rewrites them, and LocalNode::node doesn't follow a consumer's change to them.
  • DiscoveryServer::spawn API change: it now takes local_node: SharedLocalNode instead of local_node: Node, local_node_record: NodeRecord and shared_local_node. The too_many_arguments expectation is gone with them. The server keeps the node id outside the slot, since packet decoding needs it on every packet and it never changes.
  • What peers receive doesn't change: start_network seeds the slot with the record it already served. RPC (admin_nodeInfo) and shutdown now read that same record instead of the one built at startup from node_config.json. Unchanged: the served record still starts at seq 2 on every boot, regardless of the persisted seq.
  • Dependency: arc-swap is pinned to 1.9.2 (June 28) in all four lockfiles. 1.9.3 (Oct 18) is still inside ethlambda's 14-day crates.io publish-age cooldown.

Tests:

  • consumer_edit_racing_an_ip_update_keeps_both: an IP update swaps in between a consumer edit's read and its swap. The edit runs twice and the final record carries both changes at seq + 2.
  • a_consumer_edit_is_what_peers_fetch: two real discv5 servers. A's consumer edits the record after A starts, B asks A for its ENR, and B ends up with the edited record.
  • apply_predicted_ip_updates_shared_arc and the update_local_ip_* tests pass. The first now checks one slot instead of a copy and a mirror.

Checklist

  • Updated STORE_SCHEMA_VERSION (crates/storage/lib.rs) if the PR includes breaking changes to the Store requiring a re-sync.

A consensus client that embeds discovery (ethlambda) has to republish its
`eth2` entry at every fork-digest boundary, but it could only read the ENR.
Discovery served a private copy of the record and overwrote the shared
mirror on every IP or fork-id update, so an edit made through the mirror
never reached the wire and was lost on the next update.

The shared slot is now the only copy. It is an ArcSwap: readers take a
snapshot without locking, and every writer (IP voting, the fork-id
refresh, the new `SharedLocalNode::edit_record`) swaps a new identity in
by compare-and-swap, re-applying its edit on top of whatever got in first
instead of overwriting it.

`DiscoveryServer::spawn` takes the slot in place of the node, record and
mirror it took before. `start_network` seeds the slot with the record it
already served, so what peers receive is unchanged; RPC and shutdown now
read that same record instead of the one built at startup.

arc-swap is pinned to 1.9.2, since 1.9.3 is still inside ethlambda's
14-day crates.io publish-age cooldown.
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

⚠️ Known Issues — intentionally skipped tests

Source: docs/known_issues.md

rpc-compat log-bearing cases excluded

Where: KNOWN_EXCLUDED_TESTS in .github/scripts/check-hive-results.sh counts out
eight hive rpc-compat cases — the four eth_getLogs cases, eth_getBlockReceipts/get-block-receipts-latest,
and three eth_getTransactionReceipt cases. They are exactly the cases whose recorded
response contains at least one log object; every case with an empty log array still runs.
Note this leaves eth_getLogs with no rpc-compat coverage at all, since all four of its
cases are in the set.

Why: ethrex populates blockTimestamp on log objects, as geth, besu, nethermind, reth
and erigon all do. hive's rpc-compat compares responses byte-exactly (jsondiff.FullMatch;
the lenient checkJSONStructure path applies only to cases upstream marks speconly), and
the corpus is pinned to execution-apis d08382ae (2025-02-10), whose recordings predate the
field — it entered the schema in execution-apis#639 and the fixtures in #846 (2026-07-22).
So the extra key cannot match, and this is a property of the pin rather than of the response.

The pin cannot move, and this is not temporary. The pin sits one commit before
execution-apis#627, which moved the test chain to a pre-merge genesis: the current corpus has
~36 proof-of-work blocks before its terminal total difficulty. ethrex does not support
pre-merge chains and will not, so importing that chain.rlp fails at block 1 —
validate_block_header has no pre-London base-fee path. Every revision carrying
blockTimestamp in its fixtures also carries that chain, so there is no revision that
satisfies both. Nor can the corpus be patched locally: rpc-compat's Dockerfile clones
ethereum/execution-apis by hard-coded URL, so the branch buildarg cannot point at a fork.

Coverage: the field itself is pinned by
block_timestamp_is_on_the_log_and_not_on_the_receipt in
crates/networking/rpc/types/receipt.rs, which asserts it is present on each log and absent
from the receipt level.

Removal: delete the entries if ethrex ever gains pre-merge chain import, or if upstream
marks these cases speconly so they are type-checked instead of compared byte-for-byte.


rpc-compat malformed storage key cases excluded

Where: KNOWN_EXCLUDED_TESTS in .github/scripts/check-hive-results.sh counts out
eth_getStorageAt/get-storage-invalid-key and eth_getStorageAt/get-storage-invalid-key-too-large.
daily_hive_report.yaml runs the same pin without that list, so both show as failures there,
as the log-bearing cases above do.

Why: both cases send a storage key that does not decode (0xasdf, and a key of 65 hex
digits). ethrex answers -32602: Invalid params, the code JSON-RPC 2.0 reserves for malformed
params. The pinned recordings expect geth's -32000; execution-apis#712 (2025-12-04)
re-recorded both as -32602. hive strips error messages but compares the code, so the pinned
recordings cannot match. The pin cannot move, for the reason given in the section above.

Coverage: the key decode error surfaces as RpcErr::BadParams, and the BadParams →
-32602 mapping is pinned by eth_get_block_access_list_rejects_extra_params in
test/tests/rpc/block_access_list_tests.rs.

Removal: delete the entries if the pin ever moves past execution-apis#712.


The stateless schema id does not identify the encoding

Where: STATELESS_INPUT_SCHEMA_ID in crates/common/types/stateless_ssz.rs.

Upstream keeps the stateless input schema id at 0x1501
(fork_index 0x15 << 8 | revision 0x01) across incompatible body changes. Four
encodings have now shipped under it: tests-zkevm@v0.6.2, then #3248 + #3278,
then #3356, which moved state, codes and public_keys from SszList to
ProgressiveList, then #3652 in tests-zkevm@v21.0.1, which removed
public_keys outright without bumping the revision. ethrex speaks the last one.

The consequence is that the 2-byte prefix cannot be used to detect a stale or
mismatched bundle. A wrong-dialect input is accepted by the id check and then
fails later — in SSZ decode, or on a root that does not match — rather than being
rejected up front for what it is. only_amsterdam_schema_id_decodes therefore
proves less than its name suggests.

Worth raising upstream: a revision field that does not move across a body change
provides no version negotiation at all.


ZisK guest program hash changes with the unsync_cell gate

Where: crates/common/types/block.rs, transaction.rs.

The gate on the single-threaded unsync_cell::OnceCell moved from
all(feature = "eip-8025", target_arch = "riscv64") to
all(feature = "zisk", target_arch = "riscv64") when the eip-8025 feature was removed.

The guest ELFs were previously built --features "<zkvm>-build-elf,ci", which never enabled
eip-8025, so they compiled the atomic once_cell variant. bin/zisk/Cargo.toml does enable
ethrex-common/zisk, so the ZisK guest now compiles the unsafe impl Sync cell instead.
That changes the ELF bytes and therefore the program hash and verification key.

This is intended (the guest is single-threaded, so the unsync cell is sound and cheaper), but it
is a VK change rather than a no-op refactor, and the diffstat presents it as a file rename
(eip8025_cell.rs → unsync_cell.rs). Anyone pinning a ZisK VK across this change must
re-register it. The stateless-validator crate now forwards ethrex-common/zisk from its own
zisk feature so the two ZisK guests do not disagree on the cell type.


Release signing key is an unprotected repository secret

Where: .github/workflows/tag_release.yaml.

MINISIGN_SECRET_KEY is a plain repository secret. There is no environment: on
finalize-release or dry-run-release-assets, and gh api repos/lambdaclass/ethrex/rulesets
shows only branch-targeted rulesets, so the github.ref_type == 'tag' condition is a workflow
check rather than an enforced boundary: anyone who can push a tag can reach the signing key.

This is a repository-settings change, not a code change, so it is recorded here rather than
fixed in the tree. Recommended:

  1. Move MINISIGN_SECRET_KEY / MINISIGN_PASSWORD into a GitHub Environment with required
    reviewers, and add environment: to the two jobs that sign.
  2. Add a ruleset targeting refs/tags/v* restricting who may create release tags.

Until then, the compromise of that key is silent and durable: signatures would still verify
against the committed .github/minisign.pub.

@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Lines of code report

Total lines added: 59
Total lines removed: 31
Total lines changed: 90

Detailed view
+-----------------------------------------------------------+-------+------+
| File                                                      | Lines | Diff |
+-----------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex/ethrex.rs                               | 181   | -5   |
+-----------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex/l2/initializers.rs                      | 540   | -10  |
+-----------------------------------------------------------+-------+------+
| ethrex/cmd/ethrex/partial/mod.rs                          | 358   | -1   |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/discovery/discv4_handlers.rs | 512   | +1   |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/discovery/server.rs          | 736   | -10  |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/network.rs                   | 790   | +4   |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/p2p/types.rs                     | 682   | +54  |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/admin/mod.rs                 | 113   | -3   |
+-----------------------------------------------------------+-------+------+
| ethrex/crates/networking/rpc/rpc.rs                       | 2289  | -2   |
+-----------------------------------------------------------+-------+------+

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

L1 Ethereum client

Projects

Status: No status

Development

Successfully merging this pull request may close these issues.

1 participant