Skip to content

perf(beacon): fuse electra epoch steps 2-9 into one registry pass - #633

Merged
MegaRedHand merged 3 commits into
perf/beacon-epoch-participation-bfrom
perf/beacon-epoch-participation-c
Oct 22, 2026
Merged

MegaRedHand merged 3 commits into
perf/beacon-epoch-participation-bfrom
perf/beacon-epoch-participation-c

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

One of three alternative implementations of the same follow-up to the persistent-tree registry (epoch participation without per-index registry reads): (a) #630, (b) #632, (c) this PR. Stacked on (b): this PR's diff is the fusion only, and it targets (b)'s branch.

Motivation

After (b), electra and fulu still walk the tree-backed registry once per remaining epoch step: registry updates, slashings, pending deposits and effective-balance updates. Several of those steps also recompute get_total_active_balance, which descends once per active validator. This is option (c) of the plan, the Lighthouse-style single pass (process_epoch_single_pass).

Change: fuse electra/fulu epoch steps 2-9 into one loop

stf::epoch::single_pass visits each validator once, in index order, working on a local copy of the validator and its balance. Per validator, in spec order:

  1. inactivity score;
  2. rewards and penalties;
  3. registry action (queue, eject or activate);
  4. slashing penalty;
  5. planned deposit top-up;
  6. effective-balance update.

Changes are written back after the loop. What depends on more than one validator stays outside it, as in Lighthouse:

  • Exit churn: the cursor is advanced locally in index order and written back once.
  • Pending deposits: the queue is planned before the loop. A deposit counts as "exited" if the validator already has an exit or this epoch's registry update will eject it. Top-ups of existing validators apply inside the loop; deposits that create validators apply after it, and one registry walk serves all pubkey lookups.
  • Consolidations: pending consolidations run after the loop. The effective-balance update of every validator they name, and of every validator created by deposits, waits until then.
  • Total active balance: taken from the summary and reused for slashings and every churn limit. It equals get_total_active_balance through step 8.

The hysteresis test both effective-balance updates use is now one checked leaves_hysteresis_band: a balance near u64::MAX is an ArithmeticOverflow, as in the spec and Lighthouse's safe_add, rather than a debug panic or a release wrap that moves the effective balance. It keeps the spec's short-circuit or.

The genesis epoch, and states whose registry-sized lists differ in length, fall back to the step-by-step path. The step functions stay public and callable in isolation for the fixtures. The per-step rules both paths share were factored out so both run the same code: registry action, slashing penalty, effective-balance hysteresis, inactivity score and the exit-cursor advance.

Testing

  • In debug builds, a registry of up to 4096 validators also runs through the specification-shaped steps 1-9 on a clone, and the two full post-states must have equal hash_tree_root. A mismatch names the differing fields.
  • A randomized fused-vs-unfused test runs 800 electra and fulu cases. They cover ejections that move the churn cursor, activations and the activation queue, slashings, every pending-deposit branch (including new pubkeys with valid and invalid signatures), and processed, slashed and not-yet-withdrawable consolidations. Two deliberately injected bugs were caught within a few seeds. About one state in ten carries a near-u64::MAX balance, since one now fails the whole epoch; those cases check that both paths fail together. As in (b), the finalized epoch stays at or behind the previous epoch.
  • Clippy -D warnings and the ethlambda-state-transition lib tests pass, including the dev-profile Test minimal preset run, where overflow checks are on.
  • Beacon spec filters epoch_processing, rewards, sanity, finality, random, fork_choice and transition pass on both presets. That run predates the rebase onto (b)'s finality-delay fix and the checked hysteresis; CI's beacon spec jobs cover both.

Over (b), this saves about 200 ms per epoch-start block and nothing on other blocks. That is more than the plan estimated, probably because the fusion also removes the repeated total-active-balance calls and the per-deposit pubkey scans (not profiled). A per-epoch total cache (follow-up 01) and a pubkey-to-index map would recover part of that gain more simply.

Benchmark (all three options)

ethlambda benchmark import replay on ethlambda-5: cargo build --release, mainnet corpus of 128 blocks (slots 15279073..=15279200, 4 epoch-start blocks), legs interleaved base a b c for two rounds. Base against base: 0.996. Every leg imported all 128 blocks.

base (c913728) (a) iterator passes (b) summary pass (c) single pass
process wall 104.0 s 96.1 s 92.0 s 91.0 s
user CPU 215.1 s 208.4 s 205.1 s 203.8 s
stf, epoch-start block (mean of 4) 1503 ms 1126 ms 912 ms 709 ms
stf p50, other blocks 577 ms 517 ms 478 ms 479 ms
paired stf vs base (ratio of totals) 1 0.869 0.797 0.785
get_head p50 201 ms 214 ms 226 ms 225 ms
peak RSS 3.88 GiB 3.93 GiB 4.04 GiB 4.04 GiB
  • The per-block saving comes from the pulled-up tip, which reruns justification on every imported block.
  • (c) gains nothing over (b) on ordinary blocks, since their per-block path is the same.
  • get_head rises as stf falls, although no option touches the head computation. This is probably contention with the background state writer, as seen in earlier import benchmarks; it is not verified. Block wall time still falls.

Base

@MegaRedHand

Copy link
Copy Markdown
Collaborator Author

Import-replay benchmark (re-measured on the current base)

Method. ethlambda benchmark import replay, release build, 128 consecutive mainnet blocks (slots 15279073..=15279200, 4 of them epoch-start blocks) on a 16-core host. This PR's leg and a base leg (beacon-chain-integration) were interleaved, 2 rounds each; the table shows the mean of the two. Both legs carry #651, so stf excludes the state-writer hand-off wait. Block roots were identical to the base in every leg. Base vs base noise floor: 1.004.

base this PR (includes #632) change
stf epoch-start block (mean of 4) 1505 ms 713 ms −53%
stf ordinary block p50 577 ms 476 ms −18%
get_head p50 200 ms 223 ms +11%
block wall p50 777 ms 696 ms −10%
user CPU (whole run) 214.7 s 203.8 s −5%
peak RSS 3.90 GiB 4.33 GiB +0.43
paired stf, ratio of totals (round 1 / round 2) 0.789 / 0.776

Per epoch-start block (stf ms):

slot base this PR
15279104 1607 835
15279136 1521 727
15279168 1486 683
15279200 1405 609

This reproduces the earlier measurement on the older base (epoch-start 1503 → 709 ms, ordinary 577 → 479 ms), including the small get_head rise. That rise is probably contention with the background state writer's encode, which now overlaps get_head more; not verified.

All five performance PRs together (#647, #648, #649, #650, this one), measured with a 1 s delay between blocks so the state writer drains (--block-delay, #651):

  • ordinary stf 527 → 163 ms; epoch-start 1440 → 420 ms;
  • block wall p50 729 → 185 ms (3.9x); user CPU −30%.

In that build, the fused pass's inactivity-score, balance and validator writes go through #647's in-order write cursor, and the scores are #650's tree list.

MegaRedHand added a commit that referenced this pull request Oct 2, 2026
…6-638-gloas-live

Conflicted files:
- CLAUDE.md
- crates/blockchain/state_transition/src/beacon/stf/epoch/altair.rs
- crates/blockchain/state_transition/src/beacon/stf/epoch/electra.rs
- crates/blockchain/state_transition/src/beacon/stf/epoch/mod.rs
- crates/blockchain/state_transition/src/beacon/stf/epoch/rewards.rs
- crates/common/ssz-tree/tests/model.rs

Gloas adaptations (gloas keeps validators and balances in
ProgressiveList, so the cursor and BeaconState::registry_mut as written
cannot serve it):
- ssz-tree: ProgressiveList gains iter_cow / try_update_each, backed by
  ProgressiveIterCow, one IterCow per subtree in turn. IterCow gets a base
  offset so ElemCow::index is the list-wide index, and its next_cow is
  split into advance/take so the chained pass holds no borrow across
  subtrees. A unit test crosses several subtrees against libssz's root.
- types: BeaconState::registry_mut (a struct of &mut Validators and
  &mut Balances) is replaced by two methods that dispatch over both list
  kinds: try_update_balances and try_update_validators_with_balances (the
  latter stops on a short balances list, zip semantics). The public
  RegistryMut struct is dropped; the private enum of the same name stays.

Other adaptations:
- mutators.rs apply_balance_deltas uses the element accessors
  (validator_count, iter_balances) and try_update_balances.
- rewards.rs: take #647's apply_balance_deltas call.
- stf/epoch/mod.rs and electra.rs process_effective_balance_updates:
  cursor pass with #633's checked leaves_hysteresis_band /
  updated_effective_balance; the early UnknownValidator check for a short
  balances list from this branch is kept before the pass.
- altair.rs apply_rewards_and_penalties: decided balance changes are
  applied through one try_update_balances pass instead of balance_mut per
  change; the delta arithmetic order is untouched.
- model.rs: keep both sides' tests (progressive list and write cursor).
MegaRedHand added a commit that referenced this pull request Oct 2, 2026
…3-64-633-636-638-gloas-live

Conflicted files:
- crates/blockchain/state_transition/src/beacon/stf/epoch/altair.rs
- crates/blockchain/state_transition/src/beacon/stf/mod.rs
- crates/common/ssz-tree/tests/model.rs
- crates/common/types/src/beacon/containers/mod.rs
- crates/common/types/src/beacon/containers/shared.rs

Gloas adaptations:
- tree_fields! gets a gloas line: validators and balances (progressive
  trees) plus block_roots, state_roots, historical_roots, eth1_data_votes,
  randao_mixes, slashings and historical_summaries, which gloas takes from
  the shared aliases. inactivity_scores is left out on purpose: gloas
  declares its own flat libssz ProgressiveList for it, which buffers
  nothing, so there is nothing to flush or rebase.
- ssz-tree: ProgressiveList implements Buffered, so the field list can
  hold the progressive registry beside the bounded fields.
- apply_pending_mutations / has_pending_mutations dispatch over all eight
  beacon forks through buffered()/buffered_mut(); the lean guard on
  has_pending_mutations is kept (the merge had dropped it with the old
  registry match).
- rebase_on: the validators/balances list-kind match stays (so a gloas
  state still rebases onto a gloas base only); the other fields are shared
  types in every fork and rebase unconditionally; inactivity scores
  rebase only for a Tree/Tree pair.
- Inactivity scores are a tree List before gloas and a flat slice-like list
  in gloas, so no &[u64] can serve both. altair_validator_lists now returns
  an InactivityScoresRef view (len, get, in-order iter, ptr_eq,
  has_pending_updates, PartialEq) as its third element, and
  inactivity_scores_mut (a &mut [u64]) becomes inactivity_score_mut(index),
  an element write that dispatches over both kinds.
- stf/epoch/altair.rs update_inactivity_scores: decides over an in-order
  walk of the scores (in step with the summary) and writes only changed
  scores afterwards, which subsumes both #633's "write if different" and
  #650's no-op skip; the error order (first eligible validator without a
  score, then checked-add overflow) is unchanged.
- stf/mod.rs tests: BEACON_FORKS now includes Gloas, the historical
  summaries write covers gloas, and the inactivity score case expects no
  pending write on gloas. process_slot's registry check uses the new
  BeaconState::registry_has_pending_updates, since the per-slot roots
  writes stay buffered by design.
- storage store.rs test and the epoch-processing fixture runner comments
  follow the renamed accessors.

Other: shared.rs imports both List/Vector and the progressive alias; model.rs
keeps both sides' tests (the base side was empty).
MegaRedHand added a commit that referenced this pull request Oct 5, 2026
MegaRedHand added a commit that referenced this pull request Oct 5, 2026
…36-638-gloas-live

Brings gloas validator duties (produceBlockV4, envelope publication, PTC
duties and payload attestations, gloas attestation data and aggregates, VC
gloas support) onto the deployment branch, keeping every behavior of #626,
#633, #636, #638, #646, #647-#652, #656, #658-#660 and the sync-committee and
liveness endpoints.

Conflict resolutions keep both sides: the attestation pool stays in Store
(tmp) while the payload attestation pool is threaded through P2P and the RPC
handles (feature); the aggregate endpoints keep tmp's liveness recording and
attesting indices and add the feature's fork-header check and gloas pooling;
the VC tests and fake execution client serve both fulu blobs and gloas V6.

Semantic fixes:
 a. POST /eth/v2/beacon/blocks (gloas) calls publish_beacon_block(block,
    Vec::new()): gloas columns travel with the envelope. RecordingNetwork
    implements publish_beacon_block(block, sidecars) and both new methods.
 b. produceBlockV4 appends client versions to the graffiti exactly like
    produceBlockV3 (graffiti::execution_client_version run alongside the
    payload build, with_client_versions, Extension<OwnVersion>) and logs it.
 c. Attestation data, aggregate_attestation keep require_execution_client and
    require_validated for gloas slots; payload_attestation_data now applies
    the same two rules (503 without an execution client, or when the voted
    block's payload is unvalidated).
 d. Proposer duties v1 and v2 serve gloas epochs from a fulu or gloas state's
    proposer_lookahead; nothing refuses gloas any more; v2 keeps its
    dependent root.
 e. The VC's per-validator ProposerSettings apply to gloas proposals (graffiti
    in the BlockRequest, fee recipient compared with the bid's); a test pins
    the graffiti. VC tests updated to the ProposerSettings constructors.
 f. gloas production reads the attestation pool from Store and calls the
    stf with the ActiveBalanceCache the perf work added; fulu production and
    pack_operations are untouched (gloas blocks carry no pooled operations).
 g. Chain events are emitted by the chain actor only, so nothing on the RPC
    publish paths needed to move; gloas imports reach it unchanged.
 h. Cargo.lock unchanged; cargo check --locked passes.
@MegaRedHand
MegaRedHand marked this pull request as ready for review October 5, 2026 22:22
@MegaRedHand
MegaRedHand added this pull request to stack #664 October 5, 2026 22:22
@MegaRedHand
MegaRedHand removed this pull request from stack #664 October 5, 2026 22:22
@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

I reviewed the PR diff in /tmp/pr_diff.txt and the touched code paths. Overall this looks careful: the refactors preserve overflow checks, keep state writes deferred, and add a solid fused-vs-unfused equivalence oracle in debug/test builds. I did not spot an obvious consensus, memory-safety, or security bug in the changed logic.

A few concise notes:

  • Looks good overall

    • The extracted helpers like ExitChurnCursor, SlashingsContext, and next_inactivity_score keep the spec-sensitive arithmetic checks intact.
    • The fused Electra epoch path is backed by a strong equivalence harness in crates/blockchain/state_transition/src/beacon/stf/epoch/single_pass_equivalence.rs:470 and crates/blockchain/state_transition/src/beacon/stf/epoch/single_pass.rs:86, which is exactly the right safety net for consensus code.
  • Performance win

    • Reusing total_active_balance via balance_churn_limit_for / activation_exit_churn_limit_for avoids extra registry scans in hot epoch processing paths; that’s a clean optimization in crates/blockchain/state_transition/src/beacon/helpers/electra.rs:354 and crates/blockchain/state_transition/src/beacon/helpers/electra.rs:378.
    • The fused path in crates/blockchain/state_transition/src/beacon/stf/epoch/single_pass.rs:134 should materially reduce repeated full-registry passes without changing step ordering.
  • Minor maintainability nit

    • crates/blockchain/state_transition/src/beacon/stf/epoch/single_pass.rs:232 assumes at most one aggregated top-up entry per validator:
      • if let Some(&&(_, amount)) = top_ups.peek().filter(|&&&(at, _)| at == index) { ... }
    • That is currently true because DepositPlan::new aggregates with a BTreeMap, but the invariant is implicit. A brief comment there would help future readers avoid accidentally changing DepositPlan in a way that breaks this single peek()/next() consumption pattern.
  • Minor API/readability nit

    • ExitChurnCursor::read takes &mut BeaconState even though it only reads fields, in crates/blockchain/state_transition/src/beacon/helpers/electra.rs:629. If the underlying accessor permits it, &BeaconState would better signal intent and reduce accidental borrow pressure in future callers.

No blocking issues from me based on the diff provided.


Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Review of PR 633: single-pass electra/fulu epoch processing

I read the fused path in single_pass.rs and the refactors in helpers/electra.rs and altair.rs. I skimmed electra.rs and the docs, and I did not read single_pass_equivalence.rs or mod.rs. I did not build the crate or run any tests.

Overall: the fusion looks sound. The behaviour-preserving refactors hold up too.

What I checked

  • Ordering inside the loop. Inactivity score, rewards, registry action, slashing penalty, top-up, then effective balance is the spec's per-validator order. Slashings correctly read the validator after step 4. Ejections can move withdrawable_epoch, so this matters.
  • Justification before reward constants. weigh_with_totals runs first. RewardContext::new and is_leaking() are built afterwards, so they see the updated finalized checkpoint.
  • Total active balance reuse. Reusing the summary's total through step 8 is justified. Exit and activation epochs are at least compute_activation_exit_epoch, and effective balances only change in step 9.
  • Exit-churn cursor. ExitChurnCursor::advance is the old body moved verbatim, and compute_exit_epoch_and_update_churn now delegates to it. It uses checked arithmetic and leaves the cursor untouched on error.
  • Deposit planning. The reachable-prefix computation, first-match pubkey lookup, withdrawn/exited/postpone/churn branches and deposit_balance_to_consume all mirror process_pending_deposits. The "ejected now" handling is covered by the argument in the DepositPlan::new doc comment.
  • Consolidation deferral. consolidation_participants deliberately over-approximates. That is safe, because a deferred validator whose balance nothing touched ends up with the same effective balance. Newly appended validators from new-pubkey deposits also get step 9 (registry_len..new_registry_len).
  • Fallback. can_fuse falls back to the unfused steps for the genesis epoch and for mismatched list lengths. That preserves the per-step error behaviour.

Suggestions

  1. Debug oracle panics outside tests (single_pass.rs, process_steps_through_effective_balances).

    • The oracle is gated on debug_assertions and runs for registries up to 4096 validators. It clones the state, runs the slow path, and assert_eq!/panic!s on divergence.
    • The docs say release-fast keeps debug assertions, so any such profile would run it on a real node.
    • Since the oracle only uses cfg(debug_assertions), make sure no shipped or devnet profile enables them. An explicit cfg(test) or a feature flag would make that harder to get wrong.
    • Small registries (minimal-preset devnets) pay a full state clone plus two hash-tree-roots every epoch.
  2. DepositPlan::new takes the deposit queue out of state (single_pass.rs, around line 418).

    • It does mem::take(fields.pending_deposits_mut()). If a later step in fused returns an error, state is left with an empty queue.
    • That is harmless if callers always discard the state on Err. The unfused path also mutates partially on error, so this is no worse.
    • A one-line comment saying so would prevent a future caller from relying on the state after an error.
  3. first_indices_of is O(registry × reachable deposits) in the worst case.

    • At most MAX_PENDING_DEPOSITS_PER_EPOCH (16 on mainnet) keys are searched, so the inner loop is cheap, and the 8-byte prefix check helps.
    • It also exits early once every key is found. This is fine as written. I'm only noting it because the loop is hot.
  4. Minor. write_back re-indexes the validators, balances and scores with get_mut, so every change is a second tree descent. This is much cheaper than before but not free. If the ssz-tree list offers a batch or iterator-based mutation, that would be a possible follow-up.

I found no correctness or security problems. It's worth confirming in CI that the mainnet and minimal epoch_processing and sanity/transition fixtures pass, since those are the real check on the fusion. I'd also run single_pass_equivalence.rs with deposits, consolidations and ejections in the same epoch.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which introduces a single-pass optimization for Electra/Fulu epoch processing, with extensive testing infrastructure. Let me analyze the code carefully.

Overall Assessment

This is a significant optimization PR that fuses multiple epoch-processing steps into a single registry walk. The approach is well-documented and includes a debug oracle for equivalence checking. However, I've identified several issues ranging from correctness concerns to performance pitfalls.


Critical Issues

1. DepositPlan::new — Incorrect handling of is_validator_exited for ejected validators

File: single_pass.rs, lines 530-550

let ejected_now = validator.exit_epoch == FAR_FUTURE_EPOCH
    && matches!(
        registry_action(validator, current_epoch, finalized_epoch, config.ejection_balance),
        Some(RegistryAction::Eject)
    );
(
    validator.exit_epoch < FAR_FUTURE_EPOCH || ejected_now,
    !ejected_now && validator.withdrawable_epoch < next_epoch,
)

Problem: The is_validator_withdrawn calculation uses !ejected_now && validator.withdrawable_epoch < next_epoch, but this is checked before registry updates actually run. For a validator being ejected now, withdrawable_epoch is still FAR_FUTURE_EPOCH (or some old value), not the computed exit_epoch + MIN_VALIDATOR_WITHDRAWABILITY_DELAY.

However, the specification's process_pending_deposits runs after process_registry_updates, so ejected validators in the spec have their exit_epoch and withdrawable_epoch already set when deposits are processed.

In the single pass, you're checking withdrawable_epoch < next_epoch on the pre-ejection state. If a validator was previously exited (exit_epoch < FAR_FUTURE_EPOCH) with withdrawable_epoch < next_epoch, it's correctly marked withdrawn. But for a validator being ejected this epoch, ejected_now is true, so !ejected_now is false, making is_validator_withdrawn false — this matches the spec since the ejection hasn't set withdrawable_epoch yet.

Wait — let me re-check. The issue is: in the spec, process_registry_updates runs before process_pending_deposits. So by the time we process deposits, validators ejected this epoch have:

  • exit_epoch = compute_activation_exit_epoch(current_epoch) (or later)
  • withdrawable_epoch = exit_epoch + MIN_VALIDATOR_WITHDRAWABILITY_DELAY

In your fused loop, you're computing registry_action to determine if a validator will be ejected, but the withdrawable_epoch hasn't been updated yet. Your code sets is_validator_withdrawn = false for ejected_now validators because !ejected_now is false.

But this is correct behavior! A validator ejected this epoch should NOT have its deposit treated as "withdrawn" — it should be postponed (exited but not withdrawable). Let me trace through more carefully...

Actually, looking at the spec's process_pending_deposits:

  • If is_validator_withdrawn(validator) — credit the deposit
  • Elif is_validator_exited(validator) — postpone

For a validator ejected this epoch in the spec: after registry updates, exit_epoch is set and withdrawable_epoch = exit_epoch + delay > current_epoch + 1 = next_epoch. So is_validator_withdrawn is false, is_validator_exited is true → postpone.

In your code: ejected_now = true, so is_validator_exited = true || true = true, is_validator_withdrawn = false && ... = false → falls through to the is_validator_exited branch → postpone.

This appears correct. However, there's a subtle issue: what if validator.exit_epoch < FAR_FUTURE_EPOCH already (previously exited) AND ejected_now is also true? That shouldn't happen because registry_action returns Eject only when validator.exit_epoch == FAR_FUTURE_EPOCH.

Actually wait — registry_action checks validator.exit_epoch == FAR_FUTURE_EPOCH for ejection. So ejected_now implies validator.exit_epoch == FAR_FUTURE_EPOCH. The is_validator_exited becomes false || true = true. Correct.

But there's another case: what about a validator that was already exited (exit_epoch < FAR_FUTURE_EPOCH) and whose withdrawable_epoch < next_epoch? Then is_validator_withdrawn = true, and the deposit is credited. This matches the spec.

After deeper analysis, this appears correct. But the complexity warrants a comment explaining why pre-computing registry_action is valid here.


2. DepositPlan::new — found lookup uses stale validator state for first_indices_of

File: single_pass.rs, lines 484-495

let found = first_indices_of(
    state,
    deposits[..reachable].iter().map(|deposit| deposit.pubkey),
);

Problem: first_indices_of searches the registry for pubkeys. However, apply_pending_deposit (called in finish) appends new validators to the registry. The specification's process_pending_deposits processes deposits in queue order, and for deposits with "new" pubkeys, each successful deposit creates a validator that subsequent deposits in the same queue might match against.

Your DepositPlan computes found once upfront, but new validators created by earlier "new pubkey" deposits in the queue won't be found by later deposits that might match them. This is a correctness bug.

Example:

  • Deposit A: new pubkey X, valid signature
  • Deposit B: same pubkey X, any signature

In the spec: Deposit A creates validator for X. Deposit B finds existing validator for X and tops it up.

In your code: Both A and B are classified as NewPubkey (since found doesn't see the validator A will create). Then in finish, you apply new_pubkey_deposits in order, but apply_pending_deposit for A creates the validator, and for B... wait, let me check apply_pending_deposit.

Looking at apply_pending_deposit in electra.rs (line 513), it calls add_validator_from_pending_deposit which appends a new validator. For deposit B with the same pubkey, when apply_pending_deposit runs, it will find the existing validator (since it searches the current state) and top it up.

So the behavior is actually correct! The found classification as NewPubkey for B is conservative — it gets handled correctly in finish because apply_pending_deposit does its own lookup.

But wait — this means your DepositPlan classification is wrong for B (says NewPubkey, should be TopUp), but the final result is correct because apply_pending_deposit re-derives. This is inefficient but not incorrect.

However, there's a worse issue: churn accounting. Deposit B's amount was counted against processed_amount as a NewPubkey deposit. But if B actually gets matched to an existing validator created by A, does the churn limit apply correctly?

Looking at apply_pending_deposit — it doesn't check churn limits! The churn is only checked in DepositPlan::new. So if we have:

  • Churn limit: 32 ETH
  • Deposit A: 16 ETH, new pubkey
  • Deposit B: 16 ETH, same pubkey X

In spec: A creates validator (16 ETH, within churn). B finds existing validator, tops up 16 ETH (total 32, still within churn? Or does top-up not count against churn?).

Actually, re-reading the spec's process_pending_deposits: the churn limit available_for_processing is checked for ALL deposits, including top-ups of existing validators. Wait no — let me re-check...

Looking at your code in DepositPlan::new, lines 570-582:

match processed_amount.checked_add(deposit.amount) {
    Some(sum) if sum <= available_for_processing => {
        processed_amount = sum;
        match existing {
            Some(index) => {
                credit(index, deposit.amount);
                kinds.push(DepositKind::TopUp);
            }
            None => kinds.push(DepositKind::NewPubkey),
        }
    }
    _ => {
        is_churn_limit_reached = true;
        break;
    }
}

So NewPubkey deposits DO count against processed_amount / churn limit. And in finish, apply_pending_deposit for new pubkeys doesn't re-check churn — it just applies.

But for deposit B (same pubkey as A), classified as NewPubkey, it was counted in processed_amount. In finish, apply_pending_deposit for A creates the validator, then for B it finds the existing validator and... wait, does apply_pending_deposit check if the validator exists?

Looking at the code (not fully shown in diff, but implied), apply_pending_deposit likely checks if the pubkey exists and either creates or tops up. If B finds the validator A just created, it would top up without additional churn check. But B's amount was already counted in processed_amount, so this is correct.

After analysis: the behavior is correct, but the found classification is stale for new pubkeys that appear multiple times. This is a minor inefficiency (deposit might be queued as NewPubkey when it could be TopUp), not a correctness bug, because apply_pending_deposit handles the actual logic correctly.

However, I recommend adding a comment explaining this subtlety, or better, updating found dynamically as new validators are conceptually added.


3. first_indices_of — O(n × m) complexity with early exit bug potential

File: single_pass.rs, lines 620-651

fn first_indices_of(
    state: &BeaconState,
    pubkeys: impl Iterator<Item = BlsPubkey>,
) -> Vec<Option<usize>> {
    // ...
    for (index, validator) in state.validators().iter().enumerate() {
        let validator_prefix = prefix(&validator.pubkey);
        for (slot, wanted_prefix) in prefixes.iter().enumerate() {
            if found[slot].is_none()
                && *wanted_prefix == validator_prefix
                && wanted[slot] == validator.pubkey
            {
                found[slot] = Some(index);
                missing -= 1;
            }
        }
        if missing == 0 {
            break;
        }
    }
    found
}

Problem: The inner loop checks ALL prefixes for EVERY validator, even after finding a match. The found[slot].is_none() check prevents re-finding, but you still iterate all slots. With MAX_PENDING_DEPOSITS_PER_EPOCH potentially large and millions of validators, this is O(validators × deposits) in the worst case.

The prefix optimization helps (most validators fail the integer comparison), but the inner loop still runs for all deposits. Consider using a HashMap<u64, Vec<usize>> from prefix to list of wanted indices, so you only check deposits with matching prefixes.

More critically: the missing == 0 break only triggers when ALL deposits are found, not when we've scanned enough validators. If deposits reference pubkeys not in the registry, we scan the entire validator set every time. This is correct but slow.

Recommendation: Use a hash-based approach for O(validators + deposits) expected time:

let mut prefix_map: HashMap<u64, Vec<usize>> = HashMap::new();
for (i, prefix) in prefixes.iter().enumerate() {
    prefix_map.entry(*prefix).or_default().push(i);
}
// then for each validator, only check slots with matching prefix

4. consolidation_participants — Incorrect deferral logic

File: single_pass.rs, lines 403-420

fn consolidation_participants(state: &mut BeaconState) -> Result<Vec<usize>> {
    let mut fields = pending_queue_fields(state, "single-pass epoch processing")?;
    let mut participants: Vec<usize> = fields
        .pending_consolidations_mut()
        .iter()
        .flat_map(|consolidation| {
            [
                consolidation.source_index as usize,
                consolidation.target_index as usize,
            ]
        })
        .collect();
    participants.sort_unstable();
    participants.dedup();
    Ok(participants)
}

Problem: This collects ALL validators mentioned in pending_consolidations, but process_pending_consolidations (step 8) stops at the first source that is not withdrawable yet. Your comment acknowledges this: "A superset of the ones the consolidation step will actually touch... which is safe."

However, this is overly conservative. If there are many pending consolidations and only the first few are processed, you're still deferring effective-balance updates for ALL mentioned validators. This reduces the benefit of the single pass.

More importantly: the deferral includes target_index, but process_pending_consolidations only reads the source's effective balance and moves balance to the target. The target's effective balance isn't read during consolidation processing — only its balance is increased. So deferring the target's effective-balance update is unnecessary conservatism.

Wait, let me re-check. Looking at process_pending_consolidations in the spec: it decreases source balance and increases target balance. The target's effective balance isn't used for any decision. So targets don't need deferral.

Actually, looking more carefully: if target balance increases, and then effective balance update runs, the target's effective balance might increase. But if consolidation runs AFTER effective balance update in the spec order, then the target's new balance from consolidation IS visible to effective balance update.

In your fused loop, you defer effective balance update for targets, then run consolidation, then apply deferred effective balance updates. This matches the spec's order: consolidation changes balance, then effective balance update sees the new balance.

But if you didn't defer targets, the fused loop would apply effective balance update on the pre-consolidation balance, which would be wrong. So deferring targets IS necessary for correctness!

Wait, let me re-read your loop structure:

  1. Loop: for each validator, apply rewards, registry, slashings, top-ups, then effective balance update (unless deferred)
  2. After loop: process_pending_consolidations (moves balances)
  3. After consolidation: apply deferred effective balance updates

So targets DO need deferral because their balance changes in step 2 after the loop.

But sources also have their balance decreased in step 2. So sources need deferral too. Correct.

However, your consolidation_participants includes ALL pending consolidations, not just the ones that will actually be processed. If consolidation queue is [A->B, C->D] and A is not withdrawable, then neither consolidation runs, but you still defer B, C, D. This is safe but suboptimal.

Not a correctness bug, but a performance issue.


5. write_back — Potential panic on inconsistent state

File: single_pass.rs, lines 360-389

fn write_back(
    state: &mut BeaconState,
    // ...
) -> Result<()> {
    let validators = state.validators_mut();
    for (index, validator) in validator_changes {
        *validators
            .get_mut(index)
            .ok_or(Error::UnknownValidator(index as ValidatorIndex))? = validator;
    }
    // similar for balances, scores...
}

Problem: The function uses get_mut(index) and returns Error::UnknownValidator if out of bounds. But validator_changes is built from iterating state.validators() in the same fused function. How can indices become invalid?

If state.validators_mut() returns a different length than state.validators() did earlier, something went terribly wrong. But more critically: apply_pending_deposit in plan.finish() can APPEND validators, which happens AFTER write_back but BEFORE the deferred effective balance loop. The registry_len captured before plan.finish() is used to filter deferred, but new validators from new_pubkey_deposits need effective balance updates too.

Looking at lines 340-355:

let registry_len = state.validators().len();
plan.finish(state, config)?;
// ...
let new_registry_len = state.validators().len();
let mut effective_updates = Vec::new();
let patched = deferred
    .iter()
    .copied()
    .filter(|&index| index < registry_len)  // BUG: ignores new validators!
    .chain(registry_len..new_registry_len);  // adds new validators

Wait, this chains registry_len..new_registry_len which ARE the new validators. So new validators DO get effective balance updates. Good.

But what if a new pubkey deposit's validator is also a target of a consolidation? That can't happen — consolidations reference existing validators by index, and new validators get new indices after registry_len. So no overlap.

This appears correct.


6. ExitChurnCursor::read and write — Mutable borrow of state for read

File: electra.rs, lines 625-648

impl ExitChurnCursor {
    pub(crate) fn read(state: &mut BeaconState) -> Result<Self> {
        let fields = electra_state(state, "ExitChurnCursor::read")?;
        Ok(Self { /* ... */ })
    }
}

Problem: read takes &mut BeaconState but only reads from it. This is unnecessarily restrictive and prevents calling read when state is already borrowed. Should be &BeaconState.

Similarly, write correctly needs &mut BeaconState.

Also in single_pass.rs line 268: let mut exit_cursor = ExitChurnCursor::read(state)?; — this mutably borrows state for the read, but you also need state for other things. Looking at the code, state isn't used between read and when it's borrowed again for plan, so this works. But the API is still overly restrictive.

Recommendation: Change read to take &BeaconState.


7. DepositPlan::finish — core::mem::take on &mut self field

File: single_pass.rs, lines 597-608

fn finish(&mut self, state: &mut BeaconState, config: &Config) -> Result<()> {
    let remaining = core::mem::take(&mut self.remaining);
    // ...
    for deposit in core::mem::take(&mut self.new_pubkey_deposits) {
        apply_pending_deposit(state, &deposit, config)?;
    }
    Ok(())
}

Problem: This is correct Rust — core::mem::take replaces self.remaining with Vec::new() and returns the old value. But after this call, self.remaining is empty, and subsequent calls to finish would do nothing. This is fine if finish is


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

Electra and fulu walk the tree-backed registry once per epoch step:
inactivity updates, rewards, registry updates, slashings, pending deposits
and effective-balance updates, several of them with a validator(i) descent per
active validator. At mainnet scale that is most of an epoch-start block.

stf::epoch::single_pass visits each validator once, in index order, on a
local copy of it and of its balance, and writes the changes back afterwards.
What depends on more than one validator stays outside the loop: the exit-churn
cursor is advanced locally in index order, the pending-deposit queue is planned
before the loop (top-ups of existing validators inside it, deposits that create
validators after it, one registry walk for all pubkey lookups), and pending
consolidations run after it with the effective-balance update of every
validator they name deferred until they are done. The total active balance is
the summary's, which equals get_total_active_balance through step 8.

The genesis epoch and states whose registry-sized lists differ in length fall
back to the step-by-step path. The step functions stay public and callable in
isolation for the fixtures; the per-step rules the loop shares with them
(registry action, slashing penalty, effective-balance hysteresis, inactivity
score, exit-cursor advance) were factored out so both paths run the same code.

In debug builds a registry of up to 4096 validators is also run through the
specification-shaped steps on a clone and the two full states must hash equal.
A randomized test compares the fused and unfused paths on crafted electra and
fulu states.
balance + DOWNWARD_THRESHOLD and effective_balance + UPWARD_THRESHOLD were
unchecked in both effective-balance updates, so a balance near u64::MAX
panicked in debug builds and, in release, wrapped into a small sum that
moved the effective balance. The specification treats that overflow as an
invalid transition, and Lighthouse's single pass rejects it with safe_add.
Both updates now share one checked leaves_hysteresis_band, which keeps the
specification's short-circuit: the upward sum is only taken when the
downward comparison fails.

The fused-pass test gave nearly every state a near-u64::MAX balance, which
now fails the whole epoch in both paths, so only about one state in ten
carries one. Its finalized epoch also stays behind the previous epoch, as
in any reachable state, now that get_finality_delay fails past it.
@MegaRedHand
MegaRedHand force-pushed the perf/beacon-epoch-participation-c branch from 7437a06 to 512276a Compare October 20, 2026 17:32
MegaRedHand added a commit that referenced this pull request Oct 21, 2026
…#634)

Squash of feat/beacon-gloas at 1d667c4 (merge-base 35c074b).

Adapted onto #626-#633 and main (#642-#677):
- #632/#633's epoch single pass ported to the element accessors; fulu runs
  through the single pass, with the eth1-bridge gate on electra only.
- DB_VERSION 6 (gloas Config 5 + #628's table).
- compute_node_weights skips the boost outside the justified subtree
  (boost_reaches_the_descent), for #627's LiveChain prune; the
  boosted-finalized-block bound test is dropped and the differential tests
  skip pruned roots.
- #628's set_unrealized_justification slot argument in the tests;
  get_forkchoice_store keeps the anchor block's slot.
- #653's validator-by-id handler on iter_validators/balance.
@MegaRedHand
MegaRedHand added this pull request to stack #689 October 22, 2026 21:45
@MegaRedHand
MegaRedHand added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit c8057a0 Oct 22, 2026
12 checks passed
@MegaRedHand
MegaRedHand deleted the perf/beacon-epoch-participation-c branch October 22, 2026 22:33
tcoratger pushed a commit to tcoratger/ethlambda that referenced this pull request Oct 30, 2026
…/bci-626-63-64-633-636-638-gloas-live

Combines the gloas follower with the events (lambdaclass#626), liveness, sync duties,
epoch participation (lambdaclass#633), optimistic 503 (lambdaclass#636) and proposer config (lambdaclass#638)
branches. Beyond the textual conflicts, the two sides disagreed in places git
merged without complaint:

- gloas removed the whole-registry accessors (validators(), balances() and
  their _mut forms) because its registry is a different list type. lambdaclass#633's
  participation summary, fused epoch pass, reference steps and equivalence
  tests now go through the element-level accessors; a test-only
  replace_balances keeps the short-list cases those tests build.
- gloas moved PendingQueueFields into helpers::electra with take/set queue
  access and gave fulu its own process_pending_deposits, which drops the
  eth1-bridge gate. Its step-by-step fulu driver would have silently taken
  fulu off lambdaclass#633's fused pass, so fulu calls the single pass again. The pass
  and its unfused fallback now apply the gate only to electra, as
  drain_pending_deposits does.
- ExitChurnCursor reads and writes through churn_cursors_mut, which replaced
  electra_state.
- The events stream reads the fork-tagged DataColumnSidecar and gloas
  aggregates. The test chain stand-ins take the block announcement and the
  gloas envelope and payload-attestation methods.
- gloas publishes only under a held digest, and hands nothing to the chain
  otherwise. The published-aggregate hand-off test now makes its server hold
  the slot's digest instead of the placeholder one.
- gloas's debug_assert that the balances list matches the registry fired on
  lambdaclass#633's equivalence states, which shorten that list on purpose. Both
  effective-balance updates now fail with UnknownValidator at the first
  missing balance, as the specification's balances[index] does, instead of
  panicking in debug builds and truncating in release.
- gloas's block-production and attestation_data tests build routers without
  the OwnVersion (lambdaclass#638) and execution-client (lambdaclass#636) extensions those handlers
  now extract, so they got a 500; both layers are added.
- The new Beacon API readers on this side (attestation_data and
  aggregate_attestation's 503, node syncing/health, sync duties, state fork)
  judge optimism through block_is_optimistic. Gloas blocks never enter the
  pre-gloas optimistic set, so reading it directly would call every gloas
  head validated.
@MegaRedHand MegaRedHand added the performance Performance improvements or possible performance improvements label Nov 3, 2026
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
…#632)

One of three alternative implementations of the same follow-up to the
persistent-tree registry (epoch participation without per-index registry
reads): (a) lambdaclass#630, (b) this PR, (c) lambdaclass#633. They are meant to be compared,
and at most one of (a) and (b) should merge; (c) is stacked on this one.

## Motivation

The registry is now a persistent Merkle tree, so `state.validator(i)` is
a tree descent rather than an array index, while `validators().iter()`
walks leaf slices.

Epoch steps 1-3 (justification, inactivity updates, rewards) read the
registry through about 19 full scans and about 21 descents per active
validator. The pulled-up-tip check in fork choice repeats justification
on every imported block.

## Change: option (b), one summary pass, then flat passes

- New `helpers::participation`:
- `EpochSummary`: one walk of `validators().iter()`, zipped with the
flat participation lists, into per-validator flags, effective balances
and the balance totals.
- `ParticipationTotals`: the same scan without allocation. This is all
justification (and so the per-block pulled-up tip) needs.
- `RewardContext` and `ValidatorDeltas`: today's reward arithmetic per
validator, in today's operation order.
- `process_participation_steps` replaces the three step calls in the
altair, capella and electra drivers. It builds the summary once, then:
  1. runs justification on its totals;
  2. updates inactivity scores in place;
3. writes back only the balances that changed, applying each component
in spec order (never netted).
- The public step functions and `get_flag_index_deltas` /
`get_inactivity_penalty_deltas` keep their signatures, since the
fixtures call them one at a time. They become thin wrappers that build
what they need.
- Both effective-balance updates zip validators with
`balances().iter()`. Once only changed balances are written,
`balances()[i]` is a tree descent for most validators.
- `get_finality_delay` and `is_in_inactivity_leak` now return `Result`:
a finalized epoch past the previous epoch is an `ArithmeticOverflow`
rather than a debug panic or a release wrap into a permanent leak. The
spec calls that `uint64` underflow an invalid transition, and Lighthouse
and Grandine reject it the same way (Teku saturates to zero; Prysm and
Nimbus wrap). `RewardContext::new` passes the error up.
- `docs/beacon_stf.md`: a short note in "Registry and balances".
- `docs/spec_deviations.md`: from altair on, the fast path reads the
leak flag once per step, where the spec reads it per validator. So a
crafted state with the finalized epoch past the previous one fails here
even where the spec never reaches the check. No chain reaches that
state, since justification only finalizes an epoch behind the current
one.

## Testing

- The previous step 1-3 code is kept as a reference under `cfg(any(test,
debug_assertions))`.
- In debug builds, `process_participation_steps` runs the reference on a
clone for registries up to 4096. It asserts equal balances, scores,
justification bits and checkpoints, and that both paths error or neither
does. `release-fast` has debug assertions on, so this runs on every
fixture.
- Randomized tests (1500 cases across altair, bellatrix, capella,
electra and fulu) compare totals, flags, per-component deltas, the
isolated steps and error text against the reference.
- The random states keep the finalized epoch at or behind the previous
epoch, as every reachable state does. Past it, the fast path and the
reference fail at different points (see the deviation above), so the
comparison would test nothing. A unit test covers the error itself.
- The randomized tests also run in the `Test minimal preset` job, which
uses the dev profile with overflow checks on. `release-fast` turns them
off, which is how the underflow above went unnoticed under mainnet.
- Clippy `-D warnings` and the `ethlambda-state-transition` lib tests
pass.
- Beacon spec filters `epoch_processing`, `rewards`, `sanity`,
`finality`, `random`, `fork_choice` and `transition` pass on both
presets. That run predates the finality-delay change; CI's beacon spec
jobs cover it.

## Benchmark (all three options)

`ethlambda benchmark import replay` on ethlambda-5: `cargo build
--release`, mainnet corpus of 128 blocks (slots 15279073..=15279200, 4
epoch-start blocks), legs interleaved `base a b c` for two rounds. Base
against base: 0.996. Every leg imported all 128 blocks.

| | base (c913728) | (a) iterator passes | (b) summary pass | (c)
single pass |
|---|---|---|---|---|
| process wall | 104.0 s | 96.1 s | 92.0 s | 91.0 s |
| user CPU | 215.1 s | 208.4 s | 205.1 s | 203.8 s |
| `stf`, epoch-start block (mean of 4) | 1503 ms | 1126 ms | 912 ms |
709 ms |
| `stf` p50, other blocks | 577 ms | 517 ms | 478 ms | 479 ms |
| paired `stf` vs base (ratio of totals) | 1 | 0.869 | 0.797 | 0.785 |
| `get_head` p50 | 201 ms | 214 ms | 226 ms | 225 ms |
| peak RSS | 3.88 GiB | 3.93 GiB | 4.04 GiB | 4.04 GiB |

- The per-block saving comes from the pulled-up tip, which reruns
justification on every imported block.
- (c) gains nothing over (b) on ordinary blocks, since their per-block
path is the same.
- `get_head` rises as `stf` falls, although no option touches the head
computation. This is probably contention with the background state
writer, as seen in earlier import benchmarks; it is not verified. Block
wall time still falls.

## Base

- Based on `beacon-chain-integration` @ `c79fabd5`, merged into the
branch.
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
…ambdaclass#651)

## Motivation

Beacon post-states are persisted by one background thread fed through a
bounded queue. `StateWriterHandle::send` blocks while the queue is full,
and `Store::insert_state` calls it from inside `fork_choice::on_block`,
which the chain actor times as the `stf` phase. If blocks import faster
than the writer persists them, the importer's wait for the writer is
reported as state-transition time. This PR reports the wait as its own
phase, so `stf` measures state-transition work only.

Measured on the import-replay benchmark (128 mainnet blocks, eth-5) with
this change merged into each leg:

| leg | `stf` p50 (now exclusive) | `writer_wait` p50 | block wall p50 |
|---|---|---|---|
| base (beacon-chain-integration) | 581 ms | ~0 | 777 ms |
| each single performance PR (lambdaclass#647-lambdaclass#650, lambdaclass#633) | 353-618 ms | ~0 (max 4
us) | 551-778 ms |
| all five performance PRs merged | 202 ms | **326 ms** | 559 ms |

With all five merged, blocks import faster than the writer persists
their states, so the queue fills and the importer waits about 326 ms per
block. Without this phase that wait was reported as `stf` (about 530 ms
instead of 202 ms), which hid most of the combined gain and pointed at
the wrong place. The state writer, not the state transition, is now the
replay's bottleneck.

## Changes

| Area | Change |
|---|---|
| `storage/state_writer.rs` | `StateWriterHandle::send` returns the
`(start, end)` instants around its blocking `send` |
| `storage/store.rs` | `insert_state` records the hand-off span (first
start, last end if repeated) in a shared field; new
`Store::take_state_handoff()` takes and clears it |
| `blockchain/import_timing.rs` | `writer_wait_start`/`writer_wait_end`
on `ImportTimings` and `StoreTimings`, copied in `absorb_store`; new
`writer_wait` row after `stf`; header documents the nesting |
| `blockchain/lib.rs` | Beacon arm takes the hand-off instants right
after `on_block` returns (both `Ok` and `Err`); clears any stale span
before the call |
| `blockchain/metrics.rs` | `writer_wait` added to `BLOCK_IMPORT_PHASES`
(the `phase` label list) |
| `docs/metrics.md`, `docs/benchmarking.md` | Phase documented; beacon
`stf` now excludes the wait |

The replay benchmark builds its `phases` map from `BLOCK_IMPORT_PHASES`
and the same histogram, so it picks up `writer_wait` and the exclusive
`stf` with no code change.

One beacon `on_block` makes one `insert_state` call. The store still
merges repeated calls (first start, last end), the module's convention
for repeated sections.

## Semantics change

Beacon `stf` (the report row and the
`lean_block_import_phase_seconds{phase="stf"}` series) now excludes the
writer wait: it is the `on_block` span minus `writer_wait`, saturating
at zero. The rows still sum to the end-to-end time. Lean blocks never
set the new instants, so their `stf` and report are unchanged. A new
`phase="writer_wait"` series appears on beacon only. Dashboards
comparing beacon `stf` across this change will see it drop by the wait.

## Testing

| Command | Result |
|---|---|
| `cargo fmt --all` | clean |
| `cargo clippy --workspace --all-targets --profile release-fast -- -D
warnings` | clean |
| `cargo test -p ethlambda-storage --profile release-fast --lib` | 139
passed, 1 ignored |
| `cargo test -p ethlambda-blockchain --profile release-fast --lib` |
210 passed, 0 ignored |
| `cargo check -p ethlambda --profile release-fast` | clean |

New tests: `stf` minus `writer_wait`, wait absent, wait longer than
`stf` saturating to zero, row order and presence (existing label-list
test also covers the new label), and a storage test that `insert_state`
records a start <= end pair and that taking it clears it. Spec suites
and the replay benchmark were not run.

Clippy ran without `--locked`: `Cargo.lock` on the base (125dce1) is
already stale (`zk_alloc` is listed under the wrong crate), so
`--locked` fails there with or without this change. The lockfile is not
part of this PR.
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
…cursor (lambdaclass#647)

## Motivation

An epoch boundary writes every balance. `get_mut` on a tree list buffers
a copy of each element in a dense map, sorts the map into a second
`Vec`, then rebuilds every touched leaf and recomputes the root of every
written element, changed or not. The altair rewards step calls it up to
eight times per validator (four reward/penalty pairs, each as increase
then decrease), and `process_effective_balance_updates` reads
`balances()[i]` per validator, which was only cheap because the rewards
pass had just filled the map.

This PR adds an in-order lazy copy-on-write write cursor to
`ethlambda-ssz-tree` and moves the epoch balance writes onto it.
Expected gain (estimate, not measured on a real chain): tens of ms off
each epoch-start block at mainnet registry size. The import-replay
benchmark was not run for this PR.

## Changes

| commit | what |
|---|---|
| `perf(ssz-tree): add an in-order copy-on-write write cursor` |
`List`/`Vector::iter_cow`, `IterCow::next_cow`, `ElemCow::{index,
make_mut, set}`, `try_update_each`; `with_updated_leaves` generalized
over a leaf source; unit tests and model-based property tests |
| `perf(beacon): apply epoch balance and effective-balance writes
through the cursor` | `BeaconState::registry_mut` (disjoint borrows of
validators and balances); `apply_balance_deltas` used by the altair+ and
phase0 `process_rewards_and_penalties`; both
`process_effective_balance_updates`; new `tree_bench` rows; docs |

How the cursor works: it walks leaves in order and copies a leaf on the
first write that reaches it. At the leaf's end it compares only the
touched offsets with the original; if all are equal, the original leaf
(its `Arc` and cached hash) is kept. Otherwise the new leaf carries over
the element roots of unchanged elements and computes the changed ones.
On drop, one descent swaps the changed leaves in and rebuilds only their
paths. Pending buffered writes are applied before the pass starts.

## Design decisions

- `Drop` does the work, and the tree is untouched during the pass, so an
early return or a panic leaves a consistent list holding the writes so
far (tested with `catch_unwind`).
- Names as in the design: `iter_cow`, `next_cow`, `make_mut`, `set`,
`try_update_each`. No `set_with`.
- Changed composite elements get their root computed eagerly, on the
state-transition thread.
- A kept `Arc` or carried root always follows an actual comparison. The
cursor only creates fresh empty hash cells, or carried roots for a leaf
it owns; it never fills a shared `OnceLock`.
- Spec order for balances: per index, each pair is
`saturating_add(reward)` then `saturating_sub(penalty)`, pairs in order.
Deltas are never netted. Each balance depends only on its own deltas, so
index-by-index equals the spec's pair-by-pair loop.
- `apply_balance_deltas` lives in `helpers/mutators.rs` and is shared by
the phase0 and altair+ paths. It keeps the old error for a balances list
shorter than the registry, and does not touch balances past the registry
length.
- Rewards and both effective-balance passes move together: once rewards
bypass the map, the effective-balance reads would otherwise become tree
descents.
- Effective-balance updates only copy a leaf when the new effective
balance differs from the old one (it can equal it when capped at the
maximum).
- No leaf-slice API, no parallel bulk update.

## Not in this PR

- Switching `Balances` to a sparse `BTreeMap` update map: to be decided
after measuring whether a dense writer remains.
- `List::filled`: its only consumer is the participation-list work,
which is deferred.
- Migrating slashings, inactivity, and registry updates: sparse, flat
lists, or part of the epoch single-pass rewrite (lambdaclass#630, lambdaclass#632, lambdaclass#633).
Expect a merge conflict with lambdaclass#632 in `process_rewards_and_penalties`.
- Import-replay benchmark and any remote profiling.

## Testing

| command | result |
|---|---|
| `cargo test -p ethlambda-ssz-tree --profile release-fast` | 71 unit +
31 model tests passed (before the second commit) |
| `cargo test -p ethlambda-types --profile release-fast --lib` | 197
passed |
| `cargo test -p ethlambda-state-transition --profile release-fast
--lib` | 324 passed, 16 ignored |
| beacon spec suite, mainnet preset, full | 5734 passed, 4 failed |
| beacon spec suite, minimal preset, full | 40038 passed, 4 failed |
| `cargo clippy --locked --workspace --all-targets --profile
release-fast -- -D warnings` and the state-transition
`beacon-spec-tests` variant | clean |
| `cargo check -p ethlambda --profile release-fast` | ok |

The 4 spec failures in each preset are the `gossip/*` suites
(`gossip_beacon_block`, `gossip_data_column_sidecar`,
`gossip_beacon_aggregate_and_proof`, `gossip_beacon_attestation`): the
local gossip fixture directory is intentionally empty, so they match no
cases. Every other suite, including `epoch_processing`, `rewards`,
`sanity`, `finality`, `random` and `transition`, passes.

### tree_bench (2.4M validators, `--ignored`, single run)

Noisy: run while the machine was under a load average above 100 from
other builds, so absolute numbers (and even the order of the old rows)
are unreliable; read them as rough ratios only.

| row | ms |
|---|---|
| `epoch_sweep_tree` (old path) | 333.8 |
| `epoch_sweep_tree_phases` | write 58.1, apply 70.0, hash 83.1 |
| `epoch_sweep_cursor_tree` | 26.8 |
| `epoch_sweep_noop_cursor_tree` | 90.4 |
| `epoch_sweep_rebuild_tree` (rebuild baseline) | 105.8 |
| `rewards_apply_4_pairs_get_mut` | 342.5 |
| `rewards_apply_4_pairs_cursor` | 158.9 |
| `eb_updates_indexed` | 253.8 |
| `eb_updates_cursor` | 115.4 |
| `lockstep_three_lists_cursor` | 136.2 |

The bench asserts the cursor rows produce the same roots as the old
paths, and that a no-op sweep keeps the tree (`ptr_eq`).
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
## Motivation

`get_total_active_balance` rescans the registry about ten times per
ordinary block: once per attestation (through
`get_base_reward_per_increment`), twice in the sync aggregate, plus
churn and fork-choice callers. Each call built an index `Vec` of the
active validators and then read every index back with a `validator(i)`
tree descent. In the profile taken for the persistent-tree registry
change, this was the largest cost left after hashing (the plan's
estimate: about 414 ms per block at mainnet scale; an estimate, not
measured here).

## Changes

| Commit | Change |
|---|---|
| `perf(beacon): sum the total active balance in one registry pass` |
`get_total_active_balance` is one in-order `validators().iter()` pass
with the same `saturating_add` and one-increment floor as
`get_total_balance`. No index `Vec`, no descents. Every caller gets it,
cached or not. Randomized test (inline SplitMix64) against
`get_total_balance(get_active_validator_indices(..))`, plus an all-zero
registry. |
| `perf(beacon): cache the total active balance per epoch in the store`
| New `ActiveBalanceCache` (`ethlambda-storage`), held by the `Store`
beside the committee cache (`Store::active_balance_cache()`). Key
derivation and lookup in `ActiveBalanceCacheExt`
(`helpers/accessors.rs`). Attestation processing (altair, deneb,
electra) and `process_sync_aggregate` take it; plumbed up through block
processing and `state_transition`. New metric
`lean_beacon_total_active_balance_lookups_total{result}`. Docs in
`beacon_stf.md` and `metrics.md`. |
| `perf(beacon): read each attester's validator once per attestation` |
The effective balance is read once per attester, lazily on the first
newly earned flag, instead of once per flag. |

## Design decisions

- **External cache, not a state field.** The design review chose a
store-held table like `CommitteeCache` over a non-SSZ cache field on
`BeaconState`. That keeps the state type, its SSZ derive and its fork
upgrades untouched: no libssz change, no `rebase_on` adoption, no reset
at upgrades.
- **Why the key is sound.** The total for a state in epoch E is fixed by
two inputs. Effective balances are written only by epoch E-1's
`process_effective_balance_updates`; the Electra upgrade zeroes only
never-activated validators, which are not active at E. The active set at
E cannot change inside E, because every activation and exit lands at
least `MAX_SEED_LOOKAHEAD` epochs ahead; deposits append validators that
are not yet active; slashing sets `slashed` but leaves effective
balances alone (the spec's total keeps slashed validators). So `(E,
block root at the last slot of E-1)` identifies the history: two states
agreeing on it agree on the total. Sibling branches that diverge during
E-1 have different roots, so they get different entries.
- **Decision root differs from the shuffling key.** The shuffling key
uses the last slot of E-2, since the seed is fixed ahead of use; the
total is not, so it needs the later root. A new key type
(`ActiveBalanceKey`) keeps the two apart.
- **Epoch 0** keys on the genesis block root (slot 0), like the
shuffling key does for its first epochs. A state that cannot name the
root (the genesis state itself, or a root outside the history window) is
`unkeyable`: computed, not cached.
- **Only block processing uses the cache.** Between
`process_effective_balance_updates` and the slot increment a state is in
epoch E-1 but already carries E's balances, so a key rooted at the end
of E-2 would be wrong there. Block processing never runs in that window
(`process_slots` has already advanced the state), and epoch processing
keeps the uncached one-pass function. Documented on
`active_balance_key`.
- **Debug cross-check.** Every hit is compared against a fresh
computation under `debug_assert_eq!`, so a missed writer fails the
fixtures (`release-fast` keeps debug assertions on).
- **No single-flight.** A concurrent double compute on a miss is
accepted: the value is one cheap registry pass and both callers compute
the same number.
- **`on_block` reads the store's cache itself**
(`store.active_balance_cache()`), so its signature and the chain actor
are unchanged. Other callers (spec runners, block production, unit
tests) pass `ActiveBalanceCache::default()`.
- **Capacity** is a small constant with eviction of the lowest epoch;
entries are tiny, so room for several forks is cheap.
- **Per-attester read stays lazy** so that an attester earning no flag
is still never read, which keeps error behavior identical.

## Not in this PR

- Fork choice's `get_proposer_score` and `calculate_committee_fraction`
keep calling the uncached function; another change replaces that total
with a justified-balances snapshot.
- Epoch processing and the pulled-up tip: they compute totals inside the
single-pass epoch summary work (lambdaclass#630, lambdaclass#632, lambdaclass#633).
- Churn helpers (exits, consolidations, slashings): rare per block, they
use the one-pass function.
- A flat effective-balance epoch cache (progressive balances): waits
until something consumes it.

## Testing

Measured on release-fast with debug assertions. The import-replay
benchmark was not run for this PR, so there is no measured speedup to
report.

| Command | Result |
|---|---|
| `cargo clippy --locked --workspace --all-targets --profile
release-fast -- -D warnings` | clean |
| `cargo clippy -p ethlambda-state-transition ... --features
beacon-spec-tests -- -D warnings` | clean |
| `cargo test -p ethlambda-storage --lib` | 142 passed, 1 ignored (run
before the store-sharing test was added; not re-run after) |
| `cargo test -p ethlambda-state-transition --lib` | 330 passed, 16
ignored |
| beacon spec suite, mainnet (full) | 5734 passed, 152 ignored, 4 failed
|
| beacon spec suite, minimal (full) | 40038 passed, 3692 ignored, 4
failed |
| `cargo check -p ethlambda --profile release-fast` | clean |

The 4 failures per preset are the gossip `matched_fixture_cases` checks:
the gossip fixture tree is empty in this environment, so they match no
cases. Every other suite passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client performance Performance improvements or possible performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants