Skip to content

perf(beacon): run epoch steps 1-3 over one registry scan - #632

Merged
MegaRedHand merged 5 commits into
mainfrom
perf/beacon-epoch-participation-b
Oct 22, 2026
Merged

MegaRedHand merged 5 commits into
mainfrom
perf/beacon-epoch-participation-b

Conversation

@MegaRedHand

@MegaRedHand MegaRedHand commented Sep 30, 2026 •

Copy link
Copy Markdown
Collaborator

One of three alternative implementations of the same follow-up to the persistent-tree registry (epoch participation without per-index registry reads): (a) #630, (b) this PR, (c) #633. They are meant to be compared, and at most one of (a) and (b) should merge; (c) is stacked on this one.

Motivation

The registry is now a persistent Merkle tree, so state.validator(i) is a tree descent rather than an array index, while validators().iter() walks leaf slices.

Epoch steps 1-3 (justification, inactivity updates, rewards) read the registry through about 19 full scans and about 21 descents per active validator. The pulled-up-tip check in fork choice repeats justification on every imported block.

Change: option (b), one summary pass, then flat passes

  • New helpers::participation:
    • EpochSummary: one walk of validators().iter(), zipped with the flat participation lists, into per-validator flags, effective balances and the balance totals.
    • ParticipationTotals: the same scan without allocation. This is all justification (and so the per-block pulled-up tip) needs.
    • RewardContext and ValidatorDeltas: today's reward arithmetic per validator, in today's operation order.
  • process_participation_steps replaces the three step calls in the altair, capella and electra drivers. It builds the summary once, then:
    1. runs justification on its totals;
    2. updates inactivity scores in place;
    3. writes back only the balances that changed, applying each component in spec order (never netted).
  • The public step functions and get_flag_index_deltas / get_inactivity_penalty_deltas keep their signatures, since the fixtures call them one at a time. They become thin wrappers that build what they need.
  • Both effective-balance updates zip validators with balances().iter(). Once only changed balances are written, balances()[i] is a tree descent for most validators.
  • get_finality_delay and is_in_inactivity_leak now return Result: a finalized epoch past the previous epoch is an ArithmeticOverflow rather than a debug panic or a release wrap into a permanent leak. The spec calls that uint64 underflow an invalid transition, and Lighthouse and Grandine reject it the same way (Teku saturates to zero; Prysm and Nimbus wrap). RewardContext::new passes the error up.
  • docs/beacon_stf.md: a short note in "Registry and balances".
  • docs/spec_deviations.md: from altair on, the fast path reads the leak flag once per step, where the spec reads it per validator. So a crafted state with the finalized epoch past the previous one fails here even where the spec never reaches the check. No chain reaches that state, since justification only finalizes an epoch behind the current one.

Testing

  • The previous step 1-3 code is kept as a reference under cfg(any(test, debug_assertions)).
  • In debug builds, process_participation_steps runs the reference on a clone for registries up to 4096. It asserts equal balances, scores, justification bits and checkpoints, and that both paths error or neither does. release-fast has debug assertions on, so this runs on every fixture.
  • Randomized tests (1500 cases across altair, bellatrix, capella, electra and fulu) compare totals, flags, per-component deltas, the isolated steps and error text against the reference.
  • The random states keep the finalized epoch at or behind the previous epoch, as every reachable state does. Past it, the fast path and the reference fail at different points (see the deviation above), so the comparison would test nothing. A unit test covers the error itself.
  • The randomized tests also run in the Test minimal preset job, which uses the dev profile with overflow checks on. release-fast turns them off, which is how the underflow above went unnoticed under mainnet.
  • Clippy -D warnings and the ethlambda-state-transition lib tests pass.
  • Beacon spec filters epoch_processing, rewards, sanity, finality, random, fork_choice and transition pass on both presets. That run predates the finality-delay change; CI's beacon spec jobs cover it.

Benchmark (all three options)

ethlambda benchmark import replay on ethlambda-5: cargo build --release, mainnet corpus of 128 blocks (slots 15279073..=15279200, 4 epoch-start blocks), legs interleaved base a b c for two rounds. Base against base: 0.996. Every leg imported all 128 blocks.

base (c913728) (a) iterator passes (b) summary pass (c) single pass
process wall 104.0 s 96.1 s 92.0 s 91.0 s
user CPU 215.1 s 208.4 s 205.1 s 203.8 s
stf, epoch-start block (mean of 4) 1503 ms 1126 ms 912 ms 709 ms
stf p50, other blocks 577 ms 517 ms 478 ms 479 ms
paired stf vs base (ratio of totals) 1 0.869 0.797 0.785
get_head p50 201 ms 214 ms 226 ms 225 ms
peak RSS 3.88 GiB 3.93 GiB 4.04 GiB 4.04 GiB
  • The per-block saving comes from the pulled-up tip, which reruns justification on every imported block.
  • (c) gains nothing over (b) on ordinary blocks, since their per-block path is the same.
  • get_head rises as stf falls, although no option touches the head computation. This is probably contention with the background state writer, as seen in earlier import benchmarks; it is not verified. Block wall time still falls.

Base

  • Based on beacon-chain-integration @ c79fabd5, merged into the branch.

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Claude Code Review

Review: perf(beacon): run epoch steps 1-3 over one registry scan (PR 632)

I read the diff through altair_reference.rs, the driver changes and the start of participation_equivalence.rs. I did not read the rest of the equivalence tests, and I did not build or run anything.

Overall: The change looks correct and I found no consensus-affecting bugs. It reproduces the spec arithmetic and its order of operations, and the oracle-on-clone assertion in debug builds is a strong safety net.

Correctness checks that hold

  • Scan membership (participation.rs:388-479): The scan matches the reference on every rule I compared.
    • Participation is counted only for validators active in the previous epoch and unslashed.
    • The eligibility rule active_before || (slashed && previous_epoch + 1 < withdrawable_epoch) is unchanged.
    • The current target list is read only for validators active now.
    • The previous list is aliased to the current one at genesis.
    • The previous-epoch error takes priority over the current-epoch one.
    • Each total is floored at one increment, as get_total_balance does.
  • Step order (altair.rs, run_participation_steps): The order is preserved.
    • Justification runs first on the totals.
    • Inactivity scores update next, reading the post-justification leak flag.
    • RewardContext::new is built after justification, so it also reads the post-justification finalized checkpoint.
    • The pre-justification total_active_balance it reuses is unaffected by justification.
  • Balance application (ValidatorDeltas::apply): It applies each component as reward then penalty, with a saturating subtract after each. That equals the reference's per-component increase_balance / decrease_balance loops, because validators are independent.
  • Errors in apply_rewards_and_penalties: Short-balance-list errors are deferred until all deltas are computed, which matches the spec's two-phase failure order.
  • Fork gating: The current_epoch > GENESIS_EPOCH + 1 justification gate, and skipping everything at genesis, match the old per-step skip rules.

Notes and nits (non-blocking)

  1. finality.rs:232: Making get_finality_delay and is_in_inactivity_leak fallible is the right call over wrapping. It is a deliberate behaviour change for unreachable states, and it is documented in spec_deviations.md. Please confirm all other callers, such as RPC and the rewards API, handle the new Result sensibly and don't unwrap.
  2. altair.rs (update_inactivity_scores): if updated != *score { *score = updated; } is only worthwhile if get_mut on the persistent list dirties the node on access. If it doesn't, plain assignment is simpler. If it does, a short comment saying so would help.
  3. altair.rs (apply_rewards_and_penalties): changes can be large during a leak or at an epoch boundary, since nearly every eligible validator changes. It is one Vec<(usize, Gwei)>, so it is cheap. Preallocating with summary.len() would avoid regrowth, though that is minor.
  4. participation.rs: EpochSummary::get, is_empty, is_leaking and ParticipationTotals's current_epoch_target sentinel behaviour exist partly for tests. If any are unused outside cfg(test), they may trip dead_code or add API surface. Clippy -D warnings passing suggests they are fine.
  5. The debug oracle runs only for registries of 4096 validators or fewer. Mainnet-size equivalence therefore rests on the randomized tests (up to 300 validators) and the spec fixtures. That is acceptable, but nothing exercises the fast path against the reference at large scale or across tree-page boundaries. Consider one test around the leaf and node page size of ethlambda_ssz_tree::List, so balances().iter() zipped with validators().iter() is checked across page edges.
  6. get_flag_index_deltas and get_inactivity_penalty_deltas now build a full EpochSummary and RewardContext per call. Fixtures call them one flag at a time, so this is fine, but it makes each call error on a finalized epoch past the previous one even with no eligible validators. That is a documented deviation.

The perf numbers (about 20% less stf time, with a small RSS increase) support the change. Good to merge once Point 1 is confirmed.


Automated review by Claude (Anthropic) · sonnet · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Codex Code Review

Looks good overall: the refactor is performance-motivated, keeps a spec-shaped reference path for debug/test comparison, and I did not spot an obvious consensus-breaking bug in fork choice / FFG / attestation accounting from the touched code.

Findings

  • No blocking correctness issue found in the touched paths.
  • The one-pass EpochSummary / RewardContext design appears semantically aligned with the previous helpers for normal reachable states, and the explicit note about the crafted finalized-checkpoint edge case is helpful.

Nits / Follow-ups

  • crates/blockchain/state_transition/src/beacon/helpers/participation.rs:51

    • The doc says “Bits 0..3” for the timeliness flags and “Bit 3” for eligibility, but FLAG_COUNT is 3, so the flags are bits 0..2 and eligibility is bit 3.
    • This is doc-only, but worth fixing because this module is subtle and consensus-critical.
  • crates/blockchain/state_transition/src/beacon/helpers/participation.rs:124

    • total_active and the per-flag totals use saturating_add. That matches existing helper semantics, but it would be worth an inline note that this is intentionally preserving get_total_balance behavior, since silent saturation in consensus code is easy to misread as an accidental weakening of overflow checks.
  • crates/blockchain/state_transition/src/beacon/stf/epoch/altair.rs:174

    • let balance = balance_iter.next().copied(); computes deltas before checking balance presence, specifically to preserve spec failure ordering. That’s reasonable, but it is non-obvious and security-sensitive. A short comment already exists; I’d keep it, and maybe mention that this intentionally avoids failing early on malformed balances.
  • crates/blockchain/state_transition/src/beacon/stf/epoch/altair.rs:230

    • The debug-only equivalence check is a strong safeguard, but CHECK_LIMIT: usize = 4096 is effectively part of the safety story for this optimization. Consider a brief rationale for why 4096 is the chosen cutoff, since regressions above that size will only be caught by dedicated tests.

Consensus / Security assessment

  • process_justification_and_finalization, process_inactivity_updates, and process_rewards_and_penalties preserve the required ordering, especially building RewardContext after justification and after inactivity-score updates.
  • The crafted-state deviation documented in docs/spec_deviations.md is acceptable as documented; I agree it should not affect reachable chain states.
  • No new unsafe code, unchecked indexing, or obvious memory-safety concerns.
  • Error handling is generally careful: malformed participation/scores propagate explicit IndexOutOfBounds / ArithmeticOverflow errors rather than wrapping.

Validation note

  • I could not run cargo test in this environment because rustup attempted to write under a read-only home directory. If you want, I can still give you a targeted test command set to run locally for the touched epoch-processing paths.

Automated review by OpenAI Codex · gpt-5.4 · custom prompt

@github-actions

github-actions Bot commented Oct 5, 2026

Copy link
Copy Markdown

🤖 Kimi Code Review

I'll review this PR which introduces a one-pass participation accounting optimization for Altair-and-later epoch processing. This is a significant performance improvement with consensus-critical code.

High-Level Assessment

This is a well-structured optimization that replaces O(n²) epoch processing with O(n) by scanning the validator registry once. The approach of keeping a reference implementation for testing is excellent. However, I've identified several issues ranging from minor correctness concerns to potential consensus bugs.


Critical Issues

1. Missing ? operator in scan error propagation (participation.rs:185-195)

None => {
    previous_error.get_or_insert(Error::IndexOutOfBounds {
        index,
        len: previous_list.len(),
    });
}

Problem: get_or_insert returns &mut Error, but the ; discards it. The error is stored but never propagated. Same issue for current_error at lines 207-213.

Wait—let me re-read. Actually, lines 219-221 do propagate:

if let Some(error) = previous_error.or(current_error) {
    return Err(error);
}

This is correct, though the pattern is slightly unusual. No bug here.


2. apply_rewards_and_penalties balance iteration bug (altair.rs:224-226)

let mut balance_iter = balances.iter();
// ...
let balance = balance_iter.next().copied();

Problem: The balance_iter is advanced for every validator in summary.iter(), but continue on ineligible validators still consumes a balance. This is correct because summary.iter() and balances are both in registry order. However, if balances.len() < summary.len(), the check at line 246 catches it. Actually correct.

But wait—there's a subtle issue: the balance is None for ineligible validators past the end of balances, but the code continues before checking. If balances is shorter, ineligible validators at the end would continue without hitting the length check. The check at line 246 only fires after the loop if balances.len() < summary.len().

Actually re-reading: the loop runs summary.iter().enumerate(), so index goes 0..summary.len()-1. If balances.len() < summary.len(), then for index >= balances.len(), balance is None. For ineligible validators, we continue and never hit the UnknownValidator error. The post-loop check at line 246 catches this: if balances.len() < summary.len() { return Err(...) }. Correct.


3. Real bug: altair_reference module not declared in altair.rs

The file altair_reference.rs is added but altair.rs never declares mod altair_reference;. Looking at the diff, altair.rs uses super::altair_reference::process_participation_steps at line 312, but there's no mod altair_reference; in altair.rs.

Wait—checking mod.rs:

pub mod altair;
#[cfg(any(test, debug_assertions))]
pub mod altair_reference;

So altair_reference is a sibling module, not a child. The path super::altair_reference from altair.rs would go to epoch::altair_reference. But altair.rs is in epoch/, so super is epoch/. Yes, this works. Correct.


4. Real bug: process_participation_steps debug assertion clones state unnecessarily for large registries

#[cfg(debug_assertions)]
{
    const CHECK_LIMIT: usize = 4096;
    if state.validators().len() <= CHECK_LIMIT {
        let mut expected = state.clone();
        // ...
        let result = run_participation_steps(state, config);
        // ...
        return result;
    }
}
run_participation_steps(state, config)

Problem: When result.is_err() in the debug path, the function returns result without comparing states. But when result.is_ok(), it asserts state equality. However, if expected_result.is_ok() and result.is_err(), the first assertion result.is_ok() == expected_result.is_ok() fails. This is correct.

But: the CHECK_LIMIT is checked against state.validators().len(), but the clone happens before checking. Actually no—the check is before the clone. Correct.


5. Real bug: RewardContext::new uses state.fork_name() for inactivity_penalty_quotient

let inactivity_penalty_quotient = preset::retuned::inactivity_penalty_quotient(self.fork);

In the reference implementation (participation_reference.rs:158):

let inactivity_penalty_quotient =
    preset::retuned::inactivity_penalty_quotient(state.fork_name());

These match. Correct.


6. Real bug: get_finality_delay changed from infallible to fallible without updating all callers

Looking at rewards.rs:

if is_in_inactivity_leak(state)? {

This was updated. But are there other callers?

In participation.rs:254:

leaking: super::finality::is_in_inactivity_leak(state)?,

In altair.rs:

let leaking = is_in_inactivity_leak(state)?;

In participation_reference.rs:

let leaking = is_in_inactivity_leak(state)?;

All updated. Correct.

But wait—finality.rs itself has get_eligible_validator_indices which doesn't use get_finality_delay. The is_in_inactivity_leak is only used in the places shown. Correct.


7. Real bug: ValidatorDeltas::apply ordering doesn't match specification

pub fn apply(self, balance: Gwei) -> Gwei {
    self.0.iter().fold(balance, |balance, &(reward, penalty)| {
        balance.saturating_add(reward).saturating_sub(penalty)
    })
}

The comment says "reward first and then penalty" and "the balance floors at zero after each penalty, so a reward arriving after a penalty that emptied the balance still counts."

But the specification's process_rewards_and_penalties in phase0 does:

for index in range(len(state.validators)):
    increase_balance(state, index, rewards[index])
    decrease_balance(state, index, penalties[index])

For Altair, the spec's process_rewards_and_penalties calls get_flag_index_deltas and get_inactivity_penalty_deltas separately, then applies them in order. The order within get_flag_index_deltas is: source, target, head. Then inactivity penalty is separate.

The test at participation_equivalence.rs:551 verifies:

let deltas = ValidatorDeltas([(0, 10), (5, 0), (0, 0), (0, 0)]);
assert_eq!(deltas.apply(3), 5); // 3 - 10 = 0 (floor), 0 + 5 = 5

This matches the comment's intent. But does it match the reference? The reference process_rewards_and_penalties in altair_reference.rs does:

for (rewards, penalties) in deltas {
    for index in 0..validator_count {
        increase_balance(state, index, rewards[index as usize])?;
        decrease_balance(state, index, penalties[index as usize])?;
    }
}

Wait—this applies ALL rewards first, then ALL penalties for each delta pair. But get_flag_index_deltas returns (rewards, penalties) where rewards and penalties are for the SAME flag. So for flag 0 (source), it adds reward then penalty. Then for flag 1 (target), adds reward then penalty. Then flag 2 (head). Then inactivity penalty (only penalties).

So the order is: source reward, source penalty, target reward, target penalty, head reward, head penalty, inactivity penalty.

But ValidatorDeltas stores [(source_r, source_p), (target_r, target_p), (head_r, head_p), (inactivity_r, inactivity_p)] and applies them in that order: source reward, source penalty, target reward, target penalty, etc.

This matches! Correct.


8. Real bug: apply_rewards_and_penalties doesn't use increase_balance/decrease_balance mutators

The reference uses:

increase_balance(state, index, rewards[index as usize])?;
decrease_balance(state, index, penalties[index as usize])?;

But the new code writes directly to balances_mut():

*balances.get_mut(index).ok_or(...)? = updated;

This bypasses the increase_balance/decrease_balance functions. Are these just direct mutations, or do they have side effects?

Looking at mutators.rs (not in diff), increase_balance and decrease_balance likely handle the tree-backed balances correctly. Writing directly to balances_mut() should be equivalent if balances_mut() returns the same type.

But there's a subtlety: increase_balance and decrease_balance might do additional validation or use different error types. The new code uses Error::UnknownValidator while the old code would use whatever increase_balance returns.

More importantly: the new code computes updated = deltas.apply(balance) which applies rewards and penalties in a specific order with saturating arithmetic. The old code applied increase_balance and decrease_balance separately. If decrease_balance does balance.saturating_sub(penalty), then:

  • Old: balance = min(balance + reward, MAX), then balance = balance.saturating_sub(penalty)
  • New: balance = balance.saturating_add(reward).saturating_sub(penalty)

These are the same! saturating_add then saturating_sub is associative in this context. Correct.


9. Real bug: weigh_with_totals uses summary.totals() after potential mutation

In run_participation_steps:

let summary = EpochSummary::build(state, justifies)?;
if justifies {
    weigh_with_totals(state, summary.totals())?;
}
update_inactivity_scores(state, config, &summary)?;
let context = RewardContext::new(state, summary.totals())?;

weigh_with_totals mutates state (justification bits, checkpoints). RewardContext::new then reads state.finalized_checkpoint() which may have been updated. This is intentional and correct—the comment says "the leak flag reads the finalized checkpoint which justification may advance."

But wait: summary.totals() was computed BEFORE weigh_with_totals ran. The totals include total_active_balance, previous_epoch_flags, and current_epoch_target. These don't depend on justification/finality. So using the old totals is correct. Correct.


10. Real bug: scan doesn't handle current_epoch == previous_epoch correctly for current_list

let previous_list = if current_epoch == previous_epoch {
    current_list
} else {
    previous_list
};

At genesis, current_epoch == previous_epoch == GENESIS_EPOCH. The spec says to use current_epoch_participation for both. This aliases current_list to previous_list. But current_list is still used later for current_target:

if current_target && active_now {
    match current_list.get(index) {

This is correct because when current_epoch == previous_epoch, previous_list points to current_list, so both read from the same data. Correct.


11. Minor issue: EpochFlags::participated doesn't validate flag_index in release builds

pub fn participated(self, flag_index: usize) -> bool {
    debug_assert!(flag_index < FLAG_COUNT);
    self.0 & (1 << flag_index) != 0
}

In release builds, an out-of-bounds flag_index will shift by >= 8, which is undefined behavior for u8 shift. Actually, Rust defines 1u8 << 8 as 0 (shift is masked modulo 8 for u8? No—Rust panics in debug, wraps in release for shift >= bit width).

Wait: 1u8 << 8 in release mode is 0 (the shift amount is ANDed with 7 for u8). So self.0 & 0 != 0 is always false. This would silently return false for invalid flag indices rather than erroring. Given this is internal code with controlled call sites, this is acceptable but slightly risky.

Suggestion: Consider assert! instead of debug_assert! or use checked_shl. But not critical.


12. Real bug: participation_reference.rs not gated by #[cfg(any(test, debug_assertions))]

The file participation_reference.rs is in helpers/mod.rs:

#[cfg(any(test, debug_assertions))]
pub mod participation_reference;

But altair_reference.rs is in stf/epoch/mod.rs:

#[cfg(any(test, debug_assertions))]
pub mod altair_reference;

Wait, looking at the diff for mod.rs:

+#[cfg(any(test, debug_assertions))]
+pub mod altair_reference;

And for helpers/mod.rs:

+#[cfg(any(test, debug_assertions))]
+pub mod participation_reference;

Both are correctly gated. Correct.


13. Real bug: get_unslashed_participating_indices in altair.rs changed behavior for non-active validators

Old code:

for index in get_active_validator_indices(state, epoch) {
    // check participation and slashed
}

New code:

for (index, validator) in state.validators().iter().enumerate() {
    if !is_active_validator(validator, epoch) {
        continue;
    }
    // check participation and slashed
}

These are equivalent if get_active_validator_indices returns indices of active validators in order. The comment says it does. Correct.

But: the old code used state.validator(index)? to read the validator, which is a tree descent. The new code iterates directly. This is the optimization. Correct.


14. Real bug: apply_rewards_and_penalties error type changed for short balances

Old code (reference):

increase_balance(state, index, rewards[index as usize])?;

This would fail with whatever error increase_balance returns for out-of-bounds.

New code:

if balances.len() < summary.len() {
    return Err(Error::UnknownValidator(balances.len() as ValidatorIndex));
}
// ...
*balances.get_mut(index).ok_or(Error::UnknownValidator(index as ValidatorIndex))? = updated;

The new code returns UnknownValidator for both cases. Is this the same error as the reference?

Looking at mutators.rs (not in diff), increase_balance likely returns Error::UnknownValidator for out-of-bounds. So this matches. Correct.

But: the old code would fail at the FIRST missing balance during the loop. The new code fails after the loop with balances.len() as the index. This could result in different error messages but same error type. For consensus, only error vs success matters. Acceptable.


15. Real bug: process_epoch for capella and electra changed to call process_participation_steps

Old:

super::altair::process_justification_and_finalization(state)?;
super::altair::process_inactivity_updates(state, config)?;
super::altair::process_rewards_and_penalties(state, config)?;

New:

super::altair::process_participation_steps(state, config)?;

This is correct because process_participation_steps calls the three steps in order. Correct.


16. Performance concern: EpochSummary allocates two Vecs per epoch

let mut flags = Vec::with_capacity(count);
let mut effective_balances = Vec::with_capacity(count);

At mainnet scale (~1M validators), this is ~9MB per epoch (1M * 9 bytes). This is acceptable for epoch processing which already does significant work. The old code allocated multiple index lists and balance vectors, so this is likely a net reduction.


17. Code style: apply_rewards_and_penalties could use zip more idiomatically

let mut balance_iter = balances.iter();
// ...
let balance = balance_iter.next().copied();

Could be:

for ((index, (flags, effective_balance)), &balance) in summary.iter().enumerate().zip(balances.iter()) {

But the current pattern with manual iteration allows the balances.len() < summary.len() check. Acceptable as is.


18. Real bug: scan function stores errors but doesn't short-circuit

let mut previous_error = None;
// ...
None => {
    previous_error.get_or_insert(Error::IndexOutOfBounds { ... });
}

This continues scanning after finding an error. This is intentional to match "the previous epoch's error wins" semantics. But it does unnecessary work. Not a bug, just slightly inefficient. Acceptable.


19. Documentation issue: spec_deviations.md claims "Lighthouse reads the leak once per epoch as well"

This is a nice cross-client consistency note, but I can't verify it. Assuming the author checked. Acceptable.


20. Most serious concern: process_participation_steps debug assertion can panic on valid states

assert_eq!(
    result.is_ok(),
    expected_result.is_ok(),
    "participation steps disagree with the reference: {result:?} vs {expected_result:?}"
);

If result.is_ok() and expected_result.is_err(), this panics in debug builds. But the comment says "release-fast keeps debug assertions, so every fixture checks it."

Wait—release-fast with debug_assertions? That's unusual. Typically debug_assertions is only set in debug builds. If the build profile sets debug_assertions = true in release, then this runs in production!

Actually, looking at the code:

#[cfg(debug_assertions)]
{
    // ...
}

If debug_assertions is enabled in a "release-fast" profile, then this code runs in production. The assert_eq! could panic if there's a bug, which is good for testing but bad for production.

But the PR author says "release-fast keeps debug assertions, so every fixture


Automated review by Kimi (Moonshot AI) · kimi-k2.6 · custom prompt

Base automatically changed from beacon-chain-integration to main October 20, 2026 16:19
Justification, inactivity updates and rewards read the tree-backed registry
through ~19 full scans and ~21 validator(i) descents per active validator,
and the per-block pulled-up tip repeated step 1 on every imported block.

helpers::participation walks validators().iter() once, zipped with the flat
participation slices, into per-validator flags, effective balances and the
balance totals (EpochSummary). Steps 1-3 then run over flat data: scores are
updated in place, and only balances that changed are written back, per
component in spec order. Justification (and so the pulled-up tip) needs only
ParticipationTotals, one scan with no allocation.

The public step functions and the flag/inactivity delta helpers keep their
signatures for the fixtures and become thin wrappers. Effective balance
updates zip validators with balances, since balances()[i] is now a tree
descent for most validators.

The specification-shaped code stays as a reference oracle for tests and debug
builds: the driver runs it on a clone (registries up to 4096) and asserts
balances, scores, justification bits and checkpoints agree; randomized tests
compare every value and error across forks.
The participation equivalence generator could finalize the current epoch,
which no reachable state does, so get_finality_delay underflowed. The suite
runs under release-fast, where overflow checks are off and both the fast
path and the reference wrapped identically; the minimal-preset CI job runs a
dev build and panicked.
get_finality_delay subtracted the finalized epoch from the previous epoch
unchecked, so a finalized checkpoint past the previous epoch panicked in
debug builds and wrapped into a permanent inactivity leak in release. The
specification treats a uint64 underflow as an invalid state transition, and
Lighthouse and Grandine reject it the same way, so the delay and the leak
check now return ArithmeticOverflow, as the inactivity penalty's own
checked product already does. No chain reaches that state, since
justification only finalizes an epoch behind the current one.
…iation

Now that an underflowing finality delay is an error rather than a wrap, the
point where the leak flag is read decides which states fail. The altair
fast path reads it once per step where the specification reads it per
validator, so a crafted state with the finalized epoch past the previous
one fails here and can pass there.
@MegaRedHand
MegaRedHand force-pushed the perf/beacon-epoch-participation-b branch from e667902 to 05091ae Compare October 20, 2026 17:32
MegaRedHand added a commit that referenced this pull request Oct 21, 2026
…#634)

Squash of feat/beacon-gloas at 1d667c4 (merge-base 35c074b).

Adapted onto #626-#633 and main (#642-#677):
- #632/#633's epoch single pass ported to the element accessors; fulu runs
  through the single pass, with the eth1-bridge gate on electra only.
- DB_VERSION 6 (gloas Config 5 + #628's table).
- compute_node_weights skips the boost outside the justified subtree
  (boost_reaches_the_descent), for #627's LiveChain prune; the
  boosted-finalized-block bound test is dropped and the differential tests
  skip pruned roots.
- #628's set_unrealized_justification slot argument in the tests;
  get_forkchoice_store keeps the anchor block's slot.
- #653's validator-by-id handler on iter_validators/balance.
MegaRedHand added a commit that referenced this pull request Oct 21, 2026
…cursor (#647)

Squash of perf/ssz-tree-write-cursor at 6fe1dc6 (merge-base 35c074b).

Adapted onto #632-#634: the write cursor also drives gloas's progressive
registry (try_update_validators_with_balances on BeaconState), and the
single-pass epoch steps write through it.
zexoverz pushed a commit to zexoverz/ethlambda that referenced this pull request Oct 22, 2026
## 🗒️ Description / Motivation

Merges `beacon-chain-integration` into `main`. This adds an **Ethereum
Beacon Chain** client (phase0 through fulu) to the ethlambda binary,
alongside the Lean consensus client:

- **`ethlambda beacon`**: a beacon node. It checkpoint-syncs, follows
the chain over gossip and req/resp, validates execution payloads through
an execution client's Engine API, and serves the Beacon API, including
the validator endpoints.
- **`ethlambda validator`**: a beacon validator client (attest,
aggregate, propose) that talks to any beacon node over the standard REST
API.
- **`ethlambda node`** (lean) keeps its behavior, apart from the items
under "Behavior changes for lean" below.

Both chains run on the same storage layer, P2P actor, RPC crate and
chain actor. `BeaconState` has a `Lean` variant holding lean's `State`,
and each shared handler dispatches on it once, at the top.

Each piece was reviewed as its own PR into the integration branch
(listed under "Related Issues / PRs"). This PR merges that branch.

## What Changed

| Area | Change |
|---|---|
| `ethlambda-types` (`beacon/`) | Containers for phase0 to fulu, presets
(compile-time `preset-minimal` feature), runtime fork config, and a
`Lean` variant on `ForkName` and `BeaconState` |
| `ethlambda-state-transition` (`beacon/`) | State transition, epoch
processing, fork upgrades, fork choice, BLS (`blst`), KZG (`c-kzg`),
gossip validation rules, DAS custody math |
| `ethlambda-ssz-tree` (**new**) | Persistent Merkle `List`/`Vector` for
the validator registry and balances, so consecutive states share
unchanged subtrees |
| `ethlambda-storage` | One `Store` for both chains. Beacon states are
stored as snapshots plus VCDIFF deltas and written on a background
thread. Adds a committee cache, and `db_version`/`chain`/`preset`
metadata that resume checks |
| `ethlambda-blockchain` | Beacon path in the chain actor
(`BlockChain::spawn_beacon`): import, holding blocks until their parent
or columns arrive, applying gossip aggregates to fork choice, block
production and aggregation |
| `ethlambda-engine` (**new**) | Engine API client with JWT auth:
`newPayload`, `forkchoiceUpdated`, `getPayload`, `getBlobs` |
| `ethlambda-p2p` | Beacon wire: blocks, aggregates, subnet attestations
and data columns are validated before relay;
`beacon_blocks_by_{range,root}/2` served and requested; range sync;
column custody and sampling; attestation subnet backbone; discv5 always
on for `beacon` |
| `ethlambda-rpc` | Beacon API under `/eth/v1` and `/eth/v2`: JSON by
default, SSZ on request. The router is picked by `Store::chain()`, so
`/lean/v0` and the Beacon API are never served together |
| `ethlambda-validator` (**new**) | Validator client: duties,
attestations, aggregation selection, proposals, keymanager API. It has
**no slashing protection** (recorded in `docs/spec_deviations.md`) |
| `bin/ethlambda` | `beacon` and `validator` sub-commands; `--network`
takes a built-in name (`mainnet`, `sepolia`, `hoodi`) or a directory of
published network files; beacon checkpoint sync; `benchmark import`
replays a real block corpus offline; `run_node` is one startup path for
both chains |
| CI / Makefile | One beacon spec-test job per preset, minimal-preset
check, `make test` split into `test-consensus` and `test-node` (one
runner's disk no longer fits both), benchmark-smoke and tooling jobs,
disk-cleanup action |
| Docs | New: `beacon_stf.md`, `beacon_wire.md`, `beacon_engine.md`,
`cli.md`. Updated: `data_storage.md`, `metrics.md`, `rpc.md`,
`checkpoint_sync.md`, `discovery.md`, `benchmarking.md`,
`spec_deviations.md`, README (beacon quickstart with ethrex) |

## Correctness / Behavior Guarantees

- **Separation:** beacon behavior lives in `beacon` modules that lean
code never reads, and those modules never read lean's. The shared crates
dispatch on the chain once per handler.
- **Lean state at a beacon accessor:** this panics, naming the function
it reached (`lean_state_unreachable` / `lean_fork_unreachable`), so the
caller never gets a wrong answer back. `ForkName::Lean` is left out of
`ForkName::ALL`, so neither `parse` nor a fork upgrade can reach it.
- **Spec conformance:** every consensus-specs `v1.6.1` fixture case
passes on both presets: 5705 mainnet and 40009 minimal, including 150
mainnet `fork_choice` cases. Gossip fixtures are pinned to
`v1.7.0-beta.1`.
- **Resume safety:** a data directory written by another chain, preset
or format version is refused at startup, before any state is read.

### ⚠️ Behavior changes for lean (`ethlambda node`)

- **Existing data directories are refused on resume.** A directory
written by `main` has no `db_version` key, so `Store::from_db_state`
returns `DbVersionMismatch { found: 0, expected: 4 }`. There is no
migration: an upgraded node needs a fresh data directory (or checkpoint
sync).
- `blst` and `c-kzg` are now dependencies of the lean binary too, since
the state-transition crate holds both chains' rules. They are not
feature-gated.
- libssz is a git dependency pinned by `rev`: the beacon containers need
`DerefMut`/`IndexMut` on `SszList` (lambdaclass/libssz#33), which no
crates.io release has yet.
- The `release-fast` profile (used for tests) now sets `debug-assertions
= true`.
- `P2P::spawn` now runs before the chain actor is spawned, so gossip
arriving in that short window is dropped.
- `/lean/v0/blocks/finalized` answers JSON when asked for it by name.
SSZ stays the default.

## Tests Added / Run

- **Beacon spec tests:** `make test-beacon` (or `test-beacon-mainnet` /
`test-beacon-minimal`). Each fixture case is a separately named test,
e.g. `cargo test -p ethlambda-state-transition --test beacon_spec_tests
--features beacon-spec-tests -- electra/attester_slashing`. CI runs one
job per preset.
- **`make test`** still needs no consensus-spec download: the beacon
spec target is behind the `beacon-spec-tests` feature, and the BLS/KZG
vector tests report as ignored without it.
- **Minimal preset:** the `Test minimal preset` CI job builds and runs
the `ethlambda-types` unit tests with `preset-minimal` on.
- **Benchmark smoke:** a CI job runs the `benchmark` sub-command end to
end.
- Unit and integration tests in every new crate (`ssz-tree`, `engine`,
`validator`) and in the beacon paths of storage, p2p, rpc and the chain
actor.

## Related Issues / PRs

<details>
<summary>PRs merged into the integration branch (reviewed on
<code>lambdaclass/ethlambda_private</code>)</summary>

**Types and state transition**
- lambdaclass/ethlambda_private#1 Beacon Chain containers under a
`beacon` namespace
- lambdaclass/ethlambda_private#2 Beacon Chain state transition, phase0
through fulu
- lambdaclass/ethlambda_private#37 keep the registry and balances in
persistent Merkle trees

**Follower, sync and storage**
- lambdaclass/ethlambda_private#3 follow mainnet gossip behind an
`ethlambda beacon` sub-command
- lambdaclass/ethlambda_private#6 unify the lean and beacon storage
layer
- lambdaclass/ethlambda_private#7 checkpoint sync for the beacon chain
- lambdaclass/ethlambda_private#8 serve beacon blocks by range and by
root
- lambdaclass/ethlambda_private#9 give the follower a chain, from the
anchor to the tip
- lambdaclass/ethlambda_private#11 fulu data availability sampling,
custody through the availability gate
- lambdaclass/ethlambda_private#13 validate execution payloads over the
Engine API
- lambdaclass/ethlambda_private#15 follow a network described by files
on disk
- lambdaclass/ethlambda_private#26 add sepolia and hoodi as built-in
networks
- lambdaclass/ethlambda_private#48, lambdaclass/ethlambda_private#52,
lambdaclass/ethlambda_private#53, lambdaclass/ethlambda_private#54: held
blocks, their columns and range batches

**P2P**
- lambdaclass/ethlambda_private#18 stop outbound req/resp requests
crossing protocols
- lambdaclass/ethlambda_private#23 rank dial candidates by custody
redundancy
- lambdaclass/ethlambda_private#28 count why peers drop us
- lambdaclass/ethlambda_private#38 validate beacon gossip before
propagating it
- lambdaclass/ethlambda_private#43 run every data column check in p2p
- lambdaclass/ethlambda_private#47 validate beacon aggregates and subnet
attestations before relaying
- lambdaclass/ethlambda_private#19 apply gossip aggregates to fork
choice, backbone two attestation subnets

**API and validator**
- lambdaclass/ethlambda_private#21 serve the Beacon API from `ethlambda
beacon`
- lambdaclass/ethlambda_private#44 report the preset and constants from
`/eth/v1/config/spec`
- lambdaclass/ethlambda_private#16 beacon-chain validator client
- lambdaclass/ethlambda_private#34 serve the validator API
- lambdaclass/ethlambda_private#40 aggregate and propose through
`ethlambda beacon`
- lambdaclass/ethlambda_private#56 pack other nodes' attestations into
produced blocks

**Performance and metrics**
- lambdaclass/ethlambda_private#10, lambdaclass/ethlambda_private#20,
lambdaclass/ethlambda_private#22, lambdaclass/ethlambda_private#24,
lambdaclass/ethlambda_private#33, lambdaclass/ethlambda_private#49:
registry scans, parallel pubkey validation, committee cache, background
state writes, BLS key validation
- lambdaclass/ethlambda_private#17, lambdaclass/ethlambda_private#31:
import timing by phase
- lambdaclass/ethlambda_private#25 benchmark block import by replaying a
real corpus

**CI**
- lambdaclass/ethlambda_private#12, lambdaclass/ethlambda_private#30,
lambdaclass/ethlambda_private#45: split lint/test jobs, benchmark smoke
job, runner selection

</details>

Open PRs based on `beacon-chain-integration`, **not** included here:
lambdaclass#626, lambdaclass#627, lambdaclass#629, lambdaclass#630, lambdaclass#632, lambdaclass#634, lambdaclass#636, lambdaclass#638, lambdaclass#642, lambdaclass#644 to lambdaclass#652,
lambdaclass#656, lambdaclass#658.

## ✅ Verification Checklist

- [ ] Ran `make fmt` (clean)
- [ ] Ran `make lint` (clippy with `-D warnings`, clean)
- [ ] Ran `make test` (`test-consensus` plus `test-node`, at
`release-fast`), all passing
- [ ] Ran `make test-beacon` (both presets), all passing

---------

Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>
Co-authored-by: Klaus Lungwitz <klaus.lungwitz@lambdaclass.com>
Co-authored-by: Klaus @ LambdaClass <18153834+klaus993@users.noreply.github.com>
Co-authored-by: Chase Wright <chase.wright@ethereum.org>
@MegaRedHand
MegaRedHand added this pull request to stack #689 October 22, 2026 21:45
@MegaRedHand
MegaRedHand added this pull request to the merge queue Oct 22, 2026
Merged via the queue into main with commit ddd6fcb Oct 22, 2026
12 checks passed
@MegaRedHand
MegaRedHand deleted the perf/beacon-epoch-participation-b branch October 22, 2026 22:33
MegaRedHand added a commit that referenced this pull request Oct 27, 2026
…#650)

## Motivation

Now that `validators` and `balances` are persistent trees, every cached
state still holds a private deep copy of the other large fields and
rehashes them in full on each state root.

| Field(s) | Mainnet size per state | Flat cost today |
|---|---|---|
| `inactivity_scores` | ~18 MiB | copied per state, rehashed per root,
rewritten for every eligible index each epoch |
| `randao_mixes` | ~2 MiB | same |
| `block_roots`, `state_roots`, `slashings` | ~0.6 MiB | same |
| `eth1_data_votes`, `historical_roots`, `historical_summaries` | ~0.25
MiB | same |

Estimates, not measurements: ~26 MiB of flat fields per state and ~0.85M
Merkle nodes rehashed per root, of which this PR moves the large
majority of the nodes (participation lists and queues stay flat). The
import-replay benchmark was not run for this PR, and no memory or
node-count number was measured; the estimates come from the field sizes
above.

## Changes

| Commit | What |
|---|---|
| `perf(ssz-tree): flush tree fields through one Buffered list per fork`
| Object-safe `Buffered` trait (impls for `List` and `Vector`);
`tree_fields!` lists each fork's tree fields once and generates both
`apply_pending_mutations` and `has_pending_mutations`. `H256` writes
`HashTreeRoot` by hand and forwards `is_basic_type` (as `H160` does).
New ssz-tree properties: multi-leaf `u8` lists, `u64` and
32-byte-newtype vectors, decode parity, rebase of mostly-zero lists and
vectors. |
| `perf(beacon): keep inactivity scores on the tree and skip no-op
writes` | `InactivityScores` is a tree list (`BTreeMap` update map).
`process_inactivity_updates` computes the same value with the same
checked arithmetic and writes only `if new != score`. Joins the flush
list and `BeaconState::rebase_on`. |
| `perf(beacon): keep the roots and randao ring buffers on the tree` |
`BlockRoots`, `StateRoots`, `RandaoMixes`, `Slashings` become tree
vectors; `Eth1DataVotes`, `HistoricalRoots`, `HistoricalSummaries`
become tree lists (alias swap). All join the flush list and `rebase_on`.
Block production now flushes before it hashes the post-state. Test-only
sites that sliced or iterated in place use indexing. |

SSZ bytes, roots and the storage delta format are unchanged.

## Design decisions

- **`H256` change done locally**, not in libssz's `transparent` derive
(no dependency bump). The lean chain shares `H256`, so `ethlambda-types`
has unit tests that an `SszList`/`SszVector` of `H256` hashes like one
of `[u8; 32]`, and that a container holding `H256` collections keeps its
root. The lean state's pinned genesis root test passes unchanged.
- **`BTreeMap` update map for inactivity scores.** Outside a leak almost
no score changes, so writes are sparse. A leak writes nearly every
score, which costs about one B-tree insert per validator per epoch (~1M
at the benchmark's registry size) until a bulk write path exists. A
dense map (`VecMap`) was the alternative; it allocates 16 B per
validator on the first push or write.
- **Skip loop written by hand** in `process_inactivity_updates`: read
the score, compute the new one exactly as before (overflow still raises
`ArithmeticOverflow` before the recovery step), write only if different.
A write rebuilds its leaf even for an equal value, which would unshare
the list from the parent state's.
- **`rebase_on` pairs scores through the altair accessors**, so it works
across forks and skips phase0 (no scores).
- **`process_slot` leaves its two roots writes buffered** until the next
flush (the next slot's, or `process_slots`' last step).
`process_historical_summaries_update` hashes the roots vectors with that
write pending: correct, and the vectors are small. Tests pin this.
- **Block production flush:** `produce_block` hashed the post-state
without `apply_pending_mutations`, so the computed hashes were
discarded; one added call.
- `Buffered` and `tree_fields!` are additive in ssz-tree and
`containers/mod.rs`, to keep a merge with other work in the same files
easy.

## Not in this PR

| Item | Why |
|---|---|
| Participation lists on the tree
(`previous_`/`current_epoch_participation`) | Blocked on a zip pass for
`get_unslashed_participating_indices` (#632): its per-index reads would
cost ~2M tree descents per block in the pulled-up tip. Also needs
`List::filled` for an O(1) rotation. |
| `pending_deposits` on the tree | Depends on the mainnet queue length,
which has not been measured. |
| Sync committees (`Arc` + cached-root newtype) | Small win (~1.5 MiB
per state), separate design. |
| Rebase across an epoch boundary (decoded `previous` equals the base's
`current`) | Positional rebase misses it; one `if` in `rebase_on`, left
for the participation PR. |

## Testing

| Command | Result |
|---|---|
| `cargo test -p ethlambda-ssz-tree --profile release-fast` | 35 passed
(properties), 0 failed |
| `cargo test -p ethlambda-types --profile release-fast --lib` | 201
passed |
| `cargo test -p ethlambda-storage --profile release-fast --lib` | 139
passed, 1 ignored |
| `cargo test -p ethlambda-state-transition --profile release-fast
--lib` | 329 passed, 16 ignored |
| `cargo test -p ethlambda-rpc --profile release-fast --lib` | 110
passed |
| Beacon spec suite, mainnet preset (full) | 5734 passed, 152 ignored; 4
failed |
| Beacon spec suite, minimal preset (full) | 40038 passed, 3692 ignored;
4 failed |
| `cargo clippy` (workspace, and state-transition with
`beacon-spec-tests`) with `-D warnings`, `cargo fmt --check`, `cargo
check -p ethlambda` | clean |

The 4 failures in each spec run are the `gossip/*` trials, which report
"matched no fixture cases": the gossip fixture directory is
intentionally empty in the local setup. Every `ssz_static`, `sanity`,
`finality`, `random`, `epoch_processing` and `transition` trial passes.
The lean `ssz_spectests` and the import-replay benchmark were not run.

---------

Co-authored-by: Pablo Deymonnaz <pdeymon@fi.uba.ar>
@MegaRedHand MegaRedHand added the performance Performance improvements or possible performance improvements label Nov 3, 2026
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
…mbdaclass#633)

One of three alternative implementations of the same follow-up to the
persistent-tree registry (epoch participation without per-index registry
reads): (a) lambdaclass#630, (b) lambdaclass#632, (c) this PR. **Stacked on (b)**: this PR's
diff is the fusion only, and it targets (b)'s branch.

## Motivation

After (b), electra and fulu still walk the tree-backed registry once per
remaining epoch step: registry updates, slashings, pending deposits and
effective-balance updates. Several of those steps also recompute
`get_total_active_balance`, which descends once per active validator.
This is option (c) of the plan, the Lighthouse-style single pass
(`process_epoch_single_pass`).

## Change: fuse electra/fulu epoch steps 2-9 into one loop

`stf::epoch::single_pass` visits each validator once, in index order,
working on a local copy of the validator and its balance. Per validator,
in spec order:

1. inactivity score;
2. rewards and penalties;
3. registry action (queue, eject or activate);
4. slashing penalty;
5. planned deposit top-up;
6. effective-balance update.

Changes are written back after the loop. What depends on more than one
validator stays outside it, as in Lighthouse:

- **Exit churn:** the cursor is advanced locally in index order and
written back once.
- **Pending deposits:** the queue is planned before the loop. A deposit
counts as "exited" if the validator already has an exit or this epoch's
registry update will eject it. Top-ups of existing validators apply
inside the loop; deposits that create validators apply after it, and one
registry walk serves all pubkey lookups.
- **Consolidations:** pending consolidations run after the loop. The
effective-balance update of every validator they name, and of every
validator created by deposits, waits until then.
- **Total active balance:** taken from the summary and reused for
slashings and every churn limit. It equals `get_total_active_balance`
through step 8.

The hysteresis test both effective-balance updates use is now one
checked `leaves_hysteresis_band`: a balance near `u64::MAX` is an
`ArithmeticOverflow`, as in the spec and Lighthouse's `safe_add`, rather
than a debug panic or a release wrap that moves the effective balance.
It keeps the spec's short-circuit `or`.

The genesis epoch, and states whose registry-sized lists differ in
length, fall back to the step-by-step path. The step functions stay
public and callable in isolation for the fixtures. The per-step rules
both paths share were factored out so both run the same code: registry
action, slashing penalty, effective-balance hysteresis, inactivity score
and the exit-cursor advance.

## Testing

- In debug builds, a registry of up to 4096 validators also runs through
the specification-shaped steps 1-9 on a clone, and the two full
post-states must have equal `hash_tree_root`. A mismatch names the
differing fields.
- A randomized fused-vs-unfused test runs 800 electra and fulu cases.
They cover ejections that move the churn cursor, activations and the
activation queue, slashings, every pending-deposit branch (including new
pubkeys with valid and invalid signatures), and processed, slashed and
not-yet-withdrawable consolidations. Two deliberately injected bugs were
caught within a few seeds. About one state in ten carries a
near-`u64::MAX` balance, since one now fails the whole epoch; those
cases check that both paths fail together. As in (b), the finalized
epoch stays at or behind the previous epoch.
- Clippy `-D warnings` and the `ethlambda-state-transition` lib tests
pass, including the dev-profile `Test minimal preset` run, where
overflow checks are on.
- Beacon spec filters `epoch_processing`, `rewards`, `sanity`,
`finality`, `random`, `fork_choice` and `transition` pass on both
presets. That run predates the rebase onto (b)'s finality-delay fix and
the checked hysteresis; CI's beacon spec jobs cover both.

Over (b), this saves about 200 ms per epoch-start block and nothing on
other blocks. That is more than the plan estimated, probably because the
fusion also removes the repeated total-active-balance calls and the
per-deposit pubkey scans (not profiled). A per-epoch total cache
(follow-up 01) and a pubkey-to-index map would recover part of that gain
more simply.

## Benchmark (all three options)

`ethlambda benchmark import replay` on ethlambda-5: `cargo build
--release`, mainnet corpus of 128 blocks (slots 15279073..=15279200, 4
epoch-start blocks), legs interleaved `base a b c` for two rounds. Base
against base: 0.996. Every leg imported all 128 blocks.

| | base (c913728) | (a) iterator passes | (b) summary pass | (c)
single pass |
|---|---|---|---|---|
| process wall | 104.0 s | 96.1 s | 92.0 s | 91.0 s |
| user CPU | 215.1 s | 208.4 s | 205.1 s | 203.8 s |
| `stf`, epoch-start block (mean of 4) | 1503 ms | 1126 ms | 912 ms |
709 ms |
| `stf` p50, other blocks | 577 ms | 517 ms | 478 ms | 479 ms |
| paired `stf` vs base (ratio of totals) | 1 | 0.869 | 0.797 | 0.785 |
| `get_head` p50 | 201 ms | 214 ms | 226 ms | 225 ms |
| peak RSS | 3.88 GiB | 3.93 GiB | 4.04 GiB | 4.04 GiB |

- The per-block saving comes from the pulled-up tip, which reruns
justification on every imported block.
- (c) gains nothing over (b) on ordinary blocks, since their per-block
path is the same.
- `get_head` rises as `stf` falls, although no option touches the head
computation. This is probably contention with the background state
writer, as seen in earlier import benchmarks; it is not verified. Block
wall time still falls.

## Base

- Stacked on lambdaclass#632; its head, which carries `beacon-chain-integration` @
`c79fabd5`, is merged into the branch.
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
…cursor (lambdaclass#647)

## Motivation

An epoch boundary writes every balance. `get_mut` on a tree list buffers
a copy of each element in a dense map, sorts the map into a second
`Vec`, then rebuilds every touched leaf and recomputes the root of every
written element, changed or not. The altair rewards step calls it up to
eight times per validator (four reward/penalty pairs, each as increase
then decrease), and `process_effective_balance_updates` reads
`balances()[i]` per validator, which was only cheap because the rewards
pass had just filled the map.

This PR adds an in-order lazy copy-on-write write cursor to
`ethlambda-ssz-tree` and moves the epoch balance writes onto it.
Expected gain (estimate, not measured on a real chain): tens of ms off
each epoch-start block at mainnet registry size. The import-replay
benchmark was not run for this PR.

## Changes

| commit | what |
|---|---|
| `perf(ssz-tree): add an in-order copy-on-write write cursor` |
`List`/`Vector::iter_cow`, `IterCow::next_cow`, `ElemCow::{index,
make_mut, set}`, `try_update_each`; `with_updated_leaves` generalized
over a leaf source; unit tests and model-based property tests |
| `perf(beacon): apply epoch balance and effective-balance writes
through the cursor` | `BeaconState::registry_mut` (disjoint borrows of
validators and balances); `apply_balance_deltas` used by the altair+ and
phase0 `process_rewards_and_penalties`; both
`process_effective_balance_updates`; new `tree_bench` rows; docs |

How the cursor works: it walks leaves in order and copies a leaf on the
first write that reaches it. At the leaf's end it compares only the
touched offsets with the original; if all are equal, the original leaf
(its `Arc` and cached hash) is kept. Otherwise the new leaf carries over
the element roots of unchanged elements and computes the changed ones.
On drop, one descent swaps the changed leaves in and rebuilds only their
paths. Pending buffered writes are applied before the pass starts.

## Design decisions

- `Drop` does the work, and the tree is untouched during the pass, so an
early return or a panic leaves a consistent list holding the writes so
far (tested with `catch_unwind`).
- Names as in the design: `iter_cow`, `next_cow`, `make_mut`, `set`,
`try_update_each`. No `set_with`.
- Changed composite elements get their root computed eagerly, on the
state-transition thread.
- A kept `Arc` or carried root always follows an actual comparison. The
cursor only creates fresh empty hash cells, or carried roots for a leaf
it owns; it never fills a shared `OnceLock`.
- Spec order for balances: per index, each pair is
`saturating_add(reward)` then `saturating_sub(penalty)`, pairs in order.
Deltas are never netted. Each balance depends only on its own deltas, so
index-by-index equals the spec's pair-by-pair loop.
- `apply_balance_deltas` lives in `helpers/mutators.rs` and is shared by
the phase0 and altair+ paths. It keeps the old error for a balances list
shorter than the registry, and does not touch balances past the registry
length.
- Rewards and both effective-balance passes move together: once rewards
bypass the map, the effective-balance reads would otherwise become tree
descents.
- Effective-balance updates only copy a leaf when the new effective
balance differs from the old one (it can equal it when capped at the
maximum).
- No leaf-slice API, no parallel bulk update.

## Not in this PR

- Switching `Balances` to a sparse `BTreeMap` update map: to be decided
after measuring whether a dense writer remains.
- `List::filled`: its only consumer is the participation-list work,
which is deferred.
- Migrating slashings, inactivity, and registry updates: sparse, flat
lists, or part of the epoch single-pass rewrite (lambdaclass#630, lambdaclass#632, lambdaclass#633).
Expect a merge conflict with lambdaclass#632 in `process_rewards_and_penalties`.
- Import-replay benchmark and any remote profiling.

## Testing

| command | result |
|---|---|
| `cargo test -p ethlambda-ssz-tree --profile release-fast` | 71 unit +
31 model tests passed (before the second commit) |
| `cargo test -p ethlambda-types --profile release-fast --lib` | 197
passed |
| `cargo test -p ethlambda-state-transition --profile release-fast
--lib` | 324 passed, 16 ignored |
| beacon spec suite, mainnet preset, full | 5734 passed, 4 failed |
| beacon spec suite, minimal preset, full | 40038 passed, 4 failed |
| `cargo clippy --locked --workspace --all-targets --profile
release-fast -- -D warnings` and the state-transition
`beacon-spec-tests` variant | clean |
| `cargo check -p ethlambda --profile release-fast` | ok |

The 4 spec failures in each preset are the `gossip/*` suites
(`gossip_beacon_block`, `gossip_data_column_sidecar`,
`gossip_beacon_aggregate_and_proof`, `gossip_beacon_attestation`): the
local gossip fixture directory is intentionally empty, so they match no
cases. Every other suite, including `epoch_processing`, `rewards`,
`sanity`, `finality`, `random` and `transition`, passes.

### tree_bench (2.4M validators, `--ignored`, single run)

Noisy: run while the machine was under a load average above 100 from
other builds, so absolute numbers (and even the order of the old rows)
are unreliable; read them as rough ratios only.

| row | ms |
|---|---|
| `epoch_sweep_tree` (old path) | 333.8 |
| `epoch_sweep_tree_phases` | write 58.1, apply 70.0, hash 83.1 |
| `epoch_sweep_cursor_tree` | 26.8 |
| `epoch_sweep_noop_cursor_tree` | 90.4 |
| `epoch_sweep_rebuild_tree` (rebuild baseline) | 105.8 |
| `rewards_apply_4_pairs_get_mut` | 342.5 |
| `rewards_apply_4_pairs_cursor` | 158.9 |
| `eb_updates_indexed` | 253.8 |
| `eb_updates_cursor` | 115.4 |
| `lockstep_three_lists_cursor` | 136.2 |

The bench asserts the cursor rows produce the same roots as the old
paths, and that a no-op sweep keeps the tree (`ptr_eq`).
dicethedev pushed a commit to dicethedev/ethlambda that referenced this pull request Nov 4, 2026
## Motivation

`get_total_active_balance` rescans the registry about ten times per
ordinary block: once per attestation (through
`get_base_reward_per_increment`), twice in the sync aggregate, plus
churn and fork-choice callers. Each call built an index `Vec` of the
active validators and then read every index back with a `validator(i)`
tree descent. In the profile taken for the persistent-tree registry
change, this was the largest cost left after hashing (the plan's
estimate: about 414 ms per block at mainnet scale; an estimate, not
measured here).

## Changes

| Commit | Change |
|---|---|
| `perf(beacon): sum the total active balance in one registry pass` |
`get_total_active_balance` is one in-order `validators().iter()` pass
with the same `saturating_add` and one-increment floor as
`get_total_balance`. No index `Vec`, no descents. Every caller gets it,
cached or not. Randomized test (inline SplitMix64) against
`get_total_balance(get_active_validator_indices(..))`, plus an all-zero
registry. |
| `perf(beacon): cache the total active balance per epoch in the store`
| New `ActiveBalanceCache` (`ethlambda-storage`), held by the `Store`
beside the committee cache (`Store::active_balance_cache()`). Key
derivation and lookup in `ActiveBalanceCacheExt`
(`helpers/accessors.rs`). Attestation processing (altair, deneb,
electra) and `process_sync_aggregate` take it; plumbed up through block
processing and `state_transition`. New metric
`lean_beacon_total_active_balance_lookups_total{result}`. Docs in
`beacon_stf.md` and `metrics.md`. |
| `perf(beacon): read each attester's validator once per attestation` |
The effective balance is read once per attester, lazily on the first
newly earned flag, instead of once per flag. |

## Design decisions

- **External cache, not a state field.** The design review chose a
store-held table like `CommitteeCache` over a non-SSZ cache field on
`BeaconState`. That keeps the state type, its SSZ derive and its fork
upgrades untouched: no libssz change, no `rebase_on` adoption, no reset
at upgrades.
- **Why the key is sound.** The total for a state in epoch E is fixed by
two inputs. Effective balances are written only by epoch E-1's
`process_effective_balance_updates`; the Electra upgrade zeroes only
never-activated validators, which are not active at E. The active set at
E cannot change inside E, because every activation and exit lands at
least `MAX_SEED_LOOKAHEAD` epochs ahead; deposits append validators that
are not yet active; slashing sets `slashed` but leaves effective
balances alone (the spec's total keeps slashed validators). So `(E,
block root at the last slot of E-1)` identifies the history: two states
agreeing on it agree on the total. Sibling branches that diverge during
E-1 have different roots, so they get different entries.
- **Decision root differs from the shuffling key.** The shuffling key
uses the last slot of E-2, since the seed is fixed ahead of use; the
total is not, so it needs the later root. A new key type
(`ActiveBalanceKey`) keeps the two apart.
- **Epoch 0** keys on the genesis block root (slot 0), like the
shuffling key does for its first epochs. A state that cannot name the
root (the genesis state itself, or a root outside the history window) is
`unkeyable`: computed, not cached.
- **Only block processing uses the cache.** Between
`process_effective_balance_updates` and the slot increment a state is in
epoch E-1 but already carries E's balances, so a key rooted at the end
of E-2 would be wrong there. Block processing never runs in that window
(`process_slots` has already advanced the state), and epoch processing
keeps the uncached one-pass function. Documented on
`active_balance_key`.
- **Debug cross-check.** Every hit is compared against a fresh
computation under `debug_assert_eq!`, so a missed writer fails the
fixtures (`release-fast` keeps debug assertions on).
- **No single-flight.** A concurrent double compute on a miss is
accepted: the value is one cheap registry pass and both callers compute
the same number.
- **`on_block` reads the store's cache itself**
(`store.active_balance_cache()`), so its signature and the chain actor
are unchanged. Other callers (spec runners, block production, unit
tests) pass `ActiveBalanceCache::default()`.
- **Capacity** is a small constant with eviction of the lowest epoch;
entries are tiny, so room for several forks is cheap.
- **Per-attester read stays lazy** so that an attester earning no flag
is still never read, which keeps error behavior identical.

## Not in this PR

- Fork choice's `get_proposer_score` and `calculate_committee_fraction`
keep calling the uncached function; another change replaces that total
with a justified-balances snapshot.
- Epoch processing and the pulled-up tip: they compute totals inside the
single-pass epoch summary work (lambdaclass#630, lambdaclass#632, lambdaclass#633).
- Churn helpers (exits, consolidations, slashings): rare per block, they
use the one-pass function.
- A flat effective-balance epoch cache (progressive balances): waits
until something consumes it.

## Testing

Measured on release-fast with debug assertions. The import-replay
benchmark was not run for this PR, so there is no measured speedup to
report.

| Command | Result |
|---|---|
| `cargo clippy --locked --workspace --all-targets --profile
release-fast -- -D warnings` | clean |
| `cargo clippy -p ethlambda-state-transition ... --features
beacon-spec-tests -- -D warnings` | clean |
| `cargo test -p ethlambda-storage --lib` | 142 passed, 1 ignored (run
before the store-sharing test was added; not re-run after) |
| `cargo test -p ethlambda-state-transition --lib` | 330 passed, 16
ignored |
| beacon spec suite, mainnet (full) | 5734 passed, 152 ignored, 4 failed
|
| beacon spec suite, minimal (full) | 40038 passed, 3692 ignored, 4
failed |
| `cargo check -p ethlambda --profile release-fast` | clean |

The 4 failures per preset are the gossip `matched_fixture_cases` checks:
the gossip fixture tree is empty in this environment, so they match no
cases. Every other suite passes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

beacon Ethereum Beacon Chain client performance Performance improvements or possible performance improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants