Skip to content

[Security][tentative] Medium (attackable): Withdrawal Overflow Backlog Bypasses The Epoch Cap's Work Bound #362

Description

@evonide

Context

Summit limits how many withdrawals are emitted in an epoch with max_withdrawals_per_epoch. The finalizer actor also handles consensus-critical mailbox work, including finalized block updates, notarized blocks, height notifications, aux-data requests, and state queries.

Claim

A normal or adversarially large set of valid withdrawal and exit requests accumulates for one epoch with size far above max_withdrawals_per_epoch; at every epoch boundary, the finalizer performs O(backlog) queue collection, overflow rescheduling, and tree rebuild work on the consensus actor path before only a capped prefix can be emitted. The withdrawal cap limits only the withdrawals included in the block, not the finalizer work required to inspect and roll over a larger ready backlog. Epoch-final aux-data collection builds a vector for the full ready epoch before applying .take(max_withdrawals), and epoch transition later reschedules all overflow withdrawals and rebuilds the withdrawal SSZ tree.

Flow

The path is reachable from accepted withdrawal state when the number of ready withdrawals for an epoch exceeds the cap. The scan did not quantify the backlog size needed for material delay or prove malicious public ingress, so a focused workload still needs to measure finalizer occupancy.

Impact

The cap can give operators a false resource bound: a large backlog can still stall finalizer mailbox service at epoch boundaries, delaying SyncerUpdate, NotifyAtHeight, and GetAuxData handling. Repeated overflow can carry the same large work set forward into later epochs.

Root Cause

The code applies the per-epoch cap after collecting the full ready queue and reschedules overflow by moving every remaining key plus rebuilding the complete withdrawal proof subtree.

Code

  • The finalizer actor handles consensus and query mailbox work in one loop:
    select! {
    mailbox_message = self.mailbox.next() => {
    let mail = mailbox_message.expect("Finalizer mailbox closed");
    match mail {
    FinalizerMessage::SyncerUpdate { update } => {
    match update {
    Update::Tip(_height, _digest) => {
    // I don't think we need this
    }
    Update::FinalizedBlock((block, finalization), ack_tx) => {
    self.handle_finalized_block(ack_tx, block, finalization, &mut orchestrator_mailbox, &mut last_committed_timestamp).await;
    }
    Update::NotarizedBlock(block) => {
    self.handle_notarized_block(block).await;
    }
    }
    },
    FinalizerMessage::NotifyAtHeight { height, block_digest, response } => {
    if self.canonical_state.get_latest_height() > height {
    // This block proposal is trying to build a block at height + 1,
    // but the canonical chain is already at height + 1 (or higher),
    // so the proposal should be aborted.
    let _ = response.send(false);
    warn!(
    "Aborting height notification for height {} and digest {} at epoch {} and height {} because the height is outdated",
    height,
    block_digest,
    self.canonical_state.get_epoch(),
    self.canonical_state.get_latest_height()
    );
    } else if height == self.canonical_state.get_latest_height() {
    // If the height matches the height of the canonical chain,
    // we check if the digest matches the head of the canonical chain.
    // If the digests don't match, then the proposal should be aborted.
    if block_digest == self.canonical_state.get_head_digest() {
    let _ = response.send(true);
    } else {
    let _ = response.send(false);
    warn!(
    "Aborting height notification for height {} and digest {} at epoch {} and height {} because the head digest is {}",
    height,
    block_digest,
    self.canonical_state.get_epoch(),
    self.canonical_state.get_latest_height(),
    self.canonical_state.get_head_digest()
    );
    }
    } else {
    // If the block was already executed on one of the forks,
    // we send the notification immediately, otherwise we store the request
    if self.fork_states.get(&height)
    .map(|forks| forks.contains_key(&block_digest))
    .unwrap_or(false) {
    let _ = response.send(true);
    } else {
    self.pending_height_notifys.entry((height, block_digest)).or_default().push(response);
    }
    }
    },
    FinalizerMessage::GetAuxData { height, parent_digest, response } => {
    self.handle_aux_data_mailbox(height, parent_digest, response).await;
    },
    FinalizerMessage::GetEpochGenesisHash { epoch, response } => {
    // The finalizer sends a message to the orchestrator to start the new epoch.
    // The orchestrator will start the new Simplex instance, which will then request
    // the epoch genesis hash from the finalizer.
    // Since the finalizer increments `self.canonical_state.epoch` before sending the message to the
    // orchestrator, the finalizer should never receive a GetEpochGenesisHash request for the wrong epoch.
    if epoch != self.canonical_state.get_epoch() {
    error!(target: "critical", "Finalizer received epoch genesis hash request from a different epoch. This should not happen and is a bug. Our epoch: {}, requested epoch {}", self.canonical_state.get_epoch(), epoch);
    #[cfg(feature = "prom")]
    counter!("critical_errors_total", "reason" => "epoch_mismatch", "severity" => "critical").increment(1);
    }
    let _ = response.send(self.canonical_state.get_epoch_genesis_hash());
    },
    FinalizerMessage::QueryState { request, response } => {
    self.handle_consensus_state_query(request, response).await;
    },
    .
  • Aux data collects all withdrawals for the epoch before applying take(max_withdrawals):
    // Only submit withdrawals at the end of an epoch, capped by max_withdrawals_per_epoch
    let current_epoch = state.get_epoch();
    let max_withdrawals = state.get_max_withdrawals_per_epoch() as usize;
    let ready_withdrawals: Vec<_> = state
    .get_withdrawals_for_epoch(current_epoch)
    .into_iter()
    .take(max_withdrawals)
    .cloned()
    .collect();
    .
  • WithdrawalQueue::get_for_epoch iterates and collects the full scheduled epoch queue:
    /// Get all pending withdrawals for a specific epoch.
    pub fn get_for_epoch(&self, epoch: u64) -> Vec<&PendingWithdrawal> {
    self.schedule
    .get(&epoch)
    .map(|queue| {
    queue
    .iter()
    .filter_map(|pk| self.withdrawals.get(pk))
    .collect()
    })
    .unwrap_or_default()
    .
  • Epoch transition reschedules any remaining current-epoch withdrawals:
    // Reschedule any overflow withdrawals that exceeded max_withdrawals_per_epoch
    // to the next epoch (placed at the front of the queue for priority).
    let current_epoch = self.canonical_state.get_epoch();
    if self
    .canonical_state
    .get_withdrawal_count_for_epoch(current_epoch)
    > 0
    {
    let overflow_count = self
    .canonical_state
    .get_withdrawal_count_for_epoch(current_epoch);
    info!(
    current_epoch,
    overflow_count, "rescheduling overflow withdrawals to next epoch"
    );
    self.canonical_state
    .reschedule_withdrawal_epoch(current_epoch, current_epoch + 1);
    }
    .
  • Rescheduling iterates every overflow pubkey and prepends it to the next epoch:
    pub fn reschedule_epoch(&mut self, from_epoch: u64, to_epoch: u64) {
    if let Some(mut pubkeys) = self.schedule.remove(&from_epoch) {
    if pubkeys.is_empty() {
    return;
    }
    // Update the epoch on each withdrawal entry
    for pk in &pubkeys {
    if let Some(w) = self.withdrawals.get_mut(pk) {
    w.epoch = to_epoch;
    }
    }
    // Prepend to the target epoch's schedule (rescheduled withdrawals get priority)
    if let Some(existing) = self.schedule.get_mut(&to_epoch) {
    pubkeys.extend(existing.iter().copied());
    *existing = pubkeys;
    } else {
    self.schedule.insert(to_epoch, pubkeys);
    }
    .
  • Rescheduling then rebuilds withdrawal SSZ state, and rebuild_withdrawals walks all epochs and withdrawal items:
    /// Move remaining withdrawals from one epoch to another.
    pub fn reschedule_withdrawal_epoch(&mut self, from_epoch: u64, to_epoch: u64) {
    self.withdrawal_queue.reschedule_epoch(from_epoch, to_epoch);
    self.ssz_tree.rebuild_withdrawals(&self.withdrawal_queue);
    }
    ,
    pub fn rebuild_withdrawals(&mut self, queue: &WithdrawalQueue) {
    let epochs = queue.epochs_with_withdrawals();
    let epoch_count = epochs.len();
    let mut epoch_subtrees = Vec::with_capacity(epoch_count);
    let mut epoch_counts = Vec::with_capacity(epoch_count);
    let mut pubkey_index = HashMap::new();
    let mut epoch_tree = SszTree::new(epoch_count.max(1));
    for (epoch_slot, &epoch) in epochs.iter().enumerate() {
    let withdrawals = queue.get_for_epoch(epoch);
    let count = withdrawals.len();
    let leaf_count = (count * WITHDRAWAL_FIELDS_PER_ITEM).max(1);
    let mut subtree = SszTree::new(leaf_count);
    for (item_slot, withdrawal) in withdrawals.iter().enumerate() {
    Self::set_withdrawal_fields(&mut subtree, item_slot, withdrawal);
    pubkey_index.insert(withdrawal.pubkey, (epoch_slot, item_slot));
    }
    let epoch_leaf = mix_in_length(subtree.root(), count);
    epoch_tree.set_leaf(epoch_slot, epoch_leaf);
    epoch_subtrees.push(subtree);
    epoch_counts.push(count);
    }
    self.withdrawal_epoch_tree = epoch_tree;
    self.withdrawal_epoch_subtrees = epoch_subtrees;
    self.withdrawal_epoch_counts = epoch_counts;
    self.withdrawal_epoch_keys = epochs;
    self.withdrawal_pubkey_index = pubkey_index;
    self.update_withdrawal_collection_root();
    .

Related Issues/PRs

The following existing items touch the same trust boundary, adjacent root cause, or likely remediation stack.

Note that #362 is overflow backlog work outside the per-epoch cap.

Fix

Select only the capped prefix without materializing the entire epoch queue, store overflow in a structure that can roll forward incrementally, and update the withdrawal proof subtree proportionally to changed entries. Add a regression benchmark for backlog sizes above the cap.

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Fields

No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions