Skip to content

fix(consensus): open the store only after the execution engine is reachable - #467

Merged
ZhiyuCircle merged 1 commit into
circlefin:mainfrom
ZanCorDX:dx/avoid-db-repair-on-startup-failure
Sep 25, 2026
Merged

ZhiyuCircle merged 1 commit into
circlefin:mainfrom
ZanCorDX:dx/avoid-db-repair-on-startup-failure

Conversation

@ZanCorDX

Copy link
Copy Markdown
Contributor

Fixes #466

App::start() opens the redb store before connecting to the execution engine.
If the CL is terminated while it waits for the EL (up to 30 s of retries), no
SIGTERM handler is installed yet, so the process dies with the store open and
without the quick-repair commit that redb performs on Drop. The next start then
runs a full repair: 1h04 on a 103 GB testnet store.

This moves open_store to just before State::builder, its first use, so the
store is not open during the EL wait. No behavior change on the success path.

Reproduce (v0.8.0, EL stopped):

  1. systemctl start arc-malachite; wait for Database opened and the
    Failed to connect to Ethereum node retries.
  2. systemctl stop arc-malachite during the retries.
  3. Start the EL, then systemctl start arc-malachite:
    Database repair in progress: 0.00% instead of Database opened.

With this patch, step 1 logs no Opening database before the EL check, and
step 3 opens the store immediately and completes the handshake.

This is a minimal patch. The underlying issue is that a clean close depends on
the shutdown path: store.savepoint() is called explicitly on every exit path
(7 call sites across node.rs, app.rs, state.rs and coord_upgrade.rs) and
the SIGTERM handler is only installed after start() succeeds. A proper fix
would install the signal handler before the store is opened and funnel every
exit through a single teardown that stops the actors and closes the store, so
no path has to remember the savepoint.

@github-actions

Copy link
Copy Markdown
Contributor

Hi @ZanCorDX,

Thank you for your interest in contributing to Arc Node.

This PR has been automatically closed because you are not assigned to issue #466. We require contributors to be explicitly assigned to an issue before submitting a PR.

To contribute properly:

  1. Comment on issue CL: SIGTERM during startup while waiting for the EL forces a full redb repair on next start #466 requesting assignment
  2. Wait for maintainer approval
  3. Only submit a PR after you have been assigned

Please see our CONTRIBUTING.md for more details.

@github-actions github-actions Bot closed this Sep 24, 2026
@ZhiyuCircle ZhiyuCircle reopened this Sep 25, 2026
@ZhiyuCircle ZhiyuCircle added the pending-import Merged PR awaiting reverse-sync to upstream label Sep 25, 2026
@ZhiyuCircle
ZhiyuCircle merged commit 6e76402 into circlefin:main Sep 25, 2026
41 checks passed
@circle-github-action-bot circle-github-action-bot added import-conflict Import failed 3-way merge; manual resolution required and removed pending-import Merged PR awaiting reverse-sync to upstream labels Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

import-conflict Import failed 3-way merge; manual resolution required

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CL: SIGTERM during startup while waiting for the EL forces a full redb repair on next start

4 participants