Skip to content

[bug]: wallet creation slows sync if run before neutrino headers complete #10180

Closed

Description

@shocknet-justin

Background

We noticed that new nodes were taking forever to sync even under ideal environmental conditions. We have correlated this to when the timing of createwallet comes before headers have sync'd.

Our deployments use neutrino to our own colocated servers.

We've long used this deployment flow without issue, but just discovered this after testing greenfield deployments for the first time in awhile.

Your environment

Encountered this on 19.3, reproduced on 19.1

Confirmed that issue is not present in 18.3

(Also tested 19.0 and 19.2 but they have separate issues when attempting to repro, rpc hangs and crashes respectively.)

Tests were run on an Debian 13/amd64 VM with ample memory, bandwidth, and nvme storage

We reproduced this with our two separate neutrino nodes in disparate datacenters, one using btcd 24.2 and the other using Core 26.1.

Steps to reproduce

Start lnd and run create wallet before headers complete sync, observe that syncing then crawls block by block.

Image

A restart of LND then clears this issue.

Expected behaviour

Lnd should report it sync'd to chain once headers sync is complete since the wallet is new and not be delayed by what appears to be a block-level rescan.

Actual behaviour

Seems that because the wallet is created before headers lnd does not know its age and therefore triggers the much slower rescan.

Upon restart (unlock) the situation clears itself, presumably this is because the wallet exists while lnd is at the headers tip and thus no rescan is triggered.

Other thoughts

We've always used walletunlocker to create the wallet once that rpc made itself available, afaik this is the only rpc available before createwallet.

A workaround to this issue for us would be to delay createwallet until after header sync is complete, but we would need to come up with some sketchy log parsing that may not work well in many environments to do so.

Activity

  1. Roasbeef commented on Aug 28, 2025

    @Roasbeef
    Member

    cc @mohamedawnallah

  2. moawnallah commented on Aug 28, 2025

    @moawnallah
    Contributor

    cc @mohamedawnallah

    Acknowledged. Will take a deeper look into this issue 👍

  3. moawnallah commented on Sep 8, 2025

    @moawnallah
    Contributor

    It seems this issue is reproduced in the CI in some form. This test TestLightningWallet/btcwallet/neutrino:reorg_wallet_balance is failing in a flaky way in the CI:
    https://github.com/lightningnetwork/lnd/actions/runs/16982438758/job/48145091023?pr=9455

    I remember as well it failed very recently like few days ago (in recent runs in that PR #9455). It appears to me this test failure regression is directly related to this issue

    There was a PR attempt to resolve this test issue (btcsuite/btcwallet#1040) but it turns out and seems it didn't resolve it yet

    The problem seems happen when neutrino started downloading headers and we set wallet birthday to be time.now() so we rescan the blocks diff between current time and best block chain tip:
    https://github.com/lightningnetwork/lnd/blob/master/config_builder.go#L361

    cc: @yyforyongyu, @ellemouton for any thoughts

  4. yyforyongyu commented on Sep 9, 2025

    @yyforyongyu
    Member

    @mohamedawnallah I think we should re-evaluate this issue once the btcwallet refactor is complete, given it will largely change how we handle the rescanning.

  5. added this to the v0.21.0 milestone on Sep 9, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugUnintended code behaviourneutrinoLightweight neutrino backend-typesync

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions