feat(indexer): genesis import framework with account and balance seed - #3587
Conversation
…seed Balance tracking is only trustworthy from a network's genesis, so this seeds genesis state before the first block: accounts, per-denom balances (recorded as genesis-reason ledger entries), validators, and staking delegations, all in one transaction. It is gated behind GENESIS_IMPORT (default off). When enabled, the sync role runs a per-module seeder framework exactly once, using a genesis checkpoint in indexer_state so a restart is a no-op, and it rejects a fresh start that is not at the network's genesis height. Genesis is fetched over RPC /genesis_chunked and its chain_id is asserted against the node being indexed. This adds the accounts, account_balances, balance_changes, validators, and delegations tables (L-3); the ongoing per-block ledger lands in L-4. Refs CON-805
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. 🗂️ Base branches to auto review (1)
Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Comment |
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Extract mapDescription/mapCommissionRates so the staking and gentx validator mappers stop duplicating the description and commission field mapping, and add an insertChunked helper the bank and staking seeders call instead of hand-rolling the chunked insert loop.
The seed only runs on a fresh start, so turning GENESIS_IMPORT on after the indexer already has a sync checkpoint silently skips it and leaves the account and balance tables empty with no signal. Add a hasSeeded check on resume and log a warning so the operator knows genesis was never backfilled.
There was a problem hiding this comment.
I reviewed this PR (automated review) and didn't find any bugs beyond the nits already noted inline. Because it introduces a new DB migration and non-trivial genesis-parsing/import logic in a data-integrity-sensitive path, a human look would still be worthwhile before merging.
What was reviewed: the genesis-import transaction flow (marker claim, mid-chain rejection, per-module seeders), the genesis-schema parsing of Cosmos SDK auth/bank/staking/genutil shapes, address derivation (bech32 re-encoding, ed25519 consensus hex address), and the sync-runner's fresh-start vs. resume gating for the one-time seed. Also checked that account interning happens before the balance/staking seeders reference it, and that the migration is additive-only.
Extended reasoning...
Overview
This PR adds a genesis-import framework to apps/chain-indexer: a new migration (accounts, account_balances, balance_changes, delegations, validators tables + an enum), a GenesisImportService that seeds all modules in one transaction gated by a marker row in indexer_state, per-module seeders (account/bank/staking), a genesis-schema parser for Cosmos SDK genesis documents, address-derivation helpers (bech32 re-encoding, consensus hex address from ed25519 pubkey), an RPC genesis-chunk fetcher/reassembler, and changes to sync-runner.service.ts to gate the seed on a fresh start vs. resume.
Security risks
No injection, auth, or data-exposure concerns — all writes are parameterized via Drizzle, and the RPC genesis fetch validates chain_id against the node's reported network before importing. The main correctness risk is data-integrity, not security: a wrong parse or wrong height gate could silently seed incorrect balances, which is why the PR leans on strict validation (mid-chain rejection, chain_id check, marker-based idempotency).
Level of scrutiny
This warrants a higher level of scrutiny than average: it's a new, additive migration (low risk to existing data) paired with non-trivial one-time business logic (bech32/hex address derivation, nested schema variants for validators from two different genesis shapes, chunked genesis reassembly) in a data-correctness-sensitive area. The bug-hunting pass found no confirmed bugs this run — three candidate issues (mid-chain error ordering relative to the marker check, sequential genesis chunk fetching, and balance/ledger desync) were raised and refuted by verifiers — and two non-blocking nits (validator-mapping duplication, chunked-insert-loop duplication) remain from a prior pass. Given the size (XL) and the fact this is laying the foundation for balance tracking that other features will depend on, a human sanity-check of the parsing/derivation logic against real genesis exports is worth the extra pass.
Other factors
Test coverage is thorough — dedicated specs for the address helpers, schema parsing (including edge cases like malformed operator addresses and missing account numbers), each seeder, the import service's concurrency/idempotency paths, and sync-runner's genesis gating/warning behavior. The migration is additive-only and every new index is on an empty table, so it carries no lock risk against existing data.
6f4d27f
into
feat/indexer-scaffold-chain-indexer-app
1 participant
Why
Balance tracking is only trustworthy if it starts from the network's first block. The indexer already seeds validators from genesis, but account balances have no starting point, so any balance derived from block deltas alone would drift from the truth.
Closes CON-805
What
A genesis import framework that seeds per-module state in a single transaction before the first block is indexed:
genesis-reason rows in the balance-change ledgerThe seed runs exactly once. A marker row in
indexer_stateshort-circuits repeat runs, and anonConflictDoNothingclaim keeps concurrent fresh starts from double-seeding. Balance tracking also refuses to begin unless the effective start height is the network's genesis height; a mid-chain start throws with a message pointing at the correctSYNC_START_HEIGHT.Migration
0001_long_mach_iv.sqlis additive only: newaccounts,account_balances,balance_changes,delegations, andvalidatorstables plus thebalance_change_reasonenum. Every index is created on an empty table, so it runs without locking existing data.Stack
Stacked on #3579 (chain-indexer scaffold); #3585 (raw block archive) already merged into that branch, so the base here is
feat/indexer-scaffold-chain-indexer-app. It retargets tomainonce #3579 lands.