feat(indexer): chain-indexer app (indexer v2) - #3579
Draft
baktun14 wants to merge 6 commits into
Draft
Conversation
Contributor
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThe Chain Indexer app now includes validated configuration, PostgreSQL persistence, RPC failover, block decoding, synchronization with advisory locking, HTTP health and status routes, server lifecycle handling, tests, and development tooling. ChangesChain Indexer application
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested reviewers: ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
baktun14
marked this pull request as ready for review
August 11, 2026 07:55
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 14
🧹 Nitpick comments (7)
apps/chain-indexer/tsconfig.json (1)
2-4: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse a broad IDE include pattern.
Change
includeto["**/*.ts"]. Keeptsconfig.build.jsonnarrow for build tooling. The current configuration excludes test and root TypeScript files from IDE diagnostics.Proposed change
- "exclude": ["node_modules", "dist", "test", "src/**/*.spec.ts"], + "exclude": ["node_modules", "dist"], "extends": "./tsconfig.build.json", - "include": ["src/**/*"], + "include": ["**/*.ts"],Based on learnings, app
tsconfig.jsonfiles should use a broad"**/*.ts"include while separate build configurations remain narrow.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/tsconfig.json` around lines 2 - 4, Update the include setting in the app tsconfig to ["**/*.ts"] so IDE diagnostics cover all TypeScript files, including tests and root-level files. Leave the existing exclude entries and the narrow tsconfig.build.json configuration unchanged.Source: Learnings
apps/chain-indexer/src/db/schema.ts (1)
58-62: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
indexer_statelands inpublic, notcosmos.Every other table uses
cosmosSchema.table. Line 58 usespgTable, so the migration createspublic.indexer_state. Move it intocosmosSchemanow. After release, relocating a table needs a migration plus a deployment window where readers and writers disagree on the search path.♻️ Proposed change
-export const IndexerState = pgTable("indexer_state", { +export const IndexerState = cosmosSchema.table("indexer_state", { stream: text("stream").primaryKey(), lastHeight: bigint("last_height", { mode: "number" }).notNull(), updatedAt: timestamp("updated_at", { withTimezone: true }).notNull() });Regenerate the migration and snapshot after this change.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/db/schema.ts` around lines 58 - 62, Update IndexerState to use cosmosSchema.table instead of pgTable so indexer_state is created in the cosmos schema consistently with the other tables. Regenerate the associated migration and schema snapshot to reflect the corrected table location.apps/chain-indexer/src/pipeline/sync-runner.service.ts (1)
140-154: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueConfirm the restart path when the checkpoint block row is absent.
Line 145 sets
#lastHashtonullwhen no row matchesstate.lastHeight.#verifyContinuitythen skips the parent-hash check for the first block after the restart. That is the correct fallback, but it also silently masks a checkpoint that points at a height with no persisted block. Log a warning in that branch so the condition is visible.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/pipeline/sync-runner.service.ts` around lines 140 - 154, Update `#resolveStartHeight` so that when state exists but checkpointBlock is absent, it logs a warning describing the missing persisted block and checkpoint height before setting `#lastHash` to null; preserve the existing restart behavior and parent-hash fallback.apps/chain-indexer/src/pipeline/block-committer.service.ts (1)
33-41: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
as numberhides a reachable undefinedtypeId.Line 38 asserts the lookup succeeds. If
#internMessageTypesdoes not return an id for a type URL,typeIdisundefinedat runtime and the insert fails inside the transaction with a driver-level error that names no type URL. Throw an explicit error instead so the failure identifies the missing type URL.♻️ Proposed change
const messageRows = block.transactions.flatMap(tx => - tx.messages.map(message => ({ - height: block.height, - txIndex: tx.index, - index: message.index, - typeId: typeIds.get(message.typeUrl) as number, - body: message.body - })) + tx.messages.map(message => { + const typeId = typeIds.get(message.typeUrl); + + if (typeId === undefined) { + throw new Error(`Missing interned message type id for ${message.typeUrl} at height ${block.height}`); + } + + return { + height: block.height, + txIndex: tx.index, + index: message.index, + typeId, + body: message.body + }; + }) );🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/pipeline/block-committer.service.ts` around lines 33 - 41, Update the message-row construction in the block commit flow to validate the result of typeIds.get(message.typeUrl) before assigning typeId. Throw an explicit error that includes the missing message.typeUrl when no ID exists, and remove the unsafe “as number” assertion while preserving numeric IDs for successful lookups.apps/chain-indexer/src/providers/logging.provider.ts (1)
4-4: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winUse
@singleton()forLoggerService.
LoggerServiceis stateless.@injectable()uses tsyringe's default transient scope, while@singleton()reuses one instance. (raw.githubusercontent.com) Change the import and decorator.As per coding guidelines, backend stateless tsyringe services must use
@singleton().Suggested fix
-import { container, injectable } from "tsyringe"; +import { container, singleton } from "tsyringe"; -@injectable() +@singleton() export class LoggerService extends LoggerServiceOriginal {Also applies to: 8-12
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/providers/logging.provider.ts` at line 4, Update LoggerService to use tsyringe’s singleton scope: import singleton instead of injectable and replace the class decorator with `@singleton`(), preserving the existing container registration and service behavior.Source: Coding guidelines
apps/chain-indexer/src/pipeline/block-decoder.service.ts (1)
38-51: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
decodeTxRawfailure halts sync at one block.
decodeTxRawthrows on bytes it cannot parse. The error propagates throughdecodetosync-runner.service.ts, and the checkpoint never advances past that height. Decide the policy explicitly: either persist the transaction with empty messages and a marker, or let it throw but confirm that operators get an actionable alert. The current behavior is an unattended stall.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/pipeline/block-decoder.service.ts` around lines 38 - 51, Update `#decodeTransaction` and its decode caller to explicitly handle decodeTxRaw failures so one malformed transaction cannot silently stall block synchronization. Prefer persisting the transaction with empty messages and a clear decode-failure marker while retaining its index, hash, fee, and execution metadata; otherwise propagate the error only through an actionable operator alert and ensure the checkpoint still follows the intended policy.apps/chain-indexer/src/rpc/rpc-client-pool.service.ts (1)
57-76: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick winConsider one retry pass with backoff before failing.
#getmakes exactly one attempt per node, then throwsAggregateError.sync-runner.service.tspropagates that failure. A short simultaneous blip on all endpoints therefore aborts the block. Add a bounded second pass with a small delay, or wrap the call in a retry policy at the caller.Also consider draining or cancelling the body on the non-ok path in
#fetchFromNode(Line 81-83) so keep-alive sockets are reused.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/chain-indexer/src/rpc/rpc-client-pool.service.ts` around lines 57 - 76, Update `#get` to perform one bounded retry pass across RPC candidates after a short backoff before throwing AggregateError, preserving node health and inFlight bookkeeping for every attempt. In `#fetchFromNode`, consume or cancel the response body on non-OK responses before propagating the failure so connections can be reused.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/chain-indexer/src/config/env.config.ts`:
- Line 20: Update the PORT schema in the environment configuration to require an
integer between 0 and 65535, while preserving its coercion and default value of
3092.
In `@apps/chain-indexer/src/index.ts`:
- Around line 20-27: Validate the result returned by startServer in both the
sync and api branches before proceeding. If startup returns undefined, fail
bootstrap by throwing or setting a nonzero exit code; only start
SyncRunnerService after a successful server start, and prevent the api branch
from exiting successfully.
In `@apps/chain-indexer/src/pipeline/block-committer.service.ts`:
- Around line 74-89: Update
apps/chain-indexer/src/pipeline/block-committer.service.ts:74-89 in
`#internMessageTypes` to select and cache existing MessageTypes rows for the
cache-missed URLs before inserting, insert only the remaining URLs, then perform
the final stillMissing lookup to handle concurrent inserts. Update
apps/chain-indexer/src/db/schema.ts:38 to use serial/integer types for
MessageTypes.id and Messages.typeId, then regenerate the migration and snapshot.
In `@apps/chain-indexer/src/pipeline/block-decoder.service.ts`:
- Around line 24-36: Update decode in the block-decoding service to validate
that txs_results exists and contains one result for every transaction before
mapping rawTxs. Fail fast with an error when the results are missing or shorter
than rawTxs, and do not allow `#decodeTransaction` to default missing results to
successful zero-gas transactions.
In `@apps/chain-indexer/src/pipeline/canonical-json.ts`:
- Around line 1-9: Update replacer to detect the original property value with
Buffer.isBuffer before the existing bigint and Uint8Array checks, converting
Buffers to base64 so JSON.stringify does not serialize them as its toJSON object
representation.
In `@apps/chain-indexer/src/pipeline/sync-runner.service.ts`:
- Around line 124-138: Implement periodic leadership validation by adding
`#assertLeadership` to SyncRunnerService and invoking it at the start of every
outer iteration in `#run`; stop the sync loop when the advisory-lock assertion
fails. Update the IndexerState checkpoint upsert in BlockCommitterService so its
onConflictDoUpdate applies only when excluded.last_height is greater than the
existing indexer_state.last_height, preventing stale writers from rewinding
progress.
- Around line 82-95: Update the sync loop in start() to retry transient
`#syncBlock` failures with bounded backoff, while preserving a hard stop for
`#verifyContinuity` errors. Ensure fatal loop termination marks the service
unhealthy or sets process.exitCode and shuts down so /healthz cannot remain
green. In `#getTipHeight`, validate the parsed height and handle NaN by logging
and delaying before retrying, preventing a busy loop.
In `@apps/chain-indexer/src/providers/db.provider.ts`:
- Around line 20-30: Update migrateDb to acquire a migration-specific PostgreSQL
advisory lock on migrationClient before calling migrate, using a key distinct
from SYNC_LEADER_LOCK_KEY. Release the same lock in the finally path before
migrationClient.end(), ensuring lock acquisition and release use the same
client.
In `@apps/chain-indexer/src/server.ts`:
- Around line 3-5: Replace the console.error call in the bootstrap failure
handler with LoggerService, using a stable event name and preserving the caught
error as a structured field. Keep the existing process.exitCode = 1 behavior
unchanged.
In
`@apps/chain-indexer/src/services/hono-error-handler/hono-error-handler.service.ts`:
- Around line 59-68: In the unrecognized-error branch of the Hono error handler,
log the original error message for server-side diagnostics, but replace the
response’s dynamic message with a constant generic internal-server-error
message. Keep the existing 500 status and error metadata unchanged.
In `@apps/chain-indexer/src/services/shutdown-server/shutdown-server.ts`:
- Around line 21-29: Update the shutdown flow around server.close in the
shutdown handler to track active connections, immediately close idle sockets,
and force-destroy any remaining sockets after a grace period. Ensure the
forced-close path invokes shutdown exactly once so the returned Promise resolves
and disposeContainerOnce can run, while preserving the existing immediate
shutdown and error handling behavior.
In `@apps/chain-indexer/src/services/start-server/start-server.ts`:
- Line 51: Remove the processEvents “exit” listener that invokes shutdown. Keep
the existing SIGTERM and SIGINT shutdown handling unchanged, since the async
shutdown function cannot complete from the synchronous exit event and the once
wrapper can suppress a later real shutdown.
- Around line 43-52: Update the server startup flow around serve() to attach an
error listener to the returned server before returning it. When a listen failure
is emitted, invoke shutdown with the received error so startup aborts and the
sync role does not continue indexing without a listener; preserve the existing
close and process signal handling.
In `@apps/chain-indexer/test/setup-unit-tests.ts`:
- Around line 4-9: Update the afterAll teardown around container.dispose() to
handle the already-disposed case explicitly, while allowing all other disposal
errors to propagate. Remove the blanket suppression and rethrow unexpected
errors so cleanup failures cause the tests to fail.
---
Nitpick comments:
In `@apps/chain-indexer/src/db/schema.ts`:
- Around line 58-62: Update IndexerState to use cosmosSchema.table instead of
pgTable so indexer_state is created in the cosmos schema consistently with the
other tables. Regenerate the associated migration and schema snapshot to reflect
the corrected table location.
In `@apps/chain-indexer/src/pipeline/block-committer.service.ts`:
- Around line 33-41: Update the message-row construction in the block commit
flow to validate the result of typeIds.get(message.typeUrl) before assigning
typeId. Throw an explicit error that includes the missing message.typeUrl when
no ID exists, and remove the unsafe “as number” assertion while preserving
numeric IDs for successful lookups.
In `@apps/chain-indexer/src/pipeline/block-decoder.service.ts`:
- Around line 38-51: Update `#decodeTransaction` and its decode caller to
explicitly handle decodeTxRaw failures so one malformed transaction cannot
silently stall block synchronization. Prefer persisting the transaction with
empty messages and a clear decode-failure marker while retaining its index,
hash, fee, and execution metadata; otherwise propagate the error only through an
actionable operator alert and ensure the checkpoint still follows the intended
policy.
In `@apps/chain-indexer/src/pipeline/sync-runner.service.ts`:
- Around line 140-154: Update `#resolveStartHeight` so that when state exists but
checkpointBlock is absent, it logs a warning describing the missing persisted
block and checkpoint height before setting `#lastHash` to null; preserve the
existing restart behavior and parent-hash fallback.
In `@apps/chain-indexer/src/providers/logging.provider.ts`:
- Line 4: Update LoggerService to use tsyringe’s singleton scope: import
singleton instead of injectable and replace the class decorator with
`@singleton`(), preserving the existing container registration and service
behavior.
In `@apps/chain-indexer/src/rpc/rpc-client-pool.service.ts`:
- Around line 57-76: Update `#get` to perform one bounded retry pass across RPC
candidates after a short backoff before throwing AggregateError, preserving node
health and inFlight bookkeeping for every attempt. In `#fetchFromNode`, consume or
cancel the response body on non-OK responses before propagating the failure so
connections can be reused.
In `@apps/chain-indexer/tsconfig.json`:
- Around line 2-4: Update the include setting in the app tsconfig to ["**/*.ts"]
so IDE diagnostics cover all TypeScript files, including tests and root-level
files. Leave the existing exclude entries and the narrow tsconfig.build.json
configuration unchanged.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 059b2ee9-5677-4128-a798-afa1b8361792
⛔ Files ignored due to path filters (1)
package-lock.jsonis excluded by!**/package-lock.json
📒 Files selected for processing (51)
apps/chain-indexer/README.mdapps/chain-indexer/drizzle.config.tsapps/chain-indexer/drizzle/0000_nappy_james_howlett.sqlapps/chain-indexer/drizzle/meta/0000_snapshot.jsonapps/chain-indexer/drizzle/meta/_journal.jsonapps/chain-indexer/env/.env.sampleapps/chain-indexer/eslint.config.mjsapps/chain-indexer/package.jsonapps/chain-indexer/src/app.tsapps/chain-indexer/src/config/env.config.tsapps/chain-indexer/src/db/bytea.tsapps/chain-indexer/src/db/pg-client.service.tsapps/chain-indexer/src/db/schema.tsapps/chain-indexer/src/http-schemas/healthz.schema.tsapps/chain-indexer/src/http-schemas/status.schema.tsapps/chain-indexer/src/index.tsapps/chain-indexer/src/lib/create-route/create-route.tsapps/chain-indexer/src/pipeline/block-committer.service.tsapps/chain-indexer/src/pipeline/block-decoder.service.spec.tsapps/chain-indexer/src/pipeline/block-decoder.service.tsapps/chain-indexer/src/pipeline/canonical-json.spec.tsapps/chain-indexer/src/pipeline/canonical-json.tsapps/chain-indexer/src/pipeline/decoded-block.tsapps/chain-indexer/src/pipeline/sync-runner.service.tsapps/chain-indexer/src/providers/app-config.provider.tsapps/chain-indexer/src/providers/db.provider.tsapps/chain-indexer/src/providers/index.tsapps/chain-indexer/src/providers/logging.provider.tsapps/chain-indexer/src/providers/raw-app-config.provider.tsapps/chain-indexer/src/providers/type-registry.provider.tsapps/chain-indexer/src/routes/healthz/healthz.router.tsapps/chain-indexer/src/routes/index.tsapps/chain-indexer/src/routes/status/status.router.tsapps/chain-indexer/src/rpc/rpc-client-pool.service.spec.tsapps/chain-indexer/src/rpc/rpc-client-pool.service.tsapps/chain-indexer/src/rpc/rpc-types.tsapps/chain-indexer/src/server.tsapps/chain-indexer/src/services/app-config/app-config.service.tsapps/chain-indexer/src/services/config/config.service.tsapps/chain-indexer/src/services/hono-error-handler/hono-error-handler.service.tsapps/chain-indexer/src/services/open-api-hono-handler/open-api-hono-handler.tsapps/chain-indexer/src/services/shutdown-server/shutdown-server.tsapps/chain-indexer/src/services/start-server/start-server.tsapps/chain-indexer/src/services/status/status.service.tsapps/chain-indexer/src/types/app-context.tsapps/chain-indexer/test/setup-unit-env.tsapps/chain-indexer/test/setup-unit-tests.tsapps/chain-indexer/tsconfig.build.jsonapps/chain-indexer/tsconfig.jsonapps/chain-indexer/tsup.config.tsapps/chain-indexer/vitest.config.ts
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
baktun14
marked this pull request as draft
August 11, 2026 12:17
baktun14
changed the title
- fail bootstrap when the HTTP server cannot start and exit the process when the sync loop dies so healthz cannot report healthy on a dead sync - detect advisory-lock session loss via backend pid checks to prevent two sync leaders after a transparent reconnect - retry transient sync errors with capped backoff; continuity breaks and leadership loss stay fatal - serialize migrations with a pg advisory lock - select existing message types before inserting to stop id sequence burn; widen message_types.id to serial and messages.type_id to integer - fail fast when txs_results does not match block txs instead of persisting txs as successful with zero gas - serialize Buffers as base64 in canonical json despite Buffer#toJSON - treat empty SYNC_START_HEIGHT as absent; validate PORT range - force-close keep-alive sockets on shutdown; handle async listen errors; drop the no-op exit listener - return a constant message for unexpected errors; log via LoggerService - remove dead InjectTypeRegistry export and inline comments
* feat(indexer): add backfill range and tuning env config * feat(indexer): support ordered batched commits with a parameterized checkpoint stream * feat(indexer): add backfill range planner * feat(indexer): implement the backfill role for historical catch-up * fix(indexer): address backfill review findings - attach a no-op catch to prefetched blocks so a rejection settling before its height is consumed cannot crash as an unhandled rejection - check range completion before tip validity so a finished range stays a no-op even when a lagging node reports a stale tip - extract the shared PgAdvisoryLeaderLock used by both runners - use lodash chunk instead of a local helper - build the pg client test double with a nested mock instead of a cast * refactor(indexer): share retry backoff and runner-role lifecycle across roles - extract retryWithBackoff and use it in both runners - retry backfill startup reads and idempotent batch commits so a transient blip no longer kills a multi-hour job - collapse the sync and backfill bootstrap cases into one helper * refactor(indexer): move continuity error and transient retry policy to shared modules * refactor(indexer): retry the continuity seed read like other transient reads * refactor(indexer): drop advisory-lock leader election for concurrency-safe writes Inserts were already idempotent; making the indexer_state checkpoint upsert monotonic (GREATEST) makes overlapping writers harmless, so the leader lock, its backend-pid liveness checks, and LeadershipLostError can all go. Single writers per role are enforced at the orchestration level instead. * refactor(indexer): use the exported LoggerService type instead of a local alias * fix(indexer): retry an interrupted backfill instead of completing the Job Addresses review findings on the backfill role: - A backfill stopped before finishing its range (SIGTERM mid-run) now throws RunnerInterruptedError so the process exits non-zero. Backfill runs as a K8s Job where exit 0 marks it Complete, so the previous clean exit silently left the range unfinished and unretried; it now resumes from its checkpoint on the next attempt. - Drop the unreachable prefetch fallback in the consume loop; the sliding window always has the current height inflight before it is awaited. - Back the retry delay with the global setTimeout so the fetch-retry test can advance it with fake timers instead of sleeping a real second. * fix(indexer): surface env validation errors without the DI wrapper A misconfigured role (e.g. a backfill Job missing BACKFILL_FROM/TO_HEIGHT) failed with a tsyringe "Cannot inject the dependency" wrapper around the ZodError plus an internal stack trace. Validate env eagerly at bootstrap and log a CONFIG_INVALID event listing the offending fields and messages instead. * refactor(indexer): parallelize backfill startup reads and signal completion via return Fetch the checkpoint height and chain tip concurrently instead of serially, hoist the loop-invariant base height in the contiguity check, and have the backfill run return whether it completed so start() no longer re-reads the stopped flag. * fix(indexer): track backfill completion by committed height, not the stopped flag A shutdown landing during the final commit left the range fully committed but reported it as interrupted, failing the Job and triggering a spurious retry. Completion is now derived from the last committed height reaching the range end. Also dedupes the identical getTipHeight helper into RpcClientPool so both runners share one implementation.
* feat(indexer): add raw block archive layout and zstd ndjson codec * feat(indexer): add gcs block archive service behind optional ARCHIVE_BUCKET * feat(indexer): archive staged raw blocks during live sync before commit * feat(indexer): source backfill blocks from the archive and compact ranges * test(indexer): cover the chain-id failure path in staged deletes * feat(indexer): allow overriding the gcs api endpoint for local emulators * chore(indexer): persist the chain-indexer local verify recipe * refactor(indexer): tighten archive invariants after code review * fix(indexer): stop malformed /status from poisoning the chain-id cache resolveChainId cached getStatus().then(onFulfilled, onRejected); a throw inside onFulfilled (a /status body missing node_info) rejected the cached promise without running the reset, so every later archive call failed with the stale rejection forever. Switch to .then().catch() so both a rejected fetch and a malformed payload clear the cache and let the retry re-fetch. Also share a fetchRawBlock helper between sync and backfill, free each range's buffer as soon as its chunk flush succeeds, and document the staged-single orphan window left by a crash between chunk put and delete. * docs(indexer): scope chain-indexer to chain data only State explicitly that the new chain-indexer owns chain-derived data only, and that off-chain provider data (status pinging, inventory, uptime, IP geolocation, GPU breakdown) belongs to apps/provider-inventory. Pricing and Keybase stay as the deliberate off-chain exception since the daily USD aggregates and validator records need them. Note the pending ownership move on the provider-inventory "provider snapshot" glossary entry. * fix(indexer): stop a corrupt chunk from poisoning the fetch cache ArchiveBlockSource#fetchChunk cached getChunk().then(onFulfilled, onRejected); a throw inside onFulfilled (a corrupt chunk line decoding to a non-object so record.height throws) rejected the cached promise without running the reset, so every later read for that range awaited the stale rejection. Switch to .then().catch() so a mapping failure also clears entry.chunkFetch and the retry re-fetches, matching the sibling fix in resolveChainId. Also hoist the duplicated buildRecord test fixture into test/fakes/build-raw-block-record so the three archive specs share one shape.
|
Review the following changes in direct dependencies. Learn more about Socket for GitHub.
|
…#3587) * feat(indexer): add genesis import framework with account and balance seed Balance tracking is only trustworthy from a network's genesis, so this seeds genesis state before the first block: accounts, per-denom balances (recorded as genesis-reason ledger entries), validators, and staking delegations, all in one transaction. It is gated behind GENESIS_IMPORT (default off). When enabled, the sync role runs a per-module seeder framework exactly once, using a genesis checkpoint in indexer_state so a restart is a no-op, and it rejects a fresh start that is not at the network's genesis height. Genesis is fetched over RPC /genesis_chunked and its chain_id is asserted against the node being indexed. This adds the accounts, account_balances, balance_changes, validators, and delegations tables (L-3); the ongoing per-block ledger lands in L-4. Refs CON-805 * refactor(indexer): share genesis validator and chunked-insert helpers Extract mapDescription/mapCommissionRates so the staking and gentx validator mappers stop duplicating the description and commission field mapping, and add an insertChunked helper the bank and staking seeders call instead of hand-rolling the chunked insert loop. * fix(indexer): warn when genesis import is enabled too late to seed The seed only runs on a fresh start, so turning GENESIS_IMPORT on after the indexer already has a sync checkpoint silently skips it and leaves the account and balance tables empty with no signal. Add a hasSeeded check on resume and log a warning so the operator knows genesis was never backfilled.
* feat(indexer): capture block/tx events and ledger idempotency schema Add the L-4 write-path foundation: a 0002 migration adding the (height, event_index) idempotency key and tx_index to balance_changes, the account_txs address-activity table, and a staking balance reason. The block decoder now captures coin/transfer/mint/burn/slash events with base64-normalized attributes and derives per-tx signer addresses, the enabler for deriving balances from events. Genesis ledger rows move to initialHeight-1 so they never collide with block initialHeight's events. * feat(indexer): derive balance changes from coin events Add the pure, DB-free derivation layer: parseCoins for coin strings, a module-address registry deriving Cosmos SDK module accounts by name (sha256(name)[:20]), an MVP reason classifier, and the balance deriver that turns coin_spent/coin_received into ordered movements with a deterministic block-wide event_index and correlated counterparties. * feat(indexer): intern accounts and write the balance ledger idempotently Add the account interner (resolve-or-create, no permanent cache) and the balance writer. The writer seeds each running balance from the ledger baseline (not the snapshot), appends balance_changes on the (height, event_index) unique key, and advances account_balances additively by only the rows it actually inserted — so overlapping sync/backfill writers apply every delta exactly once and re-commits are no-ops. * feat(indexer): write balance ledger and activity log per block Wire balance derivation into the block committer: derive changes and address-activity rows per block, intern all touched addresses on the base connection, then inside the commit transaction write the balance ledger and account_txs after messages and before the checkpoint. Both sync and backfill flow through commitBatch, so both persist the ledger. * feat(indexer): add reconciliation script proving ledger matches chain Add abci_query historical support to the RPC pool and a one-shot npm run reconcile entrypoint: at the sync checkpoint height it compares each sampled account's current balance against the node's bank balance and the ledger's per-denom totals against chain supply, exiting non-zero on any mismatch so it can gate a deploy. * fix(indexer): correct balance reason classification and reconcile guard Scope the slash reason to the coincident burn leg so a validator slash no longer mislabels the block's routine inflation mint (and other block-level movements sharing the scope) as slash. Add the missing mint case to the reason classifier so a mint-module counterparty resolves to mint instead of falling through to transfer. Reject a non-numeric RECONCILE_SAMPLE_SIZE instead of coercing it to NaN, which slice() turned into an empty sample that could report a clean pass having verified zero accounts. Drop the unread "message" event type from the decoder whitelist and remove a dead branch in decodeIfBase64. * fix(indexer): classify escrow module movements as escrow The reason classifier fell through to `transfer` for any coin movement involving the x/escrow module account, because that module wasn't in the address registry. It mislabeled every ACT (uact) flow — which only moves through escrow, never peer-to-peer — plus uakt escrow deposits, refunds, and lease settlements, as generic transfers. Add the escrow module to the registry and map it to the existing `escrow` reason. Verified against a mainnet day of blocks: uact movements now classify as escrow (previously 100% transfer) while per-account and total supply balances still reconcile exactly to the chain. * refactor(indexer): use insertChunked for block, transaction and message inserts * fix(indexer): scope slash reason, correlate counterparty by amount, fail over abci non-zero code * fix(indexer): reconcile from a consistent snapshot with bounded concurrency Read the checkpoint height and ledger balances inside one REPEATABLE READ read-only transaction so a concurrent block commit can't pair a stale height with post-commit balances and flag spurious mismatches. Fan out the per-account bank queries with bounded concurrency instead of one sequential RPC round-trip at a time, letting the RpcClientPool load-balance the sampled accounts. * fix(indexer): classify balance reason by the holder's own module role first For a module-to-module movement (fee_collector to distribution, mint to fee_collector) both sides carry a role, so preferring the counterparty tagged each leg with the other module's reason. Prefer the holder's own role and fall back to the counterparty, so every leg is classified by the module whose balance actually changed. Account-to-module movements are unaffected: the account side has no role and still falls through to the counterparty. * fix(indexer): chunk account existence lookup and document ledger Chunk the account existence SELECT the same way the insert path already does, so a batch with tens of thousands of distinct addresses can't exceed postgres.js's bind-parameter limit and abort the commit transaction. Document the balance ledger, activity log, and the reconcile script in the app README alongside the existing feature sections. * refactor(indexer): compute each account's total once when sampling Precompute the coin total per account before sorting the reconciliation sample instead of re-reducing both operands on every comparator call. * fix(indexer): chunk balance baseline lookup under the bind-parameter limit #readLedgerBaseline built accountIds/denoms from a whole commitBatch's intents and passed them into one unchunked inArray SELECT, unlike #insertChanges/#applyNetDeltas in the same class. A dense backfill batch near BACKFILL_BATCH_SIZE's 1000-block max can touch tens of thousands of distinct accounts, exceeding postgres.js's ~65k bind-parameter cap and aborting the whole commitBatch transaction. Chunk accountIds by INSERT_CHUNK_SIZE and merge per-chunk rows, matching AccountInterner's already-chunked select. * refactor(indexer): validate RECONCILE_SAMPLE_SIZE through the env schema The reconcile CLI hand-rolled Number() parsing plus a bespoke Number.isInteger/positive check and its own CONFIG_INVALID log shape, duplicating the zod envSchema every other numeric env var already flows through. Add RECONCILE_SAMPLE_SIZE to envSchema as an optional coerced positive int (same idiom as SYNC_START_HEIGHT) and read it from parsed.data, so an invalid value fails at the existing safeParse path. The service keeps its own guard as defense in depth.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Collaboration branch
This PR doubles as the integration branch for the indexer v2 rewrite. Per-issue PRs (CON-804 and onward) target
feat/indexer-scaffold-chain-indexer-appand merge here; this PR accumulates them and goes to main once a releasable slice is ready. Note for reviewers: the CI test/lint/tsc matrix only runs on PRs based on main, so we verify child PRs locally before merging.Why
Related to CON-803
First code slice of the Indexer v2 rewrite: the new
apps/chain-indexerapp plus a working sandbox tracer bullet, built in parallel with the design review so the discussion has running code to poke at. Draft until the design review lands.What
New workspace app
@akashnetwork/chain-indexerfollowing the tx-signer/api conventions (tsyringe DI, LoggerService, zod-validated env, Hono + zod-openapi, tsup, vitest). Nothing existing is touched; the only change outside the new folder is the lockfile.INDEXER_ROLE(sync | backfill | api | jobs) andNETWORK(mainnet | sandbox | testnet).syncand a minimalapi(healthz + status) are implemented;backfillandjobsexit withROLE_NOT_IMPLEMENTED.cosmos.blocks,cosmos.transactions,cosmos.messages(internedmessage_types+ size-capped decodedbody jsonb), and anindexer_statecheckpoint row per stream. No UUIDs, natural keys, hashes as bytea. Migrations run on boot.RpcClientPool: least-loaded healthy node selection, failover, cooldown on failure, timeout via AbortSignal.DecodedBlock(chain-sdk + cosmos registry), one Postgres transaction per block including the checkpoint advance, idempotent natural-key upserts, no isProcessed flags.Verified against live sandbox: tails blocks at the tip,
/v1/statuscheckpoint advances, kill + restart resumes at checkpoint + 1 with no continuity break, and rows land in all three tables. 17 unit tests over the decoder, canonical JSON serializer, and RPC pool; lint and tsc clean.Message bodies for types outside the registry (e.g. cosmwasm on sandbox) are stored as null with the typeUrl preserved; the version-adapter and dead-letter layer is a later issue.