Fix the state_changes indexes and add a foreign key on pool-share balances by aditya1702 · Pull Request #704 · stellar/wallet-backend

aditya1702 · 2026-08-26T01:08:29Z

Start here: internal/db/migrations/2026-08-21.0-statechanges-account-id-index.sql — the only file that changes an existing database.

Two schema changes. Each ships with the query change that needs it.

`state_changes` lookups by operation now use the primary key

BatchGetByOperationID and BatchGetByOperationIDs only filtered on operation_id and ledger_created_at. That isn't enough to use the primary key (ledger_created_at, to_id, operation_id, state_change_id), so both queries needed a separate index on operation_id.

We can work out to_id from the operation ID for free, because of how TOIDs are encoded: tx_to_id = operation_id &^ 0xFFF. Adding it to the WHERE clause matches the first three columns of the primary key, so both queries use the PK directly.

Two indexes now have nothing using them, and are dropped:

idx_state_changes_operation_id — the primary key covers it.
idx_state_changes_account_category — replaced by the smaller idx_state_changes_account_id.

sql-migrate tracks migrations by filename, so editing 2025-06-10.4 in place only affects brand-new databases. 2026-08-21.0 brings existing ones in line and does nothing on new ones.

Pool-share balances get a foreign key to their pool

Ingestion writes the pool row first, in the same transaction, so a share row without a pool is always a bug. The foreign key makes it fail loudly on write instead of quietly producing a broken join on read. Same idea as the existing trustline_balances → trustline_assets constraint.

It's deferrable, and added as NOT VALID then validated outside a transaction, so ingestion keeps writing while it runs.

PR 1 of 6 replacing #684 · schema → fixes → pipeline → parallel writes → table groups → final fixes

aristidesstaffieri · 2026-08-27T19:45:33Z

@@ -0,0 +1,26 @@
+-- +migrate Up


why not notransaction here too?

Copilot

Pull request overview

Optimizes state-change queries and enforces pool-share referential integrity.

Changes:

Uses derived transaction TOIDs to query via the state_changes primary key.
Replaces redundant state-change indexes.
Adds and tests a deferred liquidity-pool balance foreign key.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
`internal/db/migrations/2026-08-21.0-statechanges-account-id-index.sql`	Replaces state-change indexes.
`internal/db/migrations/2026-08-13.0-liquidity_pool_balances-pool-fk.sql`	Adds the pool foreign key.
`internal/db/migrations/2025-06-10.4-statechanges.sql`	Documents primary-key lookup behavior.
`internal/data/statechanges.go`	Adds derived `to_id` query predicates.
`internal/data/statechanges_test.go`	Tests TOID-based operation lookups.
`internal/data/liquidity_pool_balances_test.go`	Tests foreign-key enforcement.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+-- chunk, and compressed chunks prune on the bloom sparse indexes 2025-06-10.4
+-- declares. The dropped indexes are superseded: operation-scoped lookups derive
+-- their PK prefix from the TOID encoding (to_id = operation_id & ~x'FFF').
+CREATE INDEX IF NOT EXISTS idx_state_changes_account_id ON state_changes(account_id, ledger_created_at DESC, to_id DESC, operation_id DESC, state_change_id DESC);


Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated no new comments.

Suppressed comments (1)

internal/db/migrations/2026-08-21.0-statechanges-account-id-index.sql:18

This forward migration builds a multi-column index on the existing state_changes hypertable inside sql-migrate's default wrapping transaction. Plain CREATE INDEX takes a lock that conflicts with ingestion writes and holds it until the migration commits, so a production-sized build can stop live ingestion for the full index-build duration. Use the TimescaleDB version's supported online/per-chunk index-build procedure (for example, a nontransactional per-chunk build), and only drop the old indexes after that build completes.

CREATE INDEX IF NOT EXISTS idx_state_changes_account_id ON state_changes(account_id, ledger_created_at DESC, to_id DESC, operation_id DESC, state_change_id DESC);

Copilot

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

internal/db/migrations/2026-08-21.0-statechanges-account-id-index.sql:18

On an existing state_changes hypertable, ordinary CREATE INDEX holds write-blocking locks for the full multi-chunk build. This is the forward migration production databases execute, so a large database can stop live ingestion for the duration. Build the index with the TimescaleDB-supported per-chunk/online approach in a no-transaction migration, with retry handling, before dropping the old indexes.

CREATE INDEX IF NOT EXISTS idx_state_changes_account_id ON state_changes(account_id, ledger_created_at DESC, to_id DESC, operation_id DESC, state_change_id DESC);

+    FOREIGN KEY (pool_id) REFERENCES liquidity_pools (pool_id)
+    DEFERRABLE INITIALLY DEFERRED


A pool-share balance is always read alongside its pool's reserves, and ingestion writes both tables in one transaction with the pool row first, so a balance row whose pool is missing can only come from an ingestion bug. The foreign key turns that bug into a loud failure at the write instead of a silently broken join at read time, matching the trustline_balances → trustline_assets constraint already in place. DEFERRABLE INITIALLY DEFERRED so the check runs at commit: a pool is deleted only once every share balance referencing it is deleted, and both happen in the same persist transaction. The migration runs notransaction so validation gets its own transaction — ADD CONSTRAINT NOT VALID is metadata-only, and VALIDATE takes just SHARE UPDATE EXCLUSIVE, so ingestion keeps writing through the scan. Orphaned share rows are cleared first: they are unreadable anyway now that the read path joins liquidity_pools, and a single survivor would abort validation and halt the deploy. GetByAccount's test for orphaned rows becomes a test that the insert is rejected, since the row can no longer exist.

…ary key BatchGetByOperationID and BatchGetByOperationIDs bound only operation_id and ledger_created_at, so neither could reach the state_changes primary key — (ledger_created_at, to_id, operation_id, state_change_id) — and both leaned on a standalone index on operation_id. TOID encoding makes the gap closable without extra input: a state change's parent transaction to_id is derivable from its operation ID by bit masking, tx_to_id = operation_id &^ 0xFFF. Binding that derived value turns (ledger_created_at, to_id, operation_id) into a three-column PK prefix, so both queries probe the PK directly. In BatchGetByOperationIDs the extra pinned column also drops out of the ORDER BY, which is now state_change_id alone — the only column that varies within one lateral probe. With operation-scoped lookups on the PK, idx_state_changes_operation_id has no consumer left. idx_state_changes_account_category goes too: the account-scoped queries that used it are served by a narrower index on (account_id, ledger_created_at DESC, to_id DESC, operation_id DESC, state_change_id DESC), whose trailing columns repeat the PK's sort key so an account's page is a heapsort-free prefix scan. Category and reason are left out because an account's rows are few enough to scan and filter on the active chunk, and compressed chunks prune on the bloom sparse indexes. sql-migrate keys applied migrations by filename, so editing 2025-06-10.4 in place only reshapes fresh databases. 2026-08-21.0 converges the ones already past it, and is a no-op everywhere else.

The index swap was expressed twice: 2025-06-10.4 was edited in place to create idx_state_changes_account_id instead of idx_state_changes_operation_id and idx_state_changes_account_category, and 2026-08-21.0 made the same change again for databases already past that filename. Two files then had to be hand-kept in agreement, and 2026-08-21.0's Down recreated indexes 2025-06-10.4 no longer defined. 2025-06-10.4 goes back to creating the original pair, so 2026-08-21.0 is the one place the swap is expressed and every database reaches the final shape the same way. Its CREATE carries IF NOT EXISTS and its DROPs carry IF EXISTS already, so a fresh database builds the pair and converges in the next step, at the cost of two transient index builds at setup. The primary key's comment keeps the note that operation-scoped lookups ride it via the TOID encoding: that describes how the queries are written, not which indexes exist.

This was referenced Aug 26, 2026

Three small fixes: checkpoint pool shares, worker pool sizing, metric buckets #705

Merged

Pipeline live ingestion into fetch/process/persist with seven parallel write streams #684

Closed

aditya1702 changed the title ~~Reshape the state_changes indexes and constrain pool-share balances~~ Fix the state_changes indexes and add a foreign key on pool-share balances Aug 26, 2026

aditya1702 force-pushed the persist/1-schema branch from b3105e7 to ca47c77 Compare August 26, 2026 20:25

aristidesstaffieri reviewed Aug 27, 2026

View reviewed changes

aristidesstaffieri approved these changes Aug 27, 2026

View reviewed changes

Copilot AI balanced review requested due to automatic review settings September 3, 2026 23:45

aditya1702 force-pushed the persist/1-schema branch from 2290cff to 458c7fb Compare September 3, 2026 23:45

Copilot started reviewing on behalf of aditya1702 September 3, 2026 23:46

View session

Copilot AI reviewed Sep 3, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 4, 2026 19:30

aditya1702 force-pushed the persist/1-schema branch from 458c7fb to f5d34f2 Compare September 4, 2026 19:30

Copilot started reviewing on behalf of aditya1702 September 4, 2026 19:31

View session

Copilot AI reviewed Sep 4, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings September 9, 2026 21:07

aditya1702 force-pushed the persist/1-schema branch from f5d34f2 to 31c0ac8 Compare September 9, 2026 21:07

Copilot started reviewing on behalf of aditya1702 September 9, 2026 21:07

View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes

Comment thread internal/db/migrations/2026-08-13.0-liquidity_pool_balances-pool-fk.sql

Comment on lines +29 to +30

FOREIGN KEY (pool_id) REFERENCES liquidity_pools (pool_id)

DEFERRABLE INITIALLY DEFERRED

aditya1702 self-assigned this Sep 12, 2026

aditya1702 added this to the Wallet Backend O3 Performance Testing milestone Sep 12, 2026

aditya1702 force-pushed the persist/1-schema branch from 31c0ac8 to f54b004 Compare September 15, 2026 15:41

aditya1702 added 3 commits September 15, 2026 12:01

aditya1702 force-pushed the persist/1-schema branch from f54b004 to 069ff55 Compare September 15, 2026 16:03

aditya1702 merged commit 73f6a24 into main Sep 15, 2026
13 of 14 checks passed

aditya1702 deleted the persist/1-schema branch September 15, 2026 18:18

Navigation Menu

Repository navigation

Fix the state_changes indexes and add a foreign key on pool-share balances - #704

Fix the state_changes indexes and add a foreign key on pool-share balances#704
aditya1702 merged 3 commits into
replay-loadtest-backend-pr6from
persist/1-schema

aditya1702 commented Aug 26, 2026 •

edited

Loading

Uh oh!

aristidesstaffieri Aug 27, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		FOREIGN KEY (pool_id) REFERENCES liquidity_pools (pool_id)
		DEFERRABLE INITIALLY DEFERRED

Conversation

aditya1702 commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

state_changes lookups by operation now use the primary key

Pool-share balances get a foreign key to their pool

Uh oh!

aristidesstaffieri Aug 27, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aditya1702 commented Aug 26, 2026 •

edited

Loading

`state_changes` lookups by operation now use the primary key