Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling by aditya1702 · Pull Request #686 · stellar/wallet-backend

aditya1702 · 2026-08-13T22:17:09Z

Four independent fixes, one commit each — reviewable in any order. Review time: ~20 min. One manual-DDL warning below.

Found while profiling live ingestion: a metric that clipped exactly the queries being optimized, a missing index that made an hourly job 45% of all DB disk reads, five compression policies firing at the same instant, and a phase histogram whose counts stopped matching its siblings.

1. Query-duration buckets clipped at 0.38 s

wallet_db_query_duration_seconds topped out at 0.38 s: every multi-second bulk COPY landed in +Inf and p99 panels flattened. Widened to 0.1 ms – ~17.7 s.

2. Oldest-ledger lookup had no ordered path

SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1 (backfill gap detection + the hourly reconcile_oldest_cursor job) had no index to walk — an earlier audit dropped TimescaleDB's default partition-column index after counting only WHERE-clause consumers, missing that this query consumes the index's ordering. The planner pulled the first row from every chunk: 12–187 s on an 87 GB DB; the hourly job was 45% of all DB disk-read time.

Fix: keep the default index; drop the to_id tie-break from both query sites (close times increase strictly — the tie-break could only force an extra sort). Tested by running the reconciliation job against out-of-order chunks.

Warning

Manual DDL on already-migrated databases (fresh DBs get the index from create_hypertable). Applied on the loadtest DB; pending on dev/staging/prod:

CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC);

Note

2025-06-10.2-transactions.sql (already shipped) is edited in place, not superseded. Deliberate: wallet-backend is pre-release, and the baseline files stay the single readable statement of each table's final shape.

3. Compression policies all fired at once

All five hypertables' columnstore policies were auto-created on identical schedules and compressed their just-closed chunks concurrently — an I/O storm that took persist p50 from 0.68 s to 3.0 s for 18 of every 60 minutes on the rig. Each policy is now anchored to a distinct slot offset on the schedule grid, so at most one fires at a time; jobs already on their slot are untouched across restarts.

4. insert_into_db now observes once per ledger, at full commit duration

The phase histogram recorded one observation per persist commit, so under batching its counts tracked commits while every other phase tracked ledgers — panels comparing phases (or grading p99 against ledger close time) mixed units.

Each ledger now observes the full, undivided wall time of the commit that carried it:

counts are per-ledger again, comparable with process_ledger;
slow batched commits stay fully visible (never divided by batch size — dividing is what used to hide overload: a 4 s commit for 8 ledgers read as eight healthy 0.5 s shares);
a slow commit weighs in once per ledger it delayed, which is the right weighting for the 0.6 s / 1.0 s grading buckets.

persist_batch_size still carries the coalescing alongside.

Stacked: #679 → #684 → #685 → this → #699. Combined rig result: 18,458 tx/s, process p99 0.992 s; PRs not separately re-benchmarked.

The wallet_db_query_duration_seconds histogram topped out at 0.38s (ExponentialBuckets(0.0001, 2.5, 10)), so every multi-second bulk COPY landed in +Inf and histogram_quantile panels clipped at ~0.38s, under-reporting exactly the queries being optimized. Widen to ExponentialBuckets(0.0001, 3, 12) — 0.1ms through ~17.7s.

…-ledger lookup The oldest-ledger lookup (SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1) has two consumers — backfill gap detection's left bound (GetOldestLedger) and the hourly reconcile_oldest_cursor TimescaleDB job — and neither had an ordered path: the transactions migration dropped TimescaleDB's default partition-column index as consumer-less, an audit that counted WHERE-clause consumers and missed that this query consumes the index's ordering. Without it the planner pulls the first row from every chunk instead of running an ordered ChunkAppend that stops at the oldest (12-187s on an 87GB DB; the hourly job alone accounted for 45% of all DB disk-read time and flushed shared_buffers every run). Keep the default index (fresh databases get it from create_hypertable; already-migrated environments need a one-time manual `CREATE INDEX transactions_ledger_created_at_idx ON transactions (ledger_created_at DESC)`) and drop the to_id tie-break from both query sites: close times increase strictly, so rows sharing a ledger_created_at carry the same ledger_number and the tie-break could only force an incremental sort on top of the index's pathkeys. Covered by a new test that runs the reconciliation job via run_job against out-of-order chunks.

…the schedule interval Each hypertable's columnstore policy is auto-created with an identical schedule, so all five fire at the same instant and compress their just-closed chunks concurrently — an I/O storm that starves the persist stage (measured on the loadtest rig: persist p50 0.68s -> 3.0s for 18 of every 60 minutes). Converge each policy onto a fixed schedule anchored to the interval grid with a distinct per-table slot offset, so at most one policy comes due at a time. Jobs already on their slot are left untouched across restarts, preserving next_start and run history.

…uration Per-commit observations made the series incomparable with process_ledger: counts tracked commits, not ledgers, and the p99 read as 'commit latency' where every other phase reads per-ledger. Each ledger now observes the FULL wall time of the commit that carried it — still never divided by batch size, so slow batched commits stay visible in the ledger-close-time grading buckets, weighted by how many ledgers they delayed.

Copilot

Copilot review overview

🟡 Changes recommended

Add a forward migration for existing databases and handle or reject month/year compression intervals before startup configuration.

Review effort: Lite
Findings: 1 · 2 · 2

Open (5)

Handle or reject month-based intervals before date_bin · New
Add forward migration to restore the dropped ledger index · New
Ship convergence migration for existing databases missing the ledger index · New
Assert query-duration histogram bucket boundaries · New
Test one undivided duration sample per ledger · New

What changed in this PR

This PR improves ingestion observability and TimescaleDB performance across query metrics, ledger lookups, compression scheduling, and persist timing.

Changes:

Expands database query-duration buckets.
Restores the ordered ledger index path and simplifies lookup queries.
Staggers compression jobs.
Records commit duration once per ledger.

File	Summary
`internal/services/ingest_live.go`	Records commit duration per ledger.
`internal/metrics/ingestion.go`	Documents updated phase metric semantics.
`internal/metrics/db.go`	Widens query-duration buckets.
`internal/ingest/timescaledb.go`	Updates ledger lookup and compression scheduling.
`internal/ingest/timescaledb_test.go`	Tests reconciliation and compression scheduling.
`internal/db/migrations/2025-06-10.2-transactions.sql`	Preserves the default transactions index for fresh databases.
`internal/data/ingest_store.go`	Uses the ordered oldest-ledger query.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+			AND js.next_start - $2::interval * $3::float8
+				= date_bin($2::interval, js.next_start - $2::interval * $3::float8, 'epoch'::timestamptz),


 	start := time.Now()
 	oldest, err := db.QueryOne[uint32](ctx, m.DB,
-		`SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC, to_id ASC LIMIT 1`)
+		`SELECT ledger_number FROM transactions ORDER BY ledger_created_at ASC LIMIT 1`)


-- consumers.
-DROP INDEX IF EXISTS transactions_ledger_created_at_idx;
-
 CREATE INDEX idx_transactions_hash ON transactions(hash);


+			// 0.1ms .. ~17.7s: the top buckets must cover multi-second bulk
+			// COPYs (transactions/operations/state_changes at high tx volume),
+			// or histogram_quantile clips every bulk write into +Inf.
+			Buckets: prometheus.ExponentialBuckets(0.0001, 3, 12),


+	for range batch {
+		m.appMetrics.Ingestion.PhaseDuration.WithLabelValues("insert_into_db").Observe(duration.Seconds())


An error occurred while trying to automatically change base from blend/pr6-integration-tests to blend/pr5-graphql August 17, 2026 20:47

aditya1702 force-pushed the ingest-db-observability branch from 58636ea to 65a2623 Compare August 22, 2026 13:37

aditya1702 changed the base branch from blend/pr6-integration-tests to live-ingest-processor-perf August 22, 2026 13:40

aditya1702 self-assigned this Aug 24, 2026

aditya1702 force-pushed the ingest-db-observability branch from 65a2623 to 26b73ed Compare August 25, 2026 18:49

aditya1702 added this to the Wallet Backend O3 Performance Testing milestone Sep 12, 2026

aditya1702 added this pull request to stack #760 September 30, 2026 19:24

aditya1702 added 4 commits October 1, 2026 16:36

Copilot AI lite review requested due to automatic review settings October 1, 2026 21:06

aditya1702 force-pushed the ingest-db-observability branch from 26b73ed to aead158 Compare October 1, 2026 21:06

Copilot started reviewing on behalf of aditya1702 October 1, 2026 21:07

View session

Copilot AI reviewed Oct 1, 2026

View reviewed changes

aditya1702 removed this pull request from stack #760 October 1, 2026 21:18

aditya1702 closed this Oct 1, 2026

aditya1702 reopened this Oct 1, 2026

aditya1702 added this pull request to stack #713 October 1, 2026 21:33

Navigation Menu

Repository navigation

Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling - #686

Fix the DB query-duration buckets, the oldest-ledger index path, and compression scheduling#686
aditya1702 wants to merge 4 commits into
live-ingest-processor-perffrom
ingest-db-observability

aditya1702 commented Aug 13, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		AND js.next_start - $2::interval * $3::float8
		= date_bin($2::interval, js.next_start - $2::interval * $3::float8, 'epoch'::timestamptz),

		for range batch {
		m.appMetrics.Ingestion.PhaseDuration.WithLabelValues("insert_into_db").Observe(duration.Seconds())

Conversation

aditya1702 commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. Query-duration buckets clipped at 0.38 s

2. Oldest-ledger lookup had no ordered path

3. Compression policies all fired at once

4. insert_into_db now observes once per ledger, at full commit duration

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

aditya1702 commented Aug 13, 2026 •

edited

Loading