Repository navigation
Pipeline live ingestion into fetch/process/persist with seven parallel write streams - #684
Closed
aditya1702 wants to merge 48 commits into
Closed
aditya1702 wants to merge 48 commits into
aditya1702 wants to merge 48 commits into
Conversation
This was referenced Aug 13, 2026
Open
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
2 times, most recently
from
August 20, 2026 17:23
f2ae385 to
68d3825
Compare
A dev-only LEDGER_BACKEND_TYPE=streaming-loadtest that reads stream-framed LedgerCloseMeta from named pipes written by stellar-core apply-load (one FIFO per transaction profile), renumbers each stream onto the consumer's requested sequence with per-pipe diffs, merges the per-sequence frames into one mixed-traffic ledger via the SDK's loadtest.MergeLedgers, and stamps monotone wall-clock close times (apply-load emits closeTime 0). Renumbering makes both sides restartable without a database reset: a restarted apply-load resets to raw sequence 1 and is mapped onto the next requested ledger; a restarted consumer resumes from its cursor. Pacing via --loadtest-ledger-close-duration bounds the ledger rate, and FIFO backpressure throttles the generators to match. Because apply-load's benchmark mode publishes no history archive, this backend type skips the archive connection and the cursor-0 checkpoint bootstrap: ingestion starts from ledger 1 on an empty database and balance state accumulates from the ledger stream. Everything downstream of the backend (live ingest loop, processors, persistence) is unchanged, so a load test exercises the same code path as production ingestion. Includes an opt-in corpus test (STREAMING_LOADTEST_CORPUS) that replays real apply-load output through the backend and the production transaction reader; verified against v27 sac/custom_token/soroswap corpora. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Renumber only the ledger header. The per-entry rewrite of lastModifiedLedgerSeq/TTL/seqLedger was a full XDR marshal/parse/walk/re-marshal round trip over every frame, and nothing in wallet-backend reads those fields — measured at ~5x the entire ledger-processing cost at full per-ledger volume (the stream's cadence was bound by it), and a 37x speedup on the corpus replay test once removed. Merging now appends transaction-set phases and results directly instead of going through the SDK's renumbering merge, which also drops the goxdr dependency. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…data Two fixes for protocol-framework behavior surfaced by the loadtest rig: Protocol state production (SEP-41 balances/state changes) is gated by per-protocol cursor rows that only the protocol-migrate CLI creates, and that CLI needs a replayable ledger source. A streaming-loadtest deployment has none, so a fresh database silently skipped every protocol processor. The archive-less bootstrap now seeds each registered protocol's cursors at startLedger-1 — exactly the value the first ledger's compare-and-swap expects — and refreshes the cursor snapshot so production is live from ledger 1. The SEP-41 metadata fetcher re-fetched failing contracts on every ledger they were active in: claimed contracts re-enter Prefetch each classification pass, and a persistently failing name() simulation costs its full retry-with-backoff (600ms of sleep) every time. A negative cache now skips recently failed contracts for 5 minutes, keeping eventual enrichment while removing the per-ledger tax. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prefetch runs for every claimed contract observed in a ledger's changes and has no database access by design, so it re-simulated name/symbol/decimals for tokens whose metadata was already persisted — on every ledger that touched their instance. An in-process success cache skips them; a restart refetches each contract once, which doubles as the refresh path for tokens whose on-chain metadata changed. Persist retries are unaffected: the fetched values live in the classification plan, which is reused across retry attempts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Each pipe's reader now buffers two decoded frames beyond the one in flight, so a writer streams its next frames through the FIFO while the consumer merges and processes earlier ledgers. With lockstep delivery every GetLedger waited on the slowest writer's in-flight frame — a constant ~0.6s/ledger at full volume that the one-frame handoff could not hide. Backpressure still bounds the writers, now with three frames of slack; an epoch's terminating error stays ordered behind its frames, so restart handling is unchanged.
…allel apply-load's meta write is synchronous: core does not start generating its next ledger until the consumer drains the current frame, and the reader drained only as fast as it decoded — putting XDR decode inside every producer's ledger cycle (measured: cycle = generation + decode, which capped the merged stream well under target cadence). Each pipe's reader now slurps a record's raw bytes at transfer speed and hands them to a per-pipe decode goroutine, so the producer starts its next ledger while the previous frame decodes. Frame and error ordering are preserved: a drain or decode error is always the last element delivered and ends the epoch exactly as before.
The streaming-loadtest backend's meta sources generalize beyond FIFOs: a tcp-listen://HOST:PORT entry binds a listener eagerly at construction and serves each producer connection as one stream epoch, so apply-load producers can run in their own pods and dial in. Connection close is the epoch boundary (as FIFO EOF is), the listener outlives epochs to serve producer restarts, and keepalive surfaces a vanished peer as a read error. Frame draining, decoding, renumbering, and merging are shared between both source kinds; entry order still defines merge order. The flag is now --loadtest-meta-sources / LOADTEST_META_SOURCES.
Each buffered lookahead frame is a fully decoded LedgerCloseMeta — tens of MB of pointer-dense XDR per source, ~600MB resident across 12 sources per lookahead unit — and that mass is scanned by every GC cycle. One frame of lookahead still overlaps the writer's streaming with the consumer's processing; the second bought no cadence and cost scan time.
The metadata fetcher's fetched/failure caches are in-memory only, so a token whose contract_tokens row already carries metadata was still re-fetched over RPC — once per process lifetime after every restart on a real network, and forever on a 5-minute backoff against a deployment whose RPC can never resolve metadata (the loadtest rig's dead endpoint with externally seeded rows), where each retry burned the ~850ms simulate backoff ladder inside prepare_classification. After Apply persists a batch's rows, contracts whose row has metadata are marked fetched, making the database the durable cache. Tokens whose metadata is genuinely missing stay unmarked and keep the existing backoff retry.
apply-load pins its ledger protocol to the core binary's current version (neither LEDGER_PROTOCOL_VERSION nor its testing override is settable from a config file), and every protocol since 23 emits V2 meta, so the V1 arms in appendLedger/setLedgerSeq/stampCloseTime were unreachable. Collapse the header rewrites onto one mutableHeader accessor that owns the V2-only invariant, rebuild the test fixtures on V2 (the shape production actually sees), and pin explicit rejection of V0 and V1 streams.
The fetched/failedUntil caches sit on the production ingestion path but had no coverage: nothing verified that a resolved contract stops hitting the RPC, that a failed fetch is suppressed for the backoff window, or that expiry re-fetches and success clears the failure entry. Add those tests; shortening metadataFailureBackoff here is what makes its var-for-tests comment true.
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
from
August 20, 2026 17:31
68d3825 to
745220e
Compare
aditya1702
force-pushed
the
live-ingest-persist-path
branch
from
August 20, 2026 19:35
e123d77 to
d5acde6
Compare
apply-load runs always number their ledgers up from raw 2, so the raw sequence stepping backwards inside one byte stream can only mean a new run's frames arrived without an intervening EOF — which is an epoch boundary that skipped the close/reopen handshake. A corpus replayer produces exactly this when it reattaches its pipe before the reader drains to EOF: the two runs' bytes concatenate frame-aligned, and the previous hard error escalated a benign boundary into fetch-retry exhaustion and a process exit. Re-derive the sequence diff exactly as a reopen would and continue; forward gaps inside an epoch still fail hard, since those mean frames were genuinely lost. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…flag ingestService carried four fields (appTracker, getLedgersLimit, knownContractIDs, contractMetadataService) and Indexer two (ingestionMetrics, networkPassphrase) that were assigned at construction and never read — consumers get these through their own constructors or through internal/ingest.Configs. Deleting them also drops AppTracker, GetLedgersLimit, and ContractMetadataService from IngestServiceConfig (the checkpoint service keeps its own metadata-service wiring). Also removes the --latest-ledger-cursor-name flag, which only fed a deprecation warning, and the phantom load_current_state phase label from the ProtocolStateProcessingDuration help text (never emitted).
Every remaining parameter of persistLedgerData had exactly one production value: cursorName was always the latest-ledger cursor (the blind-Update branch was reachable only from tests), ledgerMeta was never nil, and the numTxs/numOps returns only fed two counters the buffer already knows via GetNumberOfTransactions/GetNumberOfOperations. The signature is now (ctx, ledgerSeq, ledgerMeta, plan, contractData, buffer) error, the cursor write is unconditionally guarded, and the contractDataMemo nil-receiver branch (test-only) is gone. IndexerBufferInterface shrinks to its real seam — the indexer's fold-in plus the five getters insertIntoDB reads; everything else is reached on the concrete *IndexerBuffer. Tests now exercise the guarded cursor path persistLedgerData actually runs in production.
…Backoff persistLedgerDataWithRetry (né ingestProcessedDataWithRetry) hand-rolled the same attempt loop, exponential backoff, cap, and context handling that utils.RetryWithBackoff already provides — the body is now a single RetryWithBackoff call with isPermanentPersistError as the classifier. Metric semantics preserved: transient retries count RetriesTotal, permanent errors count ErrorsTotal and return immediately, exhaustion counts both plus RetryExhaustionsTotal, and context cancellation counts neither. RetryWithBackoff's exhaustion return now wraps the exported sentinel utils.ErrRetriesExhausted alongside the final error, which is how the wrapper tells exhaustion apart from a permanent-error exit.
…ist stages The live loop ran fetch (~0.3s of waiting), process_ledger (~0.8s of Go CPU), and the DB persist (~1.1s) strictly in sequence, so the per-ledger time was their sum — above a 2s ledger cadence at high transaction volume even though each stage individually fits. The loop is now a three-stage errgroup pipeline over depth-1 channels: while ledger N persists, N+1 processes and N+2 is fetched, making the ledger time the slowest stage instead of the sum. Persist stays strictly sequential in ledger order (the guarded cursor and per-protocol CAS chain advance N-1 → N) and is the only writing stage, so the advisory-lock liveness probe moves there. Classification planning also stays on the persist stage: its known-hash lookup is a non-transactional pool read that must observe the previous ledger's committed protocol_wasms rows. Two IndexerBuffers rotate through a freeBuffers channel, cleared only at take so a clear can never tear a persist still reading the maps the getters alias. Any stage error cancels the pipeline and the process exits, resuming from the cursor on restart — the Duration metric now sums the ledger's stage times, since wall time under a pipeline would count queue wait.
…with coordinated late commits The live persist ran its three COPY families — transactions(+accounts), operations(+accounts), state_changes — sequentially inside one transaction on one connection, so the insert phase cost their sum and a single Postgres backend did all the index maintenance. They now stream concurrently on three sibling connections, each in its own transaction, while the coordinating transaction stages everything else (assets, contracts, classification, protocol state, token changes, cursor). The table sets are disjoint with no FKs among them, and every goroutine only reads the quiescent buffer. Commits are the visibility point and are held until all four transactions have done their work: siblings commit first (sub-ms each), the coordinating transaction — whose cursor is the authority on which ledgers exist — strictly last. A failure before the first commit rolls everything back and stays retryable exactly as before; a failure after it wraps the new ErrPartialPersist sentinel, which isPermanentPersistError classifies as fatal, because COPY has no ON CONFLICT and re-running the ledger would collide on primary keys. The only crash state this ordering can produce is orphaned bulk rows for the single ledger past the committed cursor, so startup runs IngestStoreModel.DeleteRowsAboveLedger before resuming: one transaction of TOID-bounded deletes (rows of ledgers > cursor are exactly rows with TOID >= toid.New(cursor+1,0,0)) across the five bulk tables, kept chunk-local by each table's chunk skipping. Backfill keeps the single-transaction insertIntoDB path unchanged.
…ommits When the process stage finishes ledgers faster than persist drains them, the persist stage now folds up to --live-persist-max-batch-size (default 5) consecutive ledgers into one commit set: each sibling COPY streams the whole batch on its connection, the coordinating transaction stages the batch's ledgers in order, and the commit barrier fires once — amortizing COPY setup, index-page churn, and fsyncs across the backlog. While the pipeline keeps pace every batch has size 1 and behavior is unchanged. A ledger with classification inputs always opens its own batch: its plan's pool reads see exactly what the previous batch committed, which preserves the deployed-contract-sees-prior-wasm invariant. The cursor stays the authority — one guarded update lands on the batch's last ledger, and a pre-commit failure rolls back and retries the whole batch. wallet_ingestion_persist_batch_size observes coalescing; per-ledger duration histograms record each ledger's amortized share so panels stay comparable across batch sizes.
…on WAL flush SET LOCAL synchronous_commit = off on each sibling session removes up to three serialized fsync waits per persist. Durability is unchanged: the coordinating transaction commits synchronously and strictly last, and its flush covers all earlier WAL including the sibling commit records, so a durable cursor implies durable siblings; rows a crash could lose are exactly the unacknowledged ones startup reconciliation deletes.
The classification-safety cut isolated any ledger carrying protocol wasm/contract observations, but a plan's pool reads are only unsound for inputs no committed batch has classified yet — and re-observations of known contracts are the overwhelmingly common case (synthetic loadtest traffic re-observes the same token contracts every ledger, which cut every batch to size one and disabled batching entirely; goroutine dumps showed the process stage starved of buffers held by the always-cut pending queue). The persist goroutine now keeps seen-sets of wasm hashes and contract IDs folded in after each successful commit; only a ledger introducing unseen inputs opens its own batch. A rolled-back batch marks nothing, and a restart starts empty — conservative until re-warmed.
…ary key TOID encoding makes a state change's parent transaction to_id derivable from its operation_id (to_id = operation_id &^ 0xFFF), so BatchGetByOperationID and BatchGetByOperationIDs now bind (ledger_created_at, to_id, operation_id) — a primary-key prefix seek — instead of filtering operation_id across every entry sharing the timestamp (measured 7 vs 307 buffers on a single-chunk fixture). That leaves idx_state_changes_operation_id with no consumer; it is removed. idx_state_changes_account_category narrows to idx_state_changes_account_id (account_id + the PK sort key): the category and reason columns only ever seeded an ordered seek for the one filter shape that pins both, while every other shape and BatchGetAccountStateChangesByToIDs seek on account_id alone. Filtered variants scan-and-filter the account's rows on the active chunk and prune compressed chunks via the existing bloom sparse indexes. Two fewer btree columns per insert on the highest-volume table. Already-migrated DBs need the one-time manual DDL (drop both old indexes, create the narrowed one); TimescaleDB rejects CREATE INDEX CONCURRENTLY on hypertables.
…lings The live persist path's sibling COPY streams grow from three to five: transactions_accounts and operations_accounts — the two largest index-maintenance payloads, each carrying an account_id-leading unique PK — stream concurrently with their parent tables instead of serially behind them. TransactionModel and OperationModel each split BatchCopy into a parent-table COPY and BatchCopyAccounts for the link table, with the link COPY's duration and batch size recorded under its own metric labels rather than folded into the parent's. Each sibling now writes exactly one table, so the disjointness at the commit barrier is per-table; the backfill path calls the five inserts in the old order and is behaviorally unchanged. Persist holds six connections at its commit barrier; the pool default rises to 12 so the persist path never queues on Acquire behind the advisory-lock session or pool-side classification reads.
…mo semantics BatchCopyAccounts on transactions and operations returned a link-row count every caller discarded — the count is already observed into the BatchSize metric inside the method — so both now return plain error, matching the balance models' BatchCopy shape. contractDataMemo's contract gets a unit test: get() always yields a rangeable non-nil map, and the extraction walk runs at most once no matter how many retry attempts share the memo. The startup-reconciliation comment now says a crash orphans at most the persist batch past the cursor; batching had outgrown "the single ledger".
Protocol history rows are state_changes rows, so PersistHistory on the coordinating transaction had two of our own transactions inserting into the same hypertable concurrently. At a chunk boundary TimescaleDB serializes chunk creation and the waiter blocks on the other transaction's end — but the coordinator only commits after every sibling goroutine returns, so if the sibling is the waiter the pipeline hangs forever, invisible to Postgres's deadlock detector. All state_changes writes now go through the one state_changes sibling transaction, serialized by a mutex (pgx.Tx is not concurrency-safe). The CAS stays on the coordinator: siblings commit strictly first, so a committed cursor still implies committed history rows, and rows stranded above the cursor by a crash are already removed by DeleteRowsAboveLedger.
Mid-batch ledgers run without a classification plan, but getEffectiveProtocolContracts treated the nil plan as 'nothing is classified': any committed contract re-observed in the buffer (a TTL bump, restore, or instance-storage write buffers it again) was dropped from membership and never re-added, so that ledger's events for it were silently discarded. nil classification now means 'no reclassification' — committed membership stands. That is sound because the batch cut is made binding-aware in the same change: classifiedContracts records the wasm hash each contract was last seen bound to, so a contract upgrading to a different wasm — even an already-classified one — counts as an unclassified input and opens a batch head with a real plan. Mid-batch ledgers therefore carry zero binding changes and their buffered entries are pure re-observations. Regression tests: a two-ledger batched persist where the mid-batch ledger re-observes a committed contract and still stages its events, plus batch-cut coverage for the binding-change case.
The batch cut's seen-sets mark inputs on commit regardless of the classification outcome. Document why that is sound — verdicts are deterministic or fail-fast, so no transient failure can hide behind a seen input — and name the one deliberate trade: best-effort RPC enrichment (SEP-41 token metadata) retries on classification passes, which pause under a sustained persist backlog and resume when the pipeline catches up. Gating seen-ness on enrichment instead would cut a batch head per re-observation of any permanently-unfetchable token and disable batching on RPC-less deployments.
Sibling and coordinator commits ran on the cancellable pipeline context, so a SIGTERM (or another stage's failure cancelling the errgroup) landing after the first sibling committed aborted the remaining commits and wrapped the routine shutdown in a fatal ErrPartialPersist — forcing the next start through DeleteRowsAboveLedger and firing the db_persist error counter for a healthy rolling restart. The window widens linearly with batch size. The barrier now runs on context.WithoutCancel: a batch commits fully or not at all. Cancellation before the barrier still rolls the whole batch back, and the cancellation-counts-against-no-counter contract in the retry ladder's error switch holds again. Also updates the stale root-context comment in ingest.go that still described the pre-pipeline single-ledger atomicity.
Three deploy-safety fixes for the persist pipeline's schema footprint: - Forward migration for the state_changes index swap. 2025-06-10.4 was edited in place to trade idx_state_changes_operation_id and idx_state_changes_account_category for idx_state_changes_account_id, but sql-migrate keys applied migrations by filename, so environments already past that file would keep the old shape forever while fresh builds got the new one. The new migration converges them with idempotent statements that no-op on fresh databases. - The liquidity_pool_balances FK now applies as NOT VALID + VALIDATE under notransaction, with an orphan-cleanup DELETE first: the plain ADD CONSTRAINT validated every row under a write-blocking lock and a single pre-existing orphan (a reachable state under the old read-time join) would abort the migration mid-deploy. A DROP IF EXISTS guard keeps the unwrapped migration re-runnable after a partial failure. - Live ingestion fails fast when the pool cannot fit the persist barrier: it pins 9 connections (7 siblings + coordinator + advisory lock), and pgxpool.Acquire has no timeout, so a deployment pinning DB_MAX_CONNS at the pre-pipeline value of 8 or below wedged silently — the last Acquire waited forever on connections its own goroutine held.
… the seen-sets Three observability/robustness fixes on the persist stage: - insert_into_db (and its 0.6s/1.0s grading buckets) now observes one true wall-time value per persist commit instead of len(batch) amortized shares. Batching engages exactly when persist has fallen behind, so the dilution reported the overloaded condition as healthy — a 4s commit for 8 ledgers landed as eight 0.5s observations. The per-ledger total duration keeps amortized shares so ledger durations stay comparable. - The classification seen-set fold is skipped at the default --live-persist-max-batch-size=1, where the batch cut it feeds can never exceed 1, and the sets are bounded by a cap-and-clear (a miss only makes one batch cut conservative). Previously they grew by one entry per distinct contract ever observed, for the process lifetime, on every deployment. - The commit barrier documents the deliberate torn-read window between sibling commits (readers can briefly see a transaction without its participant links or state changes; the cursor still commits last).
DeleteRowsAboveLedger falls back to deleting with no partition-column bound when the cursor ledger left no transactions row — correct but a full scan of every chunk of the five bulk tables. Log the degenerate path so an operator can attribute a slow start to it.
TransactionModel.BatchCopyAccounts and OperationModel.BatchCopyAccounts were the same function twice — same row shape, same zero-timestamp guard, same metrics — differing only in table name, ID column, and parent type. One generic helper now serves both; the wrappers keep their public signatures.
The pool path skips liquidity pool bodies that liquidity_pools cannot represent (non-constant-product), but shares were captured unconditionally — and a share row without its pool aborts the whole checkpoint load at commit under the liquidity_pool_balances → liquidity_pools FK. A share's pool entry can arrive in any bucket, so shares are held until every pool entry has been read and joined to the final flush minus the skipped pools'. Covered by a regression test with an unsupported pool body.
The five bulk-COPY tables were enumerated independently in startup reconciliation, the hypertable settings pass, the backfill recompressor, and a test's own copy — and a table missing from DeleteRowsAboveLedger leaves orphans that crash-loop re-ingest on PK collisions. data.BulkCopyTables (table + TOID column) is now the single definition all of them consume.
…sist entry The queue between the process and persist stages carried each ledger's full decoded LedgerCloseMeta and its materialized transactions, pinning them for the pipeline depth though persist needs only the close time and the ContractData memo — both now derived at the end of the process pass. The memo keeps its laziness: extraction still runs only when a CAS-winning processor requires contract data. persistLedgerData materializes the buffer's transaction and operation slices once per ledger (two siblings consume each), rejects ledger sequence 0 at entry (the CAS chain and cursor guard both address ledgerSeq-1, which underflows unsigned), and its sibling set moves to a named constructor pinned by a test to cover data.BulkCopyTables exactly.
Same facts, easier shape: each block leads with the rule, then the why — the state_changes single-transaction rule, the commit barrier's three properties, the seen-set marking contract and its enrichment trade, the shutdown behavior split by pipeline position. Also detaches the seen-set cap's comment from markClassificationInputsSeen's doc block, which had fused into one comment attached to the wrong declaration.
aditya1702
force-pushed
the
live-ingest-persist-path
branch
from
August 22, 2026 13:40
fade6bb to
2d00170
Compare
aditya1702
force-pushed
the
replay-loadtest-backend-pr6
branch
2 times, most recently
from
August 25, 2026 22:11
646a06b to
9b51e14
Compare
Catches #684 up with its base, which gained `protocol-migrate --rebuild` (#692) and hot-archive checkpoint reads (#694). The base's four SEP-41 metadata commits are already on this branch under different SHAs from the #679 review round, so they arrive as no-ops. Conflicts, all in the ingest configuration surface, resolved as the union of two independent deletions rather than by picking a side. Both branches were removing configurable ledger-cursor names from opposite ends: this branch dropped the inert `latest-ledger-cursor-name` flag along with the dead `AppTracker` and `GetLedgersLimit` fields, while #692 dropped the `oldest-ledger-cursor-name` flag and hard-coded `data.OldestLedgerCursorName`. Neither cursor name is configurable now. - cmd/ingest.go: both flag blocks removed; `deprecatedLatestLedgerCursorName` no longer had a declaration to bind to. - internal/ingest/ingest.go: `IngestServiceConfig` has neither `OldestLedgerCursorName` nor `AppTracker`, so both assignments are gone. The ingest-package config keeps its own fields, which other callers use. - internal/services/ingest_live.go: kept the pipelined persist path. The base's side of this hunk was the sequential `ingestProcessedDataWithRetry` body this branch replaced; its only real change, the switch to `data.OldestLedgerCursorName`, is ported into the relocated cursor-sync block. - ingest_test.go, ingest_live_test.go: dead fields dropped from the `IngestServiceConfig` literals. `processHotArchive` reaches `hotArchiveIterFactory` through the service, and TestCheckpointService_PopulateFromCheckpoint_UnsupportedLiquidityPoolBodySkipsShares is new on this branch, so #694 never wired the factory into its fixture and the pass nil-panicked. It now gets the empty iterator the neighbouring checkpoint tests use. go build, go vet, make check and make unit-test all pass.
The backend replayed stellar-core apply-load meta from named pipes or TCP
listeners so live ingestion could be driven at synthetic block rates. That
rig now runs from its own deployment, so the backend, its flags, and the
product code that existed only to serve it come out of the service.
Removed with it:
- the archive-less bootstrap path. archive-url is required again, and
live ingestion always starts from a history-archive checkpoint, so the
cursor-seeding branch that started an empty database at ledger 1 is
unreachable.
- SetConfigOptionStringList, whose only consumer was the meta-sources
flag.
- protocol-migrate's rejection case for the backend.
The indexer buffer keeps keying canonical transactions by ToID. The
invariant it protects — every ToID carrying a participant entry has its
canonical row — is a property of the key domain, not of any one backend,
so only the rationale is restated.
This was referenced Aug 26, 2026
Contributor
Author
|
Superseded by six smaller PRs. Same work, same final tree — Review them in order:
The streaming-loadtest ledger backend was removed before the split (it now runs from its own deployment), which is why this PR shows 44 files rather than the original 51. The branch stays put — #685 is still based on it. |
This was referenced Aug 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
How to review (~2 h):
What this does: the three ingestion stages ran one after another, so a ledger cost the sum of the stages (7.7–8.4 s on the rig; 8.2 s of that was serialized inserts). Now the stages overlap and persist writes through seven parallel transactions. A ledger costs only the slowest stage.
Also fixed along the way: one silent transaction-loss bug, three pathological DB write patterns, an undetectable ingestion-hang class, two batched-mode silent-loss bugs.
Why now: block time is headed 2 s → 1 s → 600 ms at constant volume. The contract: the slowest stage's p99 stays under block time.
The pipeline
While ledger N persists, N+1 processes and N+2 fetches.
seq-1underflows unsigned).LedgerCloseMetaand materialized transactions never ride it.Metrics:
wallet_ingestion_duration_secondssums stage times (wall time would count queue wait)...._phase_duration_seconds{phase="insert_into_db"}records the true wall time of each persist commit — never divided by batch size. Its 0.6 s / 1.0 s grading buckets read real commit latency exactly when persist is behind.persist_batch_sizecarries the coalescing.Seven sibling write streams
Each sibling owns its own tables. Every FK is checked inside the transaction that writes its parent rows.
transactions,transactions_accounts,operations,operations_accounts,state_changesnative_balances,liquidity_pools,liquidity_pool_balancestrustline_assets,trustline_balancesThe five bulk tables have one authoritative definition:
data.BulkCopyTables. Siblings, startup reconciliation, hypertable settings, and the backfill recompressor all consume it. A test pins the sibling set to the list — a table missing from reconciliation would crash-loop re-ingest on PK collisions.state_changeshas two producers, one transaction. Protocol history rows arestate_changesrows too:state_changessibling transaction, under a mutex.Why: two of our own transactions inserting into the same hypertable can deadlock undetectably at a chunk boundary. TimescaleDB serializes chunk creation while the coordinator is blocked in Go — where Postgres's deadlock detector can't see the cycle. The CAS stays on the coordinator: siblings commit strictly first, so a committed protocol cursor still implies committed history rows.
The commit order is the whole safety story:
ErrPartialPersist). COPY has noON CONFLICT; a retry would collide.context.WithoutCancel. A SIGTERM mid-barrier cannot split the batch into a fake fatalErrPartialPersist— a batch commits fully or not at all.DeleteRowsAboveLedgerremoves bulk rows past the cursor (protocol history included — same table, same TOID predicate). Balance upserts are idempotent. The rare unbounded-scan fallback now logs itself.Three deliberate trades:
synchronous_commit = off. Durability unchanged — the coordinator's synchronous commit flushes all earlier WAL. Removes up to 7 serialized fsync waits per persist.Batching (off by default)
--live-persist-max-batch-size, default 1 = exactly the old behavior. Mainnet/testnet keep 1; only the loadtest rig raises it. When persist falls behind, up to N ledgers fold into one commit set.Four rules keep it sound:
Housekeeping: the seen-sets are skipped entirely at batch size 1 and bounded by cap-and-clear (a miss only makes one cut conservative).
One accepted trade, documented in code: best-effort RPC enrichment (SEP-41 token metadata) retries on classification passes, which pause under a sustained backlog and resume when the pipeline catches up. Gating batches on enrichment would disable batching wholesale on RPC-less deployments like the rig.
DB write-pattern fixes
IS DISTINCT FROM: an identical row writes nothing.to_idderivable fromoperation_id, so lookups bind the PK prefix: 7 buffers instead of 307. Index swap ships as a real forward migration — see the migration note below.liquidity_pool_balances.pool_id → liquidity_pools. An orphaned pool-share balance is always an ingestion bug; now it fails loudly at write time. Applied deploy-safely: orphan-cleanupDELETE→ADD CONSTRAINT ... NOT VALID(brief metadata-only lock) →VALIDATEin its own transaction, so ingestion keeps writing through the scan and a pre-existing orphan can't halt a deploy. The checkpoint loader was made symmetric: share rows of pools the pool capture skips are dropped with their pool.Correctness fix: transactions were keyed by hash
Participants were keyed by ToID, transactions by hash. Merged loadtest ledgers can repeat one envelope at several positions — distinct ToIDs, one hash. The duplicates were silently dropped and their participant links COPYed a zero timestamp into a year-0001 chunk.
Fix: both maps share the ToID key, and
BatchCopyAccounts(now one shared helper for both link tables) errors on a link row with no parent. Real networks can't repeat a hash — latent in production, live only on the rig.Also in here
GOMAXPROCS, notNumCPU— a 6-core pod on a 36-core node ran 72 workers on 6 threads.pgxpool.Acquirehas no timeout, so live ingestion now fails fast at startup if--db-max-connscan't fit the barrier — instead of self-deadlocking silently.Measured
BatchGetByOperationIDbuffersBackfill is untouched (keeps the single-transaction path).
Note
No manual DDL needed.
2025-06-10.4-statechanges.sql(already shipped) is edited in place so the baseline file stays the single readable statement of the table's final shape. A new idempotent migration (2026-08-21.0) converges environments that applied the old shape: createsidx_state_changes_account_id, drops the two superseded indexes, no-ops on fresh databases and on the loadtest DB (already swapped by hand).One caveat: TimescaleDB rejects
CREATE INDEX CONCURRENTLYon hypertables, so the CREATE takes a write-blocking lock for the build. On an environment with a deepstate_changes, run the migration step in a maintenance window.Second of four PRs from the loadtest campaign (replaces #682). Review order: #679 → this → #685; #686 is independent. Combined rig result: 18,458 tx/s, process p99 0.992 s; PRs not separately re-benchmarked. This PR took the insert phase off the critical path.