Repository navigation
Give each write family its own persist sibling - #708
Closed
aditya1702 wants to merge 11 commits into
Closed
aditya1702 wants to merge 11 commits into
aditya1702 wants to merge 11 commits into
Conversation
…lings The live persist path's sibling COPY streams grow from three to five: transactions_accounts and operations_accounts — the two largest index-maintenance payloads, each carrying an account_id-leading unique PK — stream concurrently with their parent tables instead of serially behind them. TransactionModel and OperationModel each split BatchCopy into a parent-table COPY and BatchCopyAccounts for the link table, with the link COPY's duration and batch size recorded under its own metric labels rather than folded into the parent's. Each sibling now writes exactly one table, so the disjointness at the commit barrier is per-table; the backfill path calls the five inserts in the old order and is behaviorally unchanged. Persist holds six connections at its commit barrier; the pool default rises to 12 so the persist path never queues on Acquire behind the advisory-lock session or pool-side classification reads.
… random order The native_balances and trustline_balances batch upserts arrive in Go map-iteration order and unconditionally rewrite every conflicting row. Both cost heap buffer touches far above what the row count warrants and bloat the table with dead tuples. Two changes: rows sort by the primary-key columns before the UNNEST arrays are built, so the btree descent and heap touches run in key order; and the DO UPDATE carries an IS DISTINCT FROM guard, so an identical row produces no new tuple version, no dead tuple, no WAL, and no index churn.
The buffer stored transactions keyed by hash while tracking their participants keyed by ToID. A merged ledger can carry the same envelope at several tx-set positions — distinct ToIDs, one hash — and the two maps then disagreed: duplicate transactions were silently dropped from the transactions COPY while their participant links survived, whose ledger_created_at lookup missed and COPYed the zero timestamp into a year-0001 chunk (observed on the rig: 6 orphaned transactions_accounts rows, 5 missing transaction rows across bootstrap ledgers). Real networks cannot repeat a hash within or across ledgers, so both maps now share the ToID key domain by construction, mirroring the operations pair. BatchCopyAccounts on both link tables now errors on a ToID/opID with no parent row instead of silently writing a zero timestamp.
Batching is opt-in per deployment: at mainnet/testnet cadence the close time comfortably exceeds the persist time, so every ledger commits on its own; high-TPL/short-block deployments raise the flag to amortize backlogs.
…dger's close time DeleteRowsAboveLedger scanned every chunk TOID chunk-skipping stats could not exclude — stats only cover compressed chunks, and three of the five tables have no TOID-leading index. On a database carrying hours of full ledgers that scan runs long enough to hold up startup entirely. Close times are monotone, so every orphan above the cursor carries ledger_created_at at or after the cursor ledger's own close time; binding that as a partition-column predicate statically excludes all older chunks regardless of compression, and turns state_changes' scan into a primary-key range seek. A cursor ledger with no transactions row leaves the bound unresolvable and the deletes fall back to the unbounded scan.
…eachable A buffer stays checked out from the moment the process stage takes it until the batch carrying it commits, so the rotation needs 2*cap buffers to sustain a full batch: cap for the batch in flight, cap-1 queued in processed, one being filled. It held cap+1, so process starved on freeBuffers after filling (cap+1)-k and the batch settled at the fixed point k = (cap+1)-k. The batch size was therefore pinned below the configured cap at every load, never reaching it however deep the backlog. The new test pins the old behaviour: with the pool sized the old way, batches settle below a cap of 3.
…ling The native-balance and liquidity-pool upserts ran serially inside the coordinating transaction, lengthening its critical path while the COPY siblings streamed concurrently. Their tables carry no foreign keys, so a sixth sibling now carries them; the trustline and SAC balances stay on the coordinating transaction because their parent rows (trustline_assets, contract_tokens) are staged uncommitted there, and a sibling committing first would fail FK checks against committed state. Reconciliation (DeleteRowsAboveLedger) does not cover the balance tables: a crash can leave balance rows above the committed cursor, which is safe because the upserts are idempotent and reapply when those ledgers re-ingest.
…bling Each balance family now rides the transaction that stages its foreign-key parents: trustline_assets and trustline_balances move from the coordinating transaction onto a new sibling, alongside the balances sibling that carries pools and pool-share balances under the new pool foreign key. Every FK is checked at its own transaction's commit, and the coordinator's serial path shrinks to contracts, classification, protocol state, SAC balances, and the cursor. SAC balances stay coordinated: their parent (contract_tokens) is also written by the classification path there, and a same-key insert from two concurrent transactions could deadlock at the commit barrier.
… the pool BatchGetByContractIDs queried the pool, so a mid-batch ledger's membership lookup could not see protocol_contracts rows staged earlier in the same batch's uncommitted coordinating transaction. A contract deployed and classified at the batch head, then invoked mid-batch without touching its instance entry, buffered nothing on the later ledger — its events were silently dropped for event-only processors (SEP-41; Blend re-reads through the transaction and was unaffected). Unreachable at the default batch size of 1; the loadtest rig runs 3. The lookup now takes a db.Querier and live ingestion passes the coordinating transaction, mirroring GetByProtocolID.
…mo semantics BatchCopyAccounts on transactions and operations returned a link-row count every caller discarded — the count is already observed into the BatchSize metric inside the method — so both now return plain error, matching the balance models' BatchCopy shape. contractDataMemo's contract gets a unit test: get() always yields a rangeable non-nil map, and the extraction walk runs at most once no matter how many retry attempts share the memo. The startup-reconciliation comment now says a crash orphans at most the persist batch past the cursor; batching had outgrown "the single ledger".
Protocol history rows are state_changes rows, so PersistHistory on the coordinating transaction had two of our own transactions inserting into the same hypertable concurrently. At a chunk boundary TimescaleDB serializes chunk creation and the waiter blocks on the other transaction's end — but the coordinator only commits after every sibling goroutine returns, so if the sibling is the waiter the pipeline hangs forever, invisible to Postgres's deadlock detector. All state_changes writes now go through the one state_changes sibling transaction, serialized by a mutex (pgx.Tx is not concurrency-safe). The CAS stays on the coordinator: siblings commit strictly first, so a committed cursor still implies committed history rows, and rows stranded above the cursor by a crash are already removed by DeleteRowsAboveLedger.
Contributor
Author
|
Reopened as #712 — the branch was renamed to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Start with: the sibling table in
persistLedgerData. This PR is that table growing from 3 entries to 7.One rule throughout: each write family rides the transaction that stages its own FK parents.
Siblings: 3 → 7
transactions_accounts,operations_accountsSAC balances stay on the coordinator— their parentcontract_tokensis also written by the classification path there, and two concurrent same-key inserts could deadlock at the barrier.Protocol history also moves onto the
state_changessibling. It was writingstate_changesrows from the coordinating transaction, so two of our own transactions hit the same hypertable concurrently. At a chunk boundary TimescaleDB serializes chunk creation — and since the coordinator only commits after every sibling returns, a waiting sibling hangs the pipeline forever, invisible to Postgres's deadlock detector.Correctness fixes this surfaced
protocol_contractsrows staged earlier in the same batch, silently dropping events for SEP-41.2*capbuffers, notcap+1— process starved onfreeBuffersand batches settled below the configured cap at every load.Plus data-layer hygiene: sorted +
IS DISTINCT FROM-guarded balance upserts, close-time-bounded startup reconciliation, and batching now defaults to 1 (commit per ledger).No performance numbers yet — review on design and correctness.
5 of 6 splitting #684. schema → fixes → pipeline → seams → write families → hardening