storage: replace the postgres contract blob with a columnar layout - #208
Merged
Merged
Conversation
One row per contract in a new dlc_contracts table. The offer, accept, and sign messages are stored byte for byte, so the TLV streams travel inside them on every state. The manager-only state that the messages do not carry has one typed column per field. ContractRow::from_contract is the one writer and ContractRow::into_contract the one reader; every read path of the store goes through them, and a format_version on the row guards the next layout change. Old databases keep loading. The blob tables stay as the version one layout: the store moves every blob into a row when it opens with migrations on, one transaction per contract, and reports each row it cannot move instead of stopping. A row still in the old layout loads through a fallback and moves on its next update. The store warns at startup and on every legacy read while any contract is left behind, and `ddk-node migrate` runs the move on its own and prints the report. Closes #190.
…nt before the schema exists
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the opaque contract blob in the Postgres store with one row per contract in a new
dlc_contractstable, and migrates existing databases in place. Old rows keep loading, the store moves them on startup, and it warns while any are left. Closes #190.Changes
dlc_contractstable (migration 0010). The offer, accept, and sign wire messages are stored byte for byte, so the TLV streams from messages: keep the TLV records appended to an offer instead of dropping them #188 travel inside them on every state with no suffix and no per-state table. The manager-only state that the messages do not carry has one typed column per field. Aformat_versionon each row guards the next layout change. The down migration refuses to run while the table holds rows.ContractRow::into_contractandContractRow::from_contract, inddk/src/storage/postgres/contract_row.rs. Every read path of the store goes through them. The metadata comes from the wrapped offered contract, because theContractaccessors return zeros for a closed contract and unwrap its CET.ddk/src/storage/postgres/legacy.rs.PostgresStore::migrate_legacy_contractsmoves each blob into a row, one transaction per contract, and reports each row it cannot move instead of stopping.PostgresStore::newruns it when migrations are on. A row still in the old layout loads through a fallback on every read path and moves on its next update. The metadata queries union the old table for rows not yet moved.ddk-node --postgres-url <url> migrateruns the schema migrations and the move on its own, prints the count and every failed id, and exits non-zero if anything is left.OfferedContract::keys_id()getter, since the field is crate private and the row needs it.docs/postgres-contract-migration.md, linked from the README. The follow-up to drop the legacy tables is storage: drop the legacy contract_data and contract_metadata tables #207.Testing
legacy_blobs_round_trip_against_a_live_databasereads every blob in a real database and compares bytes without writing. Run against a dlcd regtest database with 11 contracts: all 11 match. The same database migrated at dlcd startup withmigrated=11 failed=0, and the dlcd CLI reads (get-contracts,get-contract,get-status,get-messages,get-dlc-txns) return correct data for accepted and confirmed contracts.