Skip to content

storage: replace the postgres contract blob with a columnar layout - #208

Merged
bennyhodl merged 4 commits into
masterfrom
storage/columnar-contracts
Sep 15, 2026
Merged

bennyhodl merged 4 commits into
masterfrom
storage/columnar-contracts

Conversation

@bennyhodl

Copy link
Copy Markdown
Owner

Summary

Replaces the opaque contract blob in the Postgres store with one row per contract in a new dlc_contracts table, and migrates existing databases in place. Old rows keep loading, the store moves them on startup, and it warns while any are left. Closes #190.

Changes

  • New dlc_contracts table (migration 0010). The offer, accept, and sign wire messages are stored byte for byte, so the TLV streams from messages: keep the TLV records appended to an offer instead of dropping them #188 travel inside them on every state with no suffix and no per-state table. The manager-only state that the messages do not carry has one typed column per field. A format_version on each row guards the next layout change. The down migration refuses to run while the table holds rows.
  • One reader and one writer, ContractRow::into_contract and ContractRow::from_contract, in ddk/src/storage/postgres/contract_row.rs. Every read path of the store goes through them. The metadata comes from the wrapped offered contract, because the Contract accessors return zeros for a closed contract and unwrap its CET.
  • Legacy path in ddk/src/storage/postgres/legacy.rs. PostgresStore::migrate_legacy_contracts moves each blob into a row, one transaction per contract, and reports each row it cannot move instead of stopping. PostgresStore::new runs it when migrations are on. A row still in the old layout loads through a fallback on every read path and moves on its next update. The metadata queries union the old table for rows not yet moved.
  • Warnings at startup and on every legacy read while any contract is left, naming the command to run.
  • ddk-node --postgres-url <url> migrate runs the schema migrations and the move on its own, prints the count and every failed id, and exits non-zero if anything is left.
  • OfferedContract::keys_id() getter, since the field is crate private and the row needs it.
  • Operator doc at docs/postgres-contract-migration.md, linked from the README. The follow-up to drop the legacy tables is storage: drop the legacy contract_data and contract_metadata tables #207.

Testing

  • Row round trip is byte for byte for every fixture, including the pre-marker offered blob, and for the wrapped states: rejected, refunded, failed accept, failed sign. TLV records on all three layers of a closed contract survive the row. An unknown row version and a missing column both error and name the problem.
  • Postgres: migration at startup moves the blobs and empties the legacy tables; a legacy row loads through every read path with migrations off and moves on its first update; an undecodable blob is reported and does not stop startup; TLV records on a seeded legacy blob come through the migration; a contract closed by refund writes null CET and attestation columns and reads back; the offers list holds only offers we received.
  • Ignored live check legacy_blobs_round_trip_against_a_live_database reads every blob in a real database and compares bytes without writing. Run against a dlcd regtest database with 11 contracts: all 11 match. The same database migrated at dlcd startup with migrated=11 failed=0, and the dlcd CLI reads (get-contracts, get-contract, get-status, get-messages, get-dlc-txns) return correct data for accepted and confirmed contracts.
  • Clippy is clean for the changed files, including the store tests.

One row per contract in a new dlc_contracts table. The offer, accept,
and sign messages are stored byte for byte, so the TLV streams travel
inside them on every state. The manager-only state that the messages do
not carry has one typed column per field. ContractRow::from_contract is
the one writer and ContractRow::into_contract the one reader; every read
path of the store goes through them, and a format_version on the row
guards the next layout change.

Old databases keep loading. The blob tables stay as the version one
layout: the store moves every blob into a row when it opens with
migrations on, one transaction per contract, and reports each row it
cannot move instead of stopping. A row still in the old layout loads
through a fallback and moves on its next update. The store warns at
startup and on every legacy read while any contract is left behind, and
`ddk-node migrate` runs the move on its own and prints the report.

Closes #190.
@bennyhodl
bennyhodl merged commit fedabee into master Sep 15, 2026
138 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage: replace the opaque postgres contract blob with a versioned, column-based schema

1 participant