Skip to content

feat(migrations): replay idempotent SQL files instead of a checksum ledger - #45

Merged
TheGreatAxios merged 3 commits into
cl-9274-mailbox-add-e2e-test-for-sse-live-delivery-and-unsubscribefrom
cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the
Sep 27, 2026
Merged

TheGreatAxios merged 3 commits into
cl-9274-mailbox-add-e2e-test-for-sse-live-delivery-and-unsubscribefrom
cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Sep 25, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Stacked on test: e2e for SSE live delivery, isolation and purge #44.
  • Migrations ship as migrations/*.sql (added to files). runMailboxMigrations replays every file in filename order on each run under an advisory lock, the way Interchange runMigrations and @corbits/memory do. The embedded SQL, migrationChecksum and MigrationChecksumError are deleted.
  • Every column, index and backfill is guarded on the catalog in a DO block. A replay rewrites no row and takes no principal_mail/mailbox_state lock that blocks mail reads or writes.
  • 0004 seeds mailbox_state only when it creates the table, and copies each message's folder and \Seen from a pre-0.1.0 "mailbox"."mailbox" table before 0005 drops it. Because every file replays on each boot, 0005 drops "mailbox"."mailbox" only when its columns are exactly the pre-0.1.0 set, and 0007 drops the ledger only when it has exactly the 0.1.0 columns, so a host's own table under either name survives. 0005 sets uid and modseq NOT NULL, each only while nullable. 0001 no longer creates the pre-native "mailbox"."mailbox" table.
  • 0007_drop_migrations_ledger.sql drops "mailbox"."corbits_mailbox_migrations". A 0.1.0 database upgrades on its next boot with no manual step.
  • CHANGELOG, ARCHITECTURE and CONTRIBUTING describe the upgrade path.
  • The README follows the Corbits README standard, lists each route's grant, and has an upgrade section for 0.1 hosts.
  • An eslint directive is dropped from src/db.ts (the repo has no eslint).

Verification

  • tests/upgrade-from-0.1.0.test.ts: migrates with the published @corbits/mailbox@0.1.0 runner (a devDependency alias), writes mail through 0.1.0's persist, folder move and a \Seen flag, then runs the new runner twice. Every row, column and index stays identical, and the ledger is gone.
  • A migration test rebuilds a pre-0.1.0 database and asserts folder and \Seen survive for read, archived, trashed and untouched rows.
  • A migration test plants a host-owned "mailbox"."mailbox" (the seven pre-0.1.0 state columns plus one of its own) and "mailbox"."corbits_mailbox_migrations", replays, and asserts both survive.
  • Lock regression test: a replay succeeds with lock_timeout = 2s while another transaction holds ROW EXCLUSIVE on both tables.
  • npm pack --dry-run lists all 7 SQL files.
  • CI is green on this branch: install --frozen-lockfile, build, typecheck, tests against Postgres, and the Node consumer smoke test.

Closes CL-9243

@TheGreatAxios TheGreatAxios left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-review: an independent critique pass found a lock regression, where replayed ADD COLUMN/CREATE INDEX took AccessExclusive/Share locks. Fixed with catalog guards and covered by a lock_timeout test. Stale ledger docs and comments are also fixed. Out of scope: databases built by code older than 0.1.0 (with principal_mail but no uid) are not supported. The 0.1.0 runner always applied all six migrations atomically.

@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch 2 times, most recently from 171feed to b215b4e Compare September 25, 2026 17:52
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from b215b4e to f7f8c46 Compare September 25, 2026 22:55
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from f7f8c46 to a57522b Compare September 26, 2026 00:31
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from a57522b to 2c8dbdc Compare September 26, 2026 01:28
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from 2c8dbdc to b52fcdb Compare September 26, 2026 01:36
@TheGreatAxios
TheGreatAxios added this pull request to stack #47 September 26, 2026 01:58
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from b52fcdb to 1aa2597 Compare September 26, 2026 02:40
@TheGreatAxios
TheGreatAxios removed this pull request from stack #47 September 26, 2026 02:40
@TheGreatAxios
TheGreatAxios added this pull request to stack #50 September 26, 2026 02:40
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from 1aa2597 to 7db02fb Compare September 27, 2026 00:45
runMailboxMigrations replays every migrations/*.sql file in filename
order on each run, the same way Interchange runMigrations does, and the
files ship in the tarball. Every column, index and backfill is guarded
on the catalog, so a replay rewrites no row and takes no lock that
blocks mail reads or writes. mailbox_state is seeded only when it is
created, and uid and modseq each become NOT NULL only while nullable.
A pre-0.1.0 database keeps each message's folder and \Seen from its
pre-native management table, which 0005 then drops only when it has
exactly that table's columns. A new migration drops the 0.1.0 ledger,
and MigrationChecksumError is gone.

An upgrade test migrates a database with the published 0.1.0 runner,
writes, files and flags mail through 0.1.0, then runs the new runner
twice and asserts every row, column and index is unchanged.
The README opens with what the package is and where it plugs into
Interchange, lists each route's grant, and adds an upgrade section for
0.1 hosts. Setup, test and internals move to CONTRIBUTING.
@TheGreatAxios
TheGreatAxios force-pushed the cl-9243-mailbox-ship-idempotent-sql-migration-files-and-drop-the branch from 7db02fb to 1c96a52 Compare September 27, 2026 01:23
@TheGreatAxios
TheGreatAxios merged commit 7420564 into main Sep 27, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant