Skip to content

fix(store): allocate uids in Postgres and enforce uid and Message-ID uniqueness - #53

Merged
TheGreatAxios merged 1 commit into
cl-9434-mailbox-fail-requests-when-a-native-store-write-failsfrom
cl-9435-mailbox-allocate-uids-atomically-and-enforce-uniqueness
Sep 27, 2026
Merged

TheGreatAxios merged 1 commit into
cl-9434-mailbox-fail-requests-when-a-native-store-write-failsfrom
cl-9435-mailbox-allocate-uids-atomically-and-enforce-uniqueness

Conversation

@TheGreatAxios

@TheGreatAxios TheGreatAxios commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Stacked on fix(store): fail the caller when a native-store append fails #52.
  • appendMessage allocates its uid with UPDATE mailbox_state SET uid_next = uid_next + 1 ... RETURNING inside the insert's transaction, as moveNativeMailboxMessage does. That row lock serializes appends to one mailbox.
  • appendMessage takes the host-authorized sender as a fourth argument: writeMailboxMessage passes fromAddress, the send route the caller's sender address, and persist the envelope senderAddress. The insert uses ON CONFLICT ... DO NOTHING on the Message-ID and that sender, and resolves to null on a duplicate (the transaction rolls back, so the counters don't move). A forged From: header therefore cannot suppress another sender's mail. writeMailboxMessage and persist rely on this instead of checking the in-memory mirror first.
  • New migration 0008_unique_uid_and_message_id.sql:
    • adds sender_address text NOT NULL DEFAULT '', backfilled from from_address (rows without one keep '', so the key is never NULL and the dedupe partitions and the index agree);
    • adds a unique index on (tenant_id, principal_id, folder, uid) and a partial unique index on (tenant_id, principal_id, folder, message_id, sender_address). Each step is guarded on the catalog, so replays are idempotent.
  • Existing duplicates are resolved first, without deleting a row:
    • duplicate uids: the oldest row (by created_at, id) keeps its uid; each later one moves past both the mailbox's uid_next and its highest uid, since a racing 0.1.0 append could leave uid_next behind. A mailbox with no mailbox_state row gets one, and uid_next ends past the highest uid.
    • repeated Message-IDs from one sender: every row and its raw frame stay; only the cached message_id on the later copies is cleared (the header is still in raw).
  • schema.ts declares folder, uid, sender_address and both indexes, and schema-check maps bigint.
  • The synchronous MailboxStore.append must return its uid before reaching Postgres, so it cannot allocate atomically; the unique index refuses a colliding write instead of storing a duplicate. Its JSDoc and the README tell hosts to use appendMessage, writeMailboxMessage or createMailboxPersist. Nothing in this package calls it.
  • Trade-off: rows stored before the upgrade are keyed on their old From header (the backfill copies from_address), so a retry of a pre-upgrade message whose From had a display name can be delivered once more. It never suppresses mail.
  • Moving a message into a folder that already holds its Message-ID from the same sender fails on the unique index; fix: skip raw on open, validate folders, decode cached subjects, map move conflicts #55 makes that a 409.

Verification

  • e2e/write.test.ts: 5 concurrent writes give uids 1 to 5, and 3 concurrent writes with one Message-ID give 1 row. Both fail on fix(store): fail the caller when a native-store append fails #52.
  • e2e/persist.test.ts: a forged From: from another envelope sender lands first and the real sender's mail still delivers; a retry with a reformatted From: is deduped.
  • e2e/migrations.test.ts: duplicate uids with uid_next behind the highest uid are renumbered past it, a repeated Message-ID from one sender is cleared while the same id from another sender is kept, and a folder with duplicate uids and no mailbox_state row upgrades and gets a row past its uids.
  • e2e/upgrade-from-0.1.0.test.ts: two 0.1.0 stores racing on one mailbox (uids 1, 1, 2, 3 with uid_next 2) upgrade cleanly to uids 1 to 4 and uid_next 5. The plain 0.1.0 upgrade adds exactly sender_address (equal to from_address) and the two indexes, leaves every other row and column unchanged, and a second replay changes nothing.
  • bun run check, build, test:e2e and the pack smoke pass locally.

Closes CL-9435

@TheGreatAxios
TheGreatAxios added this pull request to stack #50 September 27, 2026 00:56
…uniqueness

appendMessage took its uid from a counter read when the store opened,
and Message-ID dedupe checked the in-memory mirror before inserting, so
concurrent writers produced duplicate uids and duplicate messages. It now
bumps mailbox_state with UPDATE ... RETURNING inside the insert's
transaction, the way moveNativeMailboxMessage does, and inserts with
ON CONFLICT DO NOTHING on the Message-ID and sender, resolving to null on
a duplicate. write and persist rely on that instead of checking first.
appendMessage takes the host-authorized sender: write passes fromAddress,
send the caller's sender address, and persist the envelope sender, so a
forged From header cannot suppress another sender's mail. The
synchronous append cannot allocate atomically; its JSDoc and the README
point hosts at the async write paths.

Migration 0008 adds sender_address (backfilled from from_address, or ''
when a row has none), a unique index on (tenant_id, principal_id,
folder, uid) and a partial unique index on (tenant_id, principal_id,
folder, message_id, sender_address). Existing duplicates are resolved
first without deleting a row: the oldest row keeps its uid and later
ones move past both uid_next and the mailbox's highest uid, since a
racing 0.1.0 append could leave uid_next behind; a mailbox with no
mailbox_state row gets one; and a repeated Message-ID from one sender
keeps its row and raw frame with only the cached message_id cleared.
@TheGreatAxios
TheGreatAxios force-pushed the cl-9435-mailbox-allocate-uids-atomically-and-enforce-uniqueness branch from b3c6feb to 324a401 Compare September 27, 2026 01:23
@TheGreatAxios
TheGreatAxios merged commit e5e92ca into main Sep 27, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant