fix(store): allocate uids in Postgres and enforce uid and Message-ID uniqueness - #53
Merged
TheGreatAxios merged 1 commit intoSep 27, 2026
Conversation
TheGreatAxios
added this pull request to stack #50
September 27, 2026 00:56
…uniqueness appendMessage took its uid from a counter read when the store opened, and Message-ID dedupe checked the in-memory mirror before inserting, so concurrent writers produced duplicate uids and duplicate messages. It now bumps mailbox_state with UPDATE ... RETURNING inside the insert's transaction, the way moveNativeMailboxMessage does, and inserts with ON CONFLICT DO NOTHING on the Message-ID and sender, resolving to null on a duplicate. write and persist rely on that instead of checking first. appendMessage takes the host-authorized sender: write passes fromAddress, send the caller's sender address, and persist the envelope sender, so a forged From header cannot suppress another sender's mail. The synchronous append cannot allocate atomically; its JSDoc and the README point hosts at the async write paths. Migration 0008 adds sender_address (backfilled from from_address, or '' when a row has none), a unique index on (tenant_id, principal_id, folder, uid) and a partial unique index on (tenant_id, principal_id, folder, message_id, sender_address). Existing duplicates are resolved first without deleting a row: the oldest row keeps its uid and later ones move past both uid_next and the mailbox's highest uid, since a racing 0.1.0 append could leave uid_next behind; a mailbox with no mailbox_state row gets one; and a repeated Message-ID from one sender keeps its row and raw frame with only the cached message_id cleared.
TheGreatAxios
force-pushed
the
cl-9435-mailbox-allocate-uids-atomically-and-enforce-uniqueness
branch
from
September 27, 2026 01:23
b3c6feb to
324a401
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
appendMessageallocates its uid withUPDATE mailbox_state SET uid_next = uid_next + 1 ... RETURNINGinside the insert's transaction, asmoveNativeMailboxMessagedoes. That row lock serializes appends to one mailbox.appendMessagetakes the host-authorized sender as a fourth argument:writeMailboxMessagepassesfromAddress, the send route the caller's sender address, and persist the envelopesenderAddress. The insert usesON CONFLICT ... DO NOTHINGon the Message-ID and that sender, and resolves tonullon a duplicate (the transaction rolls back, so the counters don't move). A forgedFrom:header therefore cannot suppress another sender's mail.writeMailboxMessageand persist rely on this instead of checking the in-memory mirror first.0008_unique_uid_and_message_id.sql:sender_address text NOT NULL DEFAULT '', backfilled fromfrom_address(rows without one keep'', so the key is never NULL and the dedupe partitions and the index agree);(tenant_id, principal_id, folder, uid)and a partial unique index on(tenant_id, principal_id, folder, message_id, sender_address). Each step is guarded on the catalog, so replays are idempotent.created_at,id) keeps its uid; each later one moves past both the mailbox'suid_nextand its highest uid, since a racing 0.1.0 append could leaveuid_nextbehind. A mailbox with nomailbox_staterow gets one, anduid_nextends past the highest uid.rawframe stay; only the cachedmessage_idon the later copies is cleared (the header is still inraw).schema.tsdeclaresfolder,uid,sender_addressand both indexes, andschema-checkmapsbigint.MailboxStore.appendmust return its uid before reaching Postgres, so it cannot allocate atomically; the unique index refuses a colliding write instead of storing a duplicate. Its JSDoc and the README tell hosts to useappendMessage,writeMailboxMessageorcreateMailboxPersist. Nothing in this package calls it.Fromheader (the backfill copiesfrom_address), so a retry of a pre-upgrade message whoseFromhad a display name can be delivered once more. It never suppresses mail.Verification
e2e/write.test.ts: 5 concurrent writes give uids 1 to 5, and 3 concurrent writes with one Message-ID give 1 row. Both fail on fix(store): fail the caller when a native-store append fails #52.e2e/persist.test.ts: a forgedFrom:from another envelope sender lands first and the real sender's mail still delivers; a retry with a reformattedFrom:is deduped.e2e/migrations.test.ts: duplicate uids withuid_nextbehind the highest uid are renumbered past it, a repeated Message-ID from one sender is cleared while the same id from another sender is kept, and a folder with duplicate uids and nomailbox_staterow upgrades and gets a row past its uids.e2e/upgrade-from-0.1.0.test.ts: two 0.1.0 stores racing on one mailbox (uids 1, 1, 2, 3 withuid_next2) upgrade cleanly to uids 1 to 4 anduid_next5. The plain 0.1.0 upgrade adds exactlysender_address(equal tofrom_address) and the two indexes, leaves every other row and column unchanged, and a second replay changes nothing.bun run check, build,test:e2eand the pack smoke pass locally.Closes CL-9435