From 88414727a90d7d6aa92b67a23221faa50847dc9e Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Sat, 17 Oct 2026 09:09:56 +1100 Subject: [PATCH 1/2] schemachange: enable the write fence always and check it at swap A replica-role session skipped the ORIGIN fence and its write was lost at the swap. The swap now also refuses when the fence is gone or not ALWAYS. --- docs/copy-and-swap-design.md | 2 +- docs/engine-role.md | 14 +- docs/invariants.md | 16 +- docs/refusal-classes.md | 1 + docs/schemabot-integration.md | 3 +- pkg/schemachange/cutover.go | 14 +- pkg/schemachange/cutover_swap.go | 19 ++- pkg/schemachange/doc.go | 8 +- pkg/schemachange/fence.go | 70 ++++++++- pkg/schemachange/refusal.go | 13 +- pkg/schemachange/refusal_test.go | 2 + pkg/schemachange/runner.go | 18 ++- .../runner_fence_integration_test.go | 144 ++++++++++++++++++ 13 files changed, 291 insertions(+), 33 deletions(-) diff --git a/docs/copy-and-swap-design.md b/docs/copy-and-swap-design.md index 37bf2f2a..10526d5d 100644 --- a/docs/copy-and-swap-design.md +++ b/docs/copy-and-swap-design.md @@ -573,7 +573,7 @@ next pass's to catch. | `pkg/decode` | Creates, drops, and inspects the route's slot and publication under the derived name (`CreateSlot` with exported snapshot and consistent point; `DropSlot` waiting out a holder, idempotent, never crossing clusters or databases, dropping only the route's own publication; `InspectSlot` with `wal_status` and retained WAL, the latter unknown rather than zero on a server in recovery; `SlotStatus.WithinLagCeiling` judging a slot against a ceiling in bytes — a `lost` slot is `SlotLostError` before any measure, an unknown measure is `ErrSlotLagUnknown` and never within the ceiling, over it is `SlotLagExceededError` carrying both figures; `DefaultSlotLagCeiling` is one GiB; `IsEngineSlotName` tells the engine's name shape from a name merely wearing its prefix; `InspectSlot` also reads the server's account of a lost slot — `invalidation_reason`, `conflicting`, `inactive_since` — where the server version carries it, one statement across every supported major, and `SlotStatus.LostState` renders it into the `SlotLostError` so an operator reads *why* from the error). `OpenStream` decodes the slot with pgoutput from a position — the consistent point, or the applied position a resume checkpointed — on a dedicated replication connection proven to be a session of the pool's cluster on the target's database — the same table on another cluster derives the same slot name — into one `ChangeEvent` per committed row change: every column carried with its presence, so the unchanged-TOAST marker is an absent column and NULL a present one, and `OldKey` for an UPDATE that moved the primary key under either admitted replica identity. A relation other than the target, a tuple that does not line up with it, or a change outside a transaction fails closed; a table whose shape changed under the stream is `ErrSourceShapeChanged`, a TRUNCATE is `ErrUnsupportedChange`, and a warning the server sends in copy-both mode — the walsender saying it will withhold changes, as from a publication it skipped loading — stops the stream fail-closed before a keepalive can deliver a position past them; the replication connection asks for warnings itself (`SET client_min_messages = warning` on the open session, the last word on the setting), so no role, database, or server setting, login event trigger, URL parameter, or connection pooler can keep them from it. The stream keeps `Delivered` — moved by a commit or by a keepalive between transactions, never past a transaction not yet yielded in full — apart from `Confirmed`, the one position it reports to the server: `Confirm` refuses a regression or a position beyond `Delivered`, and a keepalive reply carries only what the caller confirmed, so the slot's resume point moves on the caller's word alone. A wait that runs out in `Next` sends the same report, so the stream is never silent for longer than the caller's wait (`Next`'s doc says why that matters to the walsender). Both positions order transactions by their commit, so a change's own LSN can lie below either; each `ChangeEvent` carries the `Delivered` it arrived with, which is what a caller may confirm while the change is unapplied, and the server replays by commit position, so a resume never skips a replayed change for lying below its checkpoint. `ServerWALEnd` is the walsender's send position from its last keepalive and never below `Delivered`; a change does not move it, since its message carries only the change's own position. `ServerWALEnd − Delivered` is a lower bound on lag — the walsender reports how far it has decoded, not how far the server has written — so lag against the server's WAL end is measured on the pool with `pg_current_wal_lsn()`. The server ending replication with the slot intact is `ErrStreamEnded`, the clean restart ST-4 keeps apart from slot loss. | ST-3, ST-4, CO-4, CO-8 | | `pkg/applier` | `Buffer` merges events into one `Entry` per primary key and `Drain` judges every key against the copier's `Position`, returning a `Batch` whose `CompleteFirst` names the moved, marker-bearing images the flush must complete before it writes. `Flusher` (built from a `CopySwapTarget`, a `copier.Shadow`, and the table's `TableLockSession`) applies one batch in one guarded read-write transaction — owner role, the copy statement's `search_path` (the catalog alone, or the shadow build's path when the proof carries conversions), `ACCESS SHARE` on both relations, lock confirmation, relation-OID check, the copier's guard — bound to the lock session's context so a lost lock cancels the flush in flight. A column the statement retyped with `USING` is written through that expression (D7): the flush reads the column's source type from the catalog inside its transaction and binds the decoded value as `(SELECT FROM (SELECT $n:: AS "col") AS "source")`, so the value lands as the copy would have landed it, while a completion read from the shadow is already in the shadow's form and is bound bare; the flusher refuses, as ST-6, a proof whose conversions name the primary key, a column the copy does not carry, or an empty expression, and a converted column the source no longer carries. It first reads a completion for every `CompleteFirst` image: from the old key's shadow row, read `FOR UPDATE` so the batch's own delete of that key cannot race the read; from the source row under `Key` when `OldKeyReused` or when the old key's shadow row is absent (a move captured while the old key was uncut left no shadow row). Neither row is known to be the stream's, so the image is not written: it returns in `Result.Held` as a `HeldImage` with its completion and the `pg_current_wal_insert_lsn()` the read took in the same statement, and `Buffer.Hold` takes it back with the completion pending — an event that replaces the key's entry or reuses its old key drops the completion, an UPDATE keeps it pending behind its own values, and `Buffer.Release(passed)` fills the markers still left once the stream has delivered every change committed through the read's position, so the next `Drain` flushes the whole image (D13). An image neither row can complete is written as a delete marker at `Key`, since the source holds no row there, and its key named in `Result.Skipped`. The primary path then applies the batch column-wise inside a savepoint, every delete marker before any image: a delete marker is a `DELETE`, a whole image an `INSERT … ON CONFLICT (pk) DO UPDATE`, and a marker-bearing image of a row that did not move an `UPDATE` of only its present columns — zero rows updated is a CO-8 `ErrInvariantViolation`, never an insert that invents the omitted value. On SQLSTATE `23505` or `23P01` (a unique index or an exclusion constraint) the flush rolls back to the savepoint and reapplies the batch whole-row (D13): every remaining marker is completed from the shadow row under its own key `FOR UPDATE` (absent is a CO-8 refusal), every key in the batch is deleted in one statement, and every surviving image is inserted plain; a second collision is `ErrBatchDeferred`, and `Buffer.Requeue` puts the batch back — refusing with a CO-5 `ErrInvariantViolation` if any of its keys is held, since nothing may `Add` between `Drain` and the flush's outcome — so the caller drains again once the copier has landed the chunk the rows wait on. `Result` counts the images and deletes written and the completions, carries the held images and the skipped keys, and says whether the fallback ran. The flush pins the same rendering settings as the copy and the digest (`chunksql.Rendering`: `extra_float_digits = 3`, `bytea_output = 'hex'`) so a float or a `bytea` the decoder rendered lands, through a `USING` conversion to `text`, as the text the copy wrote and the verifier will hash, and the completion reads carry the same pin so a held value lands as the shadow renders it. `Catchup` owns the loop: it gathers the stream into the buffer, `Release`s held images at the delivered position, drains against the copier's `Position` read *after* the last `Add` (so a key the copier cuts between the read and the drain is judged in flight, never discarded), flushes, and on `ErrBatchDeferred` requeues the batch for the next pass while a chunk is in flight — once the copy has landed every key, a refusal is one no chunk can resolve, and `Run` ends with `ErrBatchDeferred` rather than requeue it forever. It confirms the lesser of the delivered position and `Buffer.OldestPending` — each keyed on the `Delivered` the event arrived with, never the event's own LSN — so the slot never advances past a change that is still buffered, held, or deferred, and never falls below what the stream had already confirmed (ST-4); the confirmation follows the requeue or hold, so it never reads a buffer emptier than the one the next drain sees. The whole loop runs under the lock session's `Bind` context, so a lost lock ends a quiet wait on the stream as it ends a flush, reported as `ErrInvariantViolation` wrapping `dbconn.ErrTableLockLost`. `Run` never returns nil: it runs once and returns the context's cause when its owner stops it, carrying any cycle error that is not the stop itself alongside. It is the `progress.WorkSource` of a catch-up step: `changes_applied`, `changes_buffered`, and `lag_bytes` measured against `pg_current_wal_lsn()` read on the pool as each cycle ends, never against the walsender's send position. Each cycle then inspects the stream's slot on the pool and judges it against `CatchupOptions.SlotLagCeiling` (zero takes `decode.DefaultSlotLagCeiling`; below one byte is refused, so the ceiling is always in force): a slot that vanished or that the server reports `lost` ends `Run` with `decode.SlotLostError`, and one retaining more WAL than the ceiling ends it with `decode.SlotLagExceededError` — the table lock stays held and the slot stays in place, so the operator reads what the slot retained and the resume path decides what follows (D11, ST-3). | CO-4, CO-5, CO-6, CO-8, LK-1, LK-2, LK-3, ST-3 | | `pkg/checkpoint` | Produces `Checkpoint` and persists it: `Store` (over a `pkg/dbconn` pool) creates `pgsprite.pgsprite_checkpoint` on first use (`Ensure`, serialized under the engine's advisory key so concurrent first users never race the create; it creates only what is absent, so a pre-provisioned schema and table the engine owns need no database `CREATE`, refuses with `ErrForeignObject` a schema or table another role owns, brings a table an older engine created up to this engine's shape by adding the columns it lacks — the shape is derived from the table's columns, and every column added after the first shape is nullable so rows survive — and refuses with `*TableNewerError` — `ErrTableNewer` to `errors.Is`, naming the columns — running no DDL, a table carrying columns this engine does not know; a read or write that reaches an older table before `Ensure` has evolved it is the typed `ErrTableBehind`, the server's undefined-column error wrapped), `Save` takes the target's `TableLockSession`, confirms it from the write's own transaction, and writes the target's one row in one `INSERT … ON CONFLICT DO UPDATE` whose update is guarded on the row's format version and fingerprints — zero rows updated is an `IncompatibleError`, never a silent overwrite — and treats the shadow proof as write-once: a `Save` without a proof keeps the one the row holds, the same proof is accepted, and a different proof is `ErrInvariantViolation` that changes nothing, since a different shadow is a fresh start and goes through `Delete`; `Load` returns the row for a run with the same `Fingerprints`, `ErrNotFound` for no row, `ErrTableMissing` when `Ensure` never ran, an `IncompatibleError` naming the first disagreeing field (format, then source, then target fingerprint) and carrying the row's whole `Identity`, or, after bounded retries through transient errors (`dbconn.Retryable` plus a session the server ended from outside it, `57P01`/`57P02`/`57P03`) with an injected sleep, an error that is none of those; `Delete`, under the same lock, is the explicit fresh start and removes only the row whose `Identity` the caller was shown; `Inspect` is the resume's first read, before the run knows which statement it is resuming — the same outcomes as `Load`, guarded on the row format alone, so the row's fingerprints, phase, and shadow proof come back for the caller to judge, while every write stays guarded on the whole identity. The watermark column is `NULL` while nothing has landed, the LSN is a `pg_lsn`, the phase is stored by its stable name, and `shadow_proof` is the JSON proof of the shadow the run built (`Checkpoint.ShadowProof`, `NULL` until there is one), which a resume after the swap needs because `InspectShadow` cannot re-mint it once the shadow no longer bears its name; `PhaseReconciling`, between `catching_up` and `verifying`, is the ST-4 marker a reconcile writes before it touches the slot, so a reconcile that dies restarts as one. `ReapOrphanSlots` is the ST-3 reaper, placed here because the checkpoint row is what makes a slot owned: it lists the logical, engine-prefixed slots of `current_database()` and then reads the non-terminal rows that name them — a run makes its slot and commits the row that names it under `WithSlotLock`, the engine's advisory lock on the slot's name, transaction-scoped on a connection dialled outside the store's pool so the holder's own work on the pool never waits on the seat the lock sits in, so a slot with no row was left by a run that ended without dropping it, and a slot a done or failed row names by one whose cleanup did not finish — and re-reads each orphan's row right before dropping it, both under the same lock, so a run that claimed the slot since the pass began keeps it and one claiming it now waits for the drop and creates afresh; the pass drops each orphan through `decode.DropIdleSlot`, publication and all, reporting it in `Reaped.Dropped`; a slot a live row names (`Live`), a slot a walsender or backend holds when the drop comes round (`Active`), and a name wearing the prefix without the engine's shape (`Foreign`) are reported and left as found; slots of other databases are never read; the drops stop at the first failure, reporting what was dropped before it; a database where `Ensure` never ran is refused with `ErrTableMissing` rather than read as all orphans; and two reapers over one database take turns at each slot, the one that waited finding the slot gone. | ST-1, ST-2, ST-3, ST-4, LK-1 | -| `pkg/schemachange` | Shadow builder (`BuildShadow` produces `BuiltShadow`: source and shadow OIDs, fingerprints, identity handoff, copy columns, the statement's `USING` conversions, fidelity snapshot; before the retargeted statement runs it tries each conversion as a stored generated column's expression on the empty shadow — the one server-side check that admits only an immutable expression — in a save point it always rolls back, and refuses a non-immutable one as `shadow-statement-conversion` wrapping `ErrConversionNotImmutable`, since the copy, the recopy, and the catch-up evaluate the expression again at other times and must get the same answer; `SourceOfDerivedName` maps a derived name back to its table), orchestrator, and cutover. `BuildShadow`, `DropShadow`, and `InspectShadow` each take the `*dbconn.TableLockSession` for the table — a dedicated direct server session, distinct from the working pool, that `dbconn.AcquireTableLock` refuses to open through a transaction-pooling proxy: the operation runs under the session's `Bind` context so a lost lock cancels the statement in flight, and its transaction re-asserts from its own connection that the session's backend holds the lock before the first write, since the lock and the work are deliberately on different sessions. `InspectShadow` is the resume path `ErrShadowExists` points at: it re-derives the `BuiltShadow` proof from the catalog for the caller to compare with its checkpoint — `BuiltShadow.Proof()` is the plain, JSON-encodable view of that proof (`BuiltShadow` marshals as it), and nothing decodes back into a `BuiltShadow`, so only the builder and the inspection mint one; `DropShadow` is the D5 cleanup, dropping only a plain table the source's owner owns, without `CASCADE`. `GateCutover` is the ST-5 fidelity gate: handed a `BuiltShadow` and a `checksum.VerifiedShadow` for the same table, it re-reads both relations under the lock, refuses on a moved OID, a fingerprint or fidelity snapshot that drifted from the build's record, an invalid shadow index, or a swap name already taken, and otherwise mints `CutoverReady` — the pairing of every source index and extended statistics object with its shadow counterpart by catalog definition (not by name, since `LIKE` renames them; the default operator class and the column's own collation of a key column the statement retyped are set aside, since the server re-derives them for the new type; a written operator class or collation still has to agree, and a relaxed definition two indexes share pairs neither), and the sequences the swap must re-own; the swap itself consumes that proof. `Cutover` is the swap: one transaction under the lock session that takes `ACCESS EXCLUSIVE` on the source and the shadow under `lock_timeout` with bounded retry and backoff, runs the caller's `DrainFunc` (nil for a quiesced table) once per attempt once writers are excluded — a rolled-back attempt takes the drain's work with it, so the next attempt drains again from the same state — re-runs the copy-and-swap shape check (RF-2) and the gate's checklist so the pairing it renames by is as fresh as the lock, performs the D8 renames (every source dependent to its derived `_old` name, the source to `_old`, the shadow to the source's name, each paired shadow dependent to its partner's name), completes the D5 handoff (re-owning shared sequences; recreating each identity column with the source sequence's declared options — bounds following a widened column's type — under the source sequence's name, carrying the source sequence's grants, and `setval` to its `(last_value, is_called)`), re-reads the catalog to confirm every rename and handoff before committing, and mints `SwappedTable`; an attempt whose connection broke or whose context ended is resolved by following its backend to its exit and then reading from a fresh connection which OID bears the source name, never assumed (LK-4), and every attempt the catalog shows rolled back is reported wrapping `ErrCutoverRolledBack`. `DropOldTable` is the D9 drop: a separate bounded transaction that drops only the relation whose OID the `SwappedTable` recorded as the source, without `CASCADE`. `InspectSwapped` is the resume path between `Cutover` and `DropOldTable`, and the one home of LK-4's committed branch: handed the `Proof` the caller checkpointed before the cutover, it re-derives the `SwappedTable` from the catalog in one bounded read transaction under the lock session — the source's name must be borne by the proof's shadow OID and the derived `_old` name by the proof's source OID (ST-6), the dependent pairing is re-derived from the names the swap gave (a live dependent pairs with the old-table dependent under its derived name, so the re-derived pairs carry no shadow names), and every identity column the proof handed off must be an identity on the live table drawing from the sequence recreated under the source sequence's name; `SwappedTable.Proof()` is the plain, JSON-encodable `SwapProof` view of it (`SwappedTable` marshals as it), and nothing decodes back into a `SwappedTable`, so only the cutover and the inspection mint one. Both relations must still be owned by the role the build recorded, since the drop runs as it. The catalog states the proof does not describe each get their own answer: the source still live is `ErrNotSwapped` (run `Cutover`), the `_old` name borne by nothing while the source's OID is gone from the catalog is `ErrOldTableNotFound` (the drop already committed; the change is complete), the `_old` name borne by another relation — or by nothing while the retained source still exists under another name — is refused as `cutover-relation-replaced`, a live or old table owned by another role is refused as `cutover-swap-mismatch`, the source's name borne by neither relation is refused as `cutover-outcome-ambiguous` (LK-4), and so is a read while another backend holds either relation exclusively, since a swap whose `COMMIT` is still deciding holds both until its outcome is visible and the same call answers once it has let go. `SwapProof.SameSwap` is the resume's comparison: the two relations by OID and name, the owner, the re-owned sequences, and the identity columns, leaving out the dependent pairing and the attempt count a re-derived proof cannot carry. `DropOldTable` locks whatever bears the `_old` name before it reads the OID it compares, so the relation it proves is the relation it drops. `InspectResume` is the resume path's first read: handed a non-terminal checkpoint row for the target, it refuses one it cannot act on before any slot is touched — a terminal phase (`resume-row-terminal`), a row in `cutover`, which only `InspectSwapped` can answer (`resume-row-cutover`), another table or another slot name (`resume-row-mismatch`) — and otherwise answers `ResumeClean` when the slot is present, the server still serves it, and nothing holds it, or `ResumeReconcile` when the row names no slot, the slot is gone, the server reports it lost (with the server's cause on the `decode.SlotLostError` it carries), or the row is already in `reconciling`; a slot of the derived name that is physical or another database's is refused as `resume-slot-foreign` in every phase (it also answers `errors.Is` for `decode.ErrForeignDecodingState`), and the engine's own slot while another backend holds it as `resume-slot-held`, to be inspected again once the holder lets go; a target proven without logical decoding is always clean, having no slot to lose. `Reconciler.Reconcile` is the ST-4 reconcile mode (D11, D14), run whole under the lock session's `Bind` context so a lost lock cancels the slot command in flight — a drop still waiting out a holder included — and a session that has lost the table never drops or makes a slot a run holding the lock since may own: it applies `InspectResume`'s row checks, refuses a row whose shadow name or fingerprints are not the `BuiltShadow`'s, confirms from a transaction of its own that the source and the shadow are still the relations the build recorded (`cutover-relation-replaced` otherwise, before anything is written), writes the row to `reconciling` naming the derived slot and publication so the slot it is about to make is the row's from the moment it exists, drops whatever bears the derived slot name, creates a fresh slot, runs one `checksum` pass at or below the watermark under `recopy` when anything landed — recopying only the chunks the gap touched, with no reread, since the new slot carries every write that lands after the recopy — and writes the row back to `copying` or `catching_up` by the watermark, naming the new slot and its consistent point as the applied position the stream reopens from; the shadow is kept and the watermark never moves, and the open slot is returned for the caller to decode from or close. `Runner.Run` is the orchestrator, routed by the target's proof — a quiesced run for a target proven without logical decoding, a live run for one proven for it: it acquires the table lock session itself, makes sure the checkpoint table exists, reads the target's row through `Store.Inspect`, and dispatches on it — no row builds the shadow, fences the source, and writes `copying` with the statement's text (a relation already under the shadow's name with no row is refused as `shadow-unrecorded`; a first save whose row does not land drops the shadow and the fence again, so nothing unrecorded is left); a `copying` or `verifying` row is held against the statement it records — compared as parsed, the same operations on the same table, so spelling and schema qualification do not matter (`resume-row-mismatch` otherwise, and a row without a statement, or one whose statement names another table, likewise) — and against the shadow `InspectShadow` re-derives, names, fingerprints, and JSON-encoded proof alike (`resume-row-mismatch` otherwise), and installs over the fence the stopped run left; a row whose swap has not committed — a `cutover` row whose swap rolled back included — and whose shadow no relation bears is refused as `resume-shadow-gone`, the counterpart of `shadow-unrecorded`, writing nothing, so the row and the slot a live row names are left for the operator to drop and delete; a `cutover` row is answered by `InspectSwapped`, a committed swap continuing at the old table drop and `ErrNotSwapped` back through verify and gate; a terminal row is refused as `resume-row-terminal` and left for the caller to delete; each kind of run refuses the other kind's row as `resume-row-decoding` (the quiesced run a row that names a slot or stands in `catching_up` or `reconciling`, since it has no slot to continue it with; the live run a row that names no slot, since the source behind it was fenced, not captured) — then runs the phases in order. The quiesced run: the copier from the row's watermark, the landed watermark saved every `CheckpointInterval` and once more when the copy ends, `verifying` with the complete watermark, one `checksum` pass under `DivergenceAbort`, `GateCutover`, the row to `cutover` before `Cutover` is called with no drain, `DropOldTable` unless `KeepOld` — under which the fence, which rode the rename to `_old`, is taken off the retained table — and the row to `done`. The live run: no row creates the slot and writes `copying` naming it, with the slot's consistent point as the applied position, under `Store.WithSlotLock` so the reaper never meets the slot before its row (a first save that does not land drops the slot and the shadow again; a slot of the derived name the run did not make is reported as the `*decode.SlotExistsError` it is and left as found); a resumed row is read through `InspectResume` right before the stream opens, and a slot that is gone or lost is replaced by `Reconciler.Reconcile`, whose outcome `Result.Reconciled` and `Result.Reconciliation` report (a `cutover` row whose swap `InspectSwapped` shows rolled back is written back to `catching_up` first, since the inspection reads a `cutover` row as a swap still deciding); the stream is then opened at the row's position and applied to the shadow by a capture (`applier.Catchup` over a `Flusher`) that runs beside the fill — the copier while the row is in `copying`, its chunk positions handed to the capture so the stream's changes to a chunk land only once the chunk has; `catching_up` until the stream has confirmed past a marker written after the copy (a resumed `verifying` row catches up again before its pass); then `verifying`, one `checksum.CheckLive` pass under `DivergenceAbort` with the capture's `Catchup` as its confirmed source — then the capture is ended, the row records the position it confirmed through, `GateCutover`, the row to `cutover`, and `Cutover` with a drain that under the swap's lock writes a marker, reopens the stream from the row's position, gathers until the marker is delivered, closes the stream, waits for the slot to go idle — all on one connection dialled outside the pool for the attempt, since the swap holds a pool connection and a drain waiting on the pool for a second would wait on every other swap doing the same — and flushes everything gathered through the same `Flusher` (a batch the flusher defers is refused as `cutover-drain-deferred`; images the flusher held back are released and flushed once more, and a hold that persists, or anything left unapplied, is `ErrInvariantViolation`, since no writer can change the shadow under the swap's lock); the slot is dropped once the swap has committed, before `DropOldTable` or the `KeepOld` settle, and the row goes to `done`. The fence is the quiescence: a statement-level `BEFORE INSERT / UPDATE / DELETE / TRUNCATE` trigger on the source, created as the engine role and calling a function in the engine's schema, that refuses every write with SQLSTATE `55006` while the run is in progress or stopped between two saves; abandoning a stopped run is `DropShadow`, `DropWriteFence` (quiesced) or `decode.DropSlot` (live), and `Store.Delete` under the table lock. `PhaseFailed` is never written: a run that stops leaves the row at its last save and the next `Run` resumes from it. Every refusal the ten operations return is a `*RefusalError` carrying a `RefusalCause` from the closed set in [refusal-classes.md](refusal-classes.md#shadow-operation-refusals-keyed-on-refusalcause), so an importer routes on the cause rather than on message text; `Runner.Run` re-mints every `copier.RefusalCause` as a shadow cause too, wrapping the copier's refusal, so a caller reads one vocabulary whether the refusal came from a shadow operation or from the copy. | LK-1, LK-2, LK-4, CO-1, ST-4, ST-5, ST-6, ST-7 | +| `pkg/schemachange` | Shadow builder (`BuildShadow` produces `BuiltShadow`: source and shadow OIDs, fingerprints, identity handoff, copy columns, the statement's `USING` conversions, fidelity snapshot; before the retargeted statement runs it tries each conversion as a stored generated column's expression on the empty shadow — the one server-side check that admits only an immutable expression — in a save point it always rolls back, and refuses a non-immutable one as `shadow-statement-conversion` wrapping `ErrConversionNotImmutable`, since the copy, the recopy, and the catch-up evaluate the expression again at other times and must get the same answer; `SourceOfDerivedName` maps a derived name back to its table), orchestrator, and cutover. `BuildShadow`, `DropShadow`, and `InspectShadow` each take the `*dbconn.TableLockSession` for the table — a dedicated direct server session, distinct from the working pool, that `dbconn.AcquireTableLock` refuses to open through a transaction-pooling proxy: the operation runs under the session's `Bind` context so a lost lock cancels the statement in flight, and its transaction re-asserts from its own connection that the session's backend holds the lock before the first write, since the lock and the work are deliberately on different sessions. `InspectShadow` is the resume path `ErrShadowExists` points at: it re-derives the `BuiltShadow` proof from the catalog for the caller to compare with its checkpoint — `BuiltShadow.Proof()` is the plain, JSON-encodable view of that proof (`BuiltShadow` marshals as it), and nothing decodes back into a `BuiltShadow`, so only the builder and the inspection mint one; `DropShadow` is the D5 cleanup, dropping only a plain table the source's owner owns, without `CASCADE`. `GateCutover` is the ST-5 fidelity gate: handed a `BuiltShadow` and a `checksum.VerifiedShadow` for the same table, it re-reads both relations under the lock, refuses on a moved OID, a fingerprint or fidelity snapshot that drifted from the build's record, an invalid shadow index, or a swap name already taken, and otherwise mints `CutoverReady` — the pairing of every source index and extended statistics object with its shadow counterpart by catalog definition (not by name, since `LIKE` renames them; the default operator class and the column's own collation of a key column the statement retyped are set aside, since the server re-derives them for the new type; a written operator class or collation still has to agree, and a relaxed definition two indexes share pairs neither), and the sequences the swap must re-own; the swap itself consumes that proof. `Cutover` is the swap: one transaction under the lock session that takes `ACCESS EXCLUSIVE` on the source and the shadow under `lock_timeout` with bounded retry and backoff, runs the caller's `DrainFunc` (nil for a quiesced table) once per attempt once writers are excluded — a rolled-back attempt takes the drain's work with it, so the next attempt drains again from the same state — re-runs the copy-and-swap shape check (RF-2) and the gate's checklist so the pairing it renames by is as fresh as the lock, performs the D8 renames (every source dependent to its derived `_old` name, the source to `_old`, the shadow to the source's name, each paired shadow dependent to its partner's name), completes the D5 handoff (re-owning shared sequences; recreating each identity column with the source sequence's declared options — bounds following a widened column's type — under the source sequence's name, carrying the source sequence's grants, and `setval` to its `(last_value, is_called)`), re-reads the catalog to confirm every rename and handoff before committing, and mints `SwappedTable`; an attempt whose connection broke or whose context ended is resolved by following its backend to its exit and then reading from a fresh connection which OID bears the source name, never assumed (LK-4), and every attempt the catalog shows rolled back is reported wrapping `ErrCutoverRolledBack`. `DropOldTable` is the D9 drop: a separate bounded transaction that drops only the relation whose OID the `SwappedTable` recorded as the source, without `CASCADE`. `InspectSwapped` is the resume path between `Cutover` and `DropOldTable`, and the one home of LK-4's committed branch: handed the `Proof` the caller checkpointed before the cutover, it re-derives the `SwappedTable` from the catalog in one bounded read transaction under the lock session — the source's name must be borne by the proof's shadow OID and the derived `_old` name by the proof's source OID (ST-6), the dependent pairing is re-derived from the names the swap gave (a live dependent pairs with the old-table dependent under its derived name, so the re-derived pairs carry no shadow names), and every identity column the proof handed off must be an identity on the live table drawing from the sequence recreated under the source sequence's name; `SwappedTable.Proof()` is the plain, JSON-encodable `SwapProof` view of it (`SwappedTable` marshals as it), and nothing decodes back into a `SwappedTable`, so only the cutover and the inspection mint one. Both relations must still be owned by the role the build recorded, since the drop runs as it. The catalog states the proof does not describe each get their own answer: the source still live is `ErrNotSwapped` (run `Cutover`), the `_old` name borne by nothing while the source's OID is gone from the catalog is `ErrOldTableNotFound` (the drop already committed; the change is complete), the `_old` name borne by another relation — or by nothing while the retained source still exists under another name — is refused as `cutover-relation-replaced`, a live or old table owned by another role is refused as `cutover-swap-mismatch`, the source's name borne by neither relation is refused as `cutover-outcome-ambiguous` (LK-4), and so is a read while another backend holds either relation exclusively, since a swap whose `COMMIT` is still deciding holds both until its outcome is visible and the same call answers once it has let go. `SwapProof.SameSwap` is the resume's comparison: the two relations by OID and name, the owner, the re-owned sequences, and the identity columns, leaving out the dependent pairing and the attempt count a re-derived proof cannot carry. `DropOldTable` locks whatever bears the `_old` name before it reads the OID it compares, so the relation it proves is the relation it drops. `InspectResume` is the resume path's first read: handed a non-terminal checkpoint row for the target, it refuses one it cannot act on before any slot is touched — a terminal phase (`resume-row-terminal`), a row in `cutover`, which only `InspectSwapped` can answer (`resume-row-cutover`), another table or another slot name (`resume-row-mismatch`) — and otherwise answers `ResumeClean` when the slot is present, the server still serves it, and nothing holds it, or `ResumeReconcile` when the row names no slot, the slot is gone, the server reports it lost (with the server's cause on the `decode.SlotLostError` it carries), or the row is already in `reconciling`; a slot of the derived name that is physical or another database's is refused as `resume-slot-foreign` in every phase (it also answers `errors.Is` for `decode.ErrForeignDecodingState`), and the engine's own slot while another backend holds it as `resume-slot-held`, to be inspected again once the holder lets go; a target proven without logical decoding is always clean, having no slot to lose. `Reconciler.Reconcile` is the ST-4 reconcile mode (D11, D14), run whole under the lock session's `Bind` context so a lost lock cancels the slot command in flight — a drop still waiting out a holder included — and a session that has lost the table never drops or makes a slot a run holding the lock since may own: it applies `InspectResume`'s row checks, refuses a row whose shadow name or fingerprints are not the `BuiltShadow`'s, confirms from a transaction of its own that the source and the shadow are still the relations the build recorded (`cutover-relation-replaced` otherwise, before anything is written), writes the row to `reconciling` naming the derived slot and publication so the slot it is about to make is the row's from the moment it exists, drops whatever bears the derived slot name, creates a fresh slot, runs one `checksum` pass at or below the watermark under `recopy` when anything landed — recopying only the chunks the gap touched, with no reread, since the new slot carries every write that lands after the recopy — and writes the row back to `copying` or `catching_up` by the watermark, naming the new slot and its consistent point as the applied position the stream reopens from; the shadow is kept and the watermark never moves, and the open slot is returned for the caller to decode from or close. `Runner.Run` is the orchestrator, routed by the target's proof — a quiesced run for a target proven without logical decoding, a live run for one proven for it: it acquires the table lock session itself, makes sure the checkpoint table exists, reads the target's row through `Store.Inspect`, and dispatches on it — no row builds the shadow, fences the source, and writes `copying` with the statement's text (a relation already under the shadow's name with no row is refused as `shadow-unrecorded`; a first save whose row does not land drops the shadow and the fence again, so nothing unrecorded is left); a `copying` or `verifying` row is held against the statement it records — compared as parsed, the same operations on the same table, so spelling and schema qualification do not matter (`resume-row-mismatch` otherwise, and a row without a statement, or one whose statement names another table, likewise) — and against the shadow `InspectShadow` re-derives, names, fingerprints, and JSON-encoded proof alike (`resume-row-mismatch` otherwise), and installs over the fence the stopped run left; a row whose swap has not committed — a `cutover` row whose swap rolled back included — and whose shadow no relation bears is refused as `resume-shadow-gone`, the counterpart of `shadow-unrecorded`, writing nothing, so the row and the slot a live row names are left for the operator to drop and delete; a `cutover` row is answered by `InspectSwapped`, a committed swap continuing at the old table drop and `ErrNotSwapped` back through verify and gate; a terminal row is refused as `resume-row-terminal` and left for the caller to delete; each kind of run refuses the other kind's row as `resume-row-decoding` (the quiesced run a row that names a slot or stands in `catching_up` or `reconciling`, since it has no slot to continue it with; the live run a row that names no slot, since the source behind it was fenced, not captured) — then runs the phases in order. The quiesced run: the copier from the row's watermark, the landed watermark saved every `CheckpointInterval` and once more when the copy ends, `verifying` with the complete watermark, one `checksum` pass under `DivergenceAbort`, `GateCutover`, the row to `cutover` before `Cutover` is called with no drain, `DropOldTable` unless `KeepOld` — under which the fence, which rode the rename to `_old`, is taken off the retained table — and the row to `done`. The live run: no row creates the slot and writes `copying` naming it, with the slot's consistent point as the applied position, under `Store.WithSlotLock` so the reaper never meets the slot before its row (a first save that does not land drops the slot and the shadow again; a slot of the derived name the run did not make is reported as the `*decode.SlotExistsError` it is and left as found); a resumed row is read through `InspectResume` right before the stream opens, and a slot that is gone or lost is replaced by `Reconciler.Reconcile`, whose outcome `Result.Reconciled` and `Result.Reconciliation` report (a `cutover` row whose swap `InspectSwapped` shows rolled back is written back to `catching_up` first, since the inspection reads a `cutover` row as a swap still deciding); the stream is then opened at the row's position and applied to the shadow by a capture (`applier.Catchup` over a `Flusher`) that runs beside the fill — the copier while the row is in `copying`, its chunk positions handed to the capture so the stream's changes to a chunk land only once the chunk has; `catching_up` until the stream has confirmed past a marker written after the copy (a resumed `verifying` row catches up again before its pass); then `verifying`, one `checksum.CheckLive` pass under `DivergenceAbort` with the capture's `Catchup` as its confirmed source — then the capture is ended, the row records the position it confirmed through, `GateCutover`, the row to `cutover`, and `Cutover` with a drain that under the swap's lock writes a marker, reopens the stream from the row's position, gathers until the marker is delivered, closes the stream, waits for the slot to go idle — all on one connection dialled outside the pool for the attempt, since the swap holds a pool connection and a drain waiting on the pool for a second would wait on every other swap doing the same — and flushes everything gathered through the same `Flusher` (a batch the flusher defers is refused as `cutover-drain-deferred`; images the flusher held back are released and flushed once more, and a hold that persists, or anything left unapplied, is `ErrInvariantViolation`, since no writer can change the shadow under the swap's lock); the slot is dropped once the swap has committed, before `DropOldTable` or the `KeepOld` settle, and the row goes to `done`. The fence is the quiescence: a statement-level `BEFORE INSERT / UPDATE / DELETE / TRUNCATE` trigger on the source, created as the engine role, enabled `ALWAYS` so a session in `session_replication_role = replica` is refused too, and calling a function in the engine's schema, that refuses every write with SQLSTATE `55006` while the run is in progress or stopped between two saves (a resumed run installs over it, which enables it `ALWAYS` again); the quiesced `Runner`'s swap re-reads the fence under its `ACCESS EXCLUSIVE` lock and refuses as `cutover-unfenced` (CO-1) when the engine's trigger is not on the source, enabled `ALWAYS`, leaving the row in `cutover` for the next `Run` to install again and verify again (a live run installs no fence, so its swap does not check one); abandoning a stopped run is `DropShadow`, `DropWriteFence` (quiesced) or `decode.DropSlot` (live), and `Store.Delete` under the table lock. `PhaseFailed` is never written: a run that stops leaves the row at its last save and the next `Run` resumes from it. Every refusal the ten operations return is a `*RefusalError` carrying a `RefusalCause` from the closed set in [refusal-classes.md](refusal-classes.md#shadow-operation-refusals-keyed-on-refusalcause), so an importer routes on the cause rather than on message text; `Runner.Run` re-mints every `copier.RefusalCause` as a shadow cause too, wrapping the copier's refusal, so a caller reads one vocabulary whether the refusal came from a shadow operation or from the copy. | LK-1, LK-2, LK-4, CO-1, ST-4, ST-5, ST-6, ST-7 | | `pkg/refusal` | `Class`, the routing class a typed refusal cause names — `capability-boundary`, `by-design`, `environmental`, or `invariant-violation`, the classes the cause tables of [refusal-classes.md](refusal-classes.md) use — answered by `copier.RefusalCause.Class()`, `schemachange.RefusalCause.Class()`, `preflight.PartitionRefusalCause.Class()`, `preflight.CopySwapRefusalCause.Class()`, and `executor.CreateShapeCause.Class()` from each cause's `class` cell, so an importer triages on the class without reading the cause's text; the per-cause row still decides the action, and a class never licenses a retry on its own. `verdict.Class` declares its tokens from this package, so the periphery converts with a plain conversion and the core never imports the verdict renderer. | — | Each producing package owns its types. `pkg/schemachange` imports every producer; no producer diff --git a/docs/engine-role.md b/docs/engine-role.md index a2650bd3..576c23e5 100644 --- a/docs/engine-role.md +++ b/docs/engine-role.md @@ -99,21 +99,23 @@ A quiesced copy-and-swap run (`pkg/schemachange.Runner`) does not take the calle nothing writes the table: it installs a statement-level `BEFORE INSERT OR UPDATE OR DELETE OR TRUNCATE` trigger, `pgsprite_write_fence`, on the source before its first checkpoint save, and every write the trigger sees is refused with SQLSTATE `55006` (`object_in_use`) and a message -naming the table. The trigger calls a function of the same name in the engine's own `pgsprite` +naming the table. The trigger is enabled `ALWAYS`, so it fires for a session running under +`session_replication_role = replica` as well. The trigger calls a function of the same name in the engine's own `pgsprite` schema, created once per database as the engine role and owned by it, so the table's owner needs no access to that schema for the fence to fire. The fence stays on a stopped run's source until the run is resumed or abandoned (`DropWriteFence`); a completed run carries it to the retained `_old` table with the swap, where it is dropped with the table or, under `KeepOld`, taken off so the retained table can be renamed back and take writes at once. -What this needs: `CREATE TRIGGER` on a table requires the `TRIGGER` privilege on it or +What this needs: `CREATE TRIGGER` and `ALTER TABLE … ENABLE ALWAYS TRIGGER` on a table require ownership, which the Tier 1 membership in the owning role already gives; the function needs `CREATE` on the `pgsprite` schema, which the engine owns. A pre-provisioned deployment that created the schema for the engine needs nothing further once the schema is owned by the engine -role. The fence does not stop a superuser, nor a session running under -`session_replication_role = replica`, which fires no ordinary triggers; a deployment whose -writers can hold either has to quiesce the table by its own means, and a difference the -verification pass then finds is reported, never repaired. Before `Run`, quiesce the table as far +role. The fence does not stop a role that owns the table, which can disable or drop it; the +swap refuses (`cutover-unfenced`) a fence found disabled or gone under its lock, but a role that +disables the fence, writes, and restores it between the verification pass and the swap is not +seen. A deployment whose writers can hold table ownership has to quiesce the table by its own +means, and a difference the verification pass finds is reported, never repaired. Before `Run`, quiesce the table as far as the application allows — stop or pause its writers, or route them around it — so the fence refuses nothing in practice; the fence is the proof, not the plan. diff --git a/docs/invariants.md b/docs/invariants.md index 21643dd1..f27e8a63 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -57,10 +57,18 @@ since under the swap lock (`cutover-undrained`); the mechanism is in [copy-and-swap-design.md](copy-and-swap-design.md#live-verification). The `Runner` on a quiesced target makes the quiescence its pass relies on true rather than assumed: it fences the source with a statement-level `BEFORE` trigger (`pgsprite_write_fence`, SQLSTATE `55006`) -before its first save and keeps it through every stop until the swap carries it to the -retained table, so the copy and one quiesced pass are the whole proof (fences-the-source, -leaves-the-fence-on-a-stopped-run, unfences-the-old-table-it-keeps, -stops-on-a-shadow-that-differs tests). On a decoding target it runs `CheckLive` under +enabled `ALWAYS`, so a session in `session_replication_role = replica` is refused too, before +its first save, and keeps it through every stop until the swap carries it to the retained +table, so the copy and one quiesced pass are the whole proof (fences-the-source, +refuses-a-replica-role-session, leaves-the-fence-on-a-stopped-run, +unfences-the-old-table-it-keeps, stops-on-a-shadow-that-differs tests). A resumed run installs +over the fence, which enables it `ALWAYS` again (resume-upgrades-an-origin-fence test). The +quiesced swap re-reads the fence under its `ACCESS EXCLUSIVE` lock and refuses the proof as +`cutover-unfenced` unless the engine's trigger is still on the source, enabled `ALWAYS` +(refuses-when-the-fence-was-disabled and refuses-when-the-fence-was-dropped tests); a role +that owns the table can still disable the fence, write, and restore it between the pass and +the swap, and that write is not seen. A live run installs no fence, so its swap checks none. +On a decoding target it runs `CheckLive` under `DivergenceAbort` once the catch-up has passed a marker written after the copy, with the capture's own `Catchup` as the pass's confirmed source, and hands the swap a drain that reopens the stream from the position the capture confirmed and applies it under the swap's diff --git a/docs/refusal-classes.md b/docs/refusal-classes.md index 8c0eb11c..bea31172 100644 --- a/docs/refusal-classes.md +++ b/docs/refusal-classes.md @@ -196,6 +196,7 @@ causes above: the verdict's `cause` is the shadow cause unchanged and its `class | `cutover-unverified` | The cutover gate was handed an empty built-shadow or verified-shadow proof, two proofs naming different relations or different relation OIDs (a shadow rebuilt under the same name after the checksum ran), or a verified watermark that does not cover the whole key space ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Only the copier and verifier mint those proofs for the same table; a caller that reaches the gate without them skipped the checksum, which no retry corrects. | | `cutover-undrained` | The cutover was handed a verified-shadow proof a live pass minted — the shadow equal to the source as of a position the stream had reached, with every change since still in the stream — and no drain to apply those changes under the swap lock ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Pass a `DrainFunc` that applies the stream through the final position under the swap lock, or stop the stream and re-verify with a quiesced `Check`; the proof is sound, the hand-off to the swap is not. | | `cutover-drain-deferred` | The shadow refused a batch the swap's drain had to apply under the swap lock: a unique-key move among the changes committed since the live verification pass that cannot be ordered into the shadow, with no copy left to change the outcome ([CO-6](invariants.md#co-6--unique-secondary-key-moves-must-converge-postgresql-specific-gap)) | `environmental` | The swap transaction is rolled back and the source stays live; the row is left in `cutover` with the position the drain reopened from, so a resume drains the same changes again once the rows the move touched have settled. A move the shadow cannot ever order is re-planned. | +| `cutover-unfenced` | The quiesced `Runner`'s swap, under its `ACCESS EXCLUSIVE` lock, did not find the engine's write fence on the source enabled `ALWAYS`: the trigger was disabled, dropped, or replaced since the run installed it ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `environmental` | Something other than the engine changed the source's triggers under the run, so a write may have reached it after the verification pass. The swap rolled back and nothing is swapped; the next `Run` installs the fence again, verifies again, and swaps, and a write that did land makes that pass stop on a divergence. | | `cutover-relation-replaced` | The source or the shadow under its name is no longer the relation the build proved: its OID moved; or, after the swap, the derived `_old` name is borne by a relation other than the source the swap retained, or by nothing while the retained source still exists under another name ([ST-6](invariants.md#st-6--preflight-before-the-first-write)) | `environmental` | The proof describes a table that no longer exists; before the swap, drop the shadow and start the change over from preflight; after it, an operator reads the catalog to learn what took the `_old` name, or where the retained source went, before anything is dropped. | | `cutover-schema-drift` | The source's or the shadow's introspected model no longer matches the fingerprint the build recorded: a column, constraint, index, storage parameter, or column-owned sequence declaration changed since ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | Someone changed a table under the run; the shadow no longer embodies the gated statement against this source, so it is dropped and the change re-planned. | | `cutover-fidelity-drift` | The source's or the shadow's owner, grants, policies, comment, replica identity, per-column or extended statistics targets, unvalidated checks, or identity columns and their sequences no longer match the snapshot the build recorded, or the source's grants no longer match the shadow's ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | The metadata the swap would carry forward moved after it was copied; re-synchronising it under the lock is a planned capability, and until then the shadow is rebuilt. | diff --git a/docs/schemabot-integration.md b/docs/schemabot-integration.md index f7a9bc00..fd941edf 100644 --- a/docs/schemabot-integration.md +++ b/docs/schemabot-integration.md @@ -327,7 +327,8 @@ the slot under the row (decoding target): | `Run` outcome | What it means | Orchestrator action | | --- | --- | --- | | `*decode.SlotExistsError` (decoding target, first start) | A logical slot of the engine's derived name already exists and no row names it: a run that ended before its row landed, or a slot made by hand. The run made nothing and dropped nothing — the slot, its publication, and the absence of a row are as the run found them | Run `Store.ReapOrphanSlots`, which drops the slot once nothing holds it, and then `Run` again; a slot that stays held is an operator's to inspect | -| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a superuser, or `session_replication_role = replica`) to find | +| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a role that owns the table and disables it) to find | +| `cutover-unfenced` (quiesced target, from `Cutover`, through `Run`) | The swap, under its lock, found the engine's write fence on the source disabled, dropped, or not enabled `ALWAYS`; nothing is swapped and the row is in `cutover` | Retry `Run`: it installs the fence again, verifies again, and swaps; a write that landed while the fence was down makes the verification pass stop on a `*checksum.DivergenceError`. Find what changed the source's triggers | | `*checksum.UnresolvedError` (decoding target) | The live pass could not settle a chunk the source kept writing within its reread bound; nothing was found wrong and nothing was minted, and the row stays in `verifying`. The bound is `LiveOptions.Attempts` fresh rereads of one chunk (default `checksum.DefaultSettleAttempts`), each waited on for the catch-up up to `LiveOptions.SettleWait`: a chunk written again between every two of its rereads never settles, so a table whose writes land on one chunk faster than the pass can read it twice is the case, whatever the table's total write rate | Retry `Run`: the resumed row catches up again and runs a fresh pass; a chunk that never settles is a hot spot to wait out, write less often, or give more `Attempts` | | `cutover-drain-deferred` (decoding target) | The swap's drain met a unique-key move it could not order into the shadow; the swap rolled back and the row is in `cutover` with the position the drain reopened from | Retry `Run`: `InspectSwapped` answers `ErrNotSwapped`, the row goes back to `catching_up`, and the same changes are drained again once the rows the move touched have settled | | `*decode.SlotLagExceededError` (decoding target) | The slot retained more WAL than `CatchupOptions.SlotLagCeiling` while the capture ran; the run stopped with the slot in place under the row | Retry `Run` once the source's write rate allows: the stream reopens at the row's position and drains the backlog. Abandoning the run (`DropShadow`, `decode.DropSlot`, `Store.Delete`) releases the WAL | diff --git a/pkg/schemachange/cutover.go b/pkg/schemachange/cutover.go index 076c7c0b..86e20365 100644 --- a/pkg/schemachange/cutover.go +++ b/pkg/schemachange/cutover.go @@ -82,6 +82,18 @@ type CutoverOptions struct { // wall clock. Tests inject one to drive the retry loop without real // time passing. Sleep func(ctx context.Context, d time.Duration) error + + // requireWriteFence makes the swap of a quiesced proof refuse unless the + // source still carries the engine's write fence. Only the Runner, which + // installs the fence, sets it; a caller of Cutover that quiesces the + // table by its own means has no fence to find. + requireWriteFence bool +} + +// withWriteFence is the options with the swap held to the write fence. +func (o CutoverOptions) withWriteFence() CutoverOptions { + o.requireWriteFence = true + return o } // validate refuses bounds the retry loop could not honour. @@ -206,7 +218,7 @@ func cutoverWithRetry(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.Tabl attempts := opts.lockAttempts() var last error for attempt := 1; attempt <= attempts; attempt++ { - swapped, backend, err := swapOnce(ctx, pool, lock, ready, drain, opts.Options) + swapped, backend, err := swapOnce(ctx, pool, lock, ready, drain, opts.requireWriteFence, opts.Options) if err == nil { swapped.attempts = attempt return swapped, nil diff --git a/pkg/schemachange/cutover_swap.go b/pkg/schemachange/cutover_swap.go index af09be50..500e2c3b 100644 --- a/pkg/schemachange/cutover_swap.go +++ b/pkg/schemachange/cutover_swap.go @@ -11,13 +11,14 @@ import ( "github.com/block/pg-sprite/pkg/preflight" ) -// swapOnce is one attempt at the swap transaction: lock, drain, re-check -// the shape, re-gate, rename, hand off, recheck, commit. Any error leaves +// swapOnce is one attempt at the swap transaction: lock, confirm the write +// fence when the caller requires it, drain, re-check the shape, re-gate, +// rename, hand off, recheck, commit. Any error leaves // the transaction rolled back on the server unless the client stopped // hearing from it, which the caller resolves by following the backend // returned here to its exit and then inspecting the catalog. The backend // is zero when the attempt ended before its transaction ran a statement. -func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, opts Options) (SwappedTable, swapBackend, error) { +func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, requireFence bool, opts Options) (SwappedTable, swapBackend, error) { built := ready.built tx, err := pool.Begin(ctx) if err != nil { @@ -33,7 +34,7 @@ func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSes if err != nil { return SwappedTable{}, swapBackend{}, err } - swapped, err := swapInTx(ctx, tx, lock, ready, drain, opts) + swapped, err := swapInTx(ctx, tx, lock, ready, drain, requireFence, opts) if err != nil { return SwappedTable{}, backend, err } @@ -45,7 +46,7 @@ func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSes // swapInTx runs every statement of the swap inside the open transaction, // up to but not including COMMIT. -func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, opts Options) (SwappedTable, error) { +func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, requireFence bool, opts Options) (SwappedTable, error) { built := ready.built if err := setGateSession(ctx, tx, built, opts); err != nil { return SwappedTable{}, err @@ -56,6 +57,14 @@ func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, rea if err := lockForSwap(ctx, tx, built); err != nil { return SwappedTable{}, err } + // A quiesced proof is only as good as the fence that kept the source + // quiet; a live proof is finished by the drain instead. + // INV: CO-1 + if requireFence && !ready.verified.Live() { + if err := confirmWriteFence(ctx, tx, built); err != nil { + return SwappedTable{}, err + } + } if drain != nil { if err := drain(ctx, tx); err != nil { return SwappedTable{}, fmt.Errorf("drain captured changes into %s.%s: %w", built.Schema(), built.ShadowTable(), err) diff --git a/pkg/schemachange/doc.go b/pkg/schemachange/doc.go index fca971f2..01ca0aaf 100644 --- a/pkg/schemachange/doc.go +++ b/pkg/schemachange/doc.go @@ -16,9 +16,11 @@ // the checkpoint row at every phase transition and dispatching on the row // it finds, so a run that stops anywhere resumes from the row. The target's // proof chooses the route. A quiesced table's quiescence is enforced, not -// assumed: the run fences the source with a statement trigger that refuses -// every write until the swap, and the fence stays on a stopped run's -// source until it is resumed or abandoned with DropWriteFence. A table +// assumed: the run fences the source with a statement trigger, enabled +// ALWAYS, that refuses every write until the swap, and the fence stays on a +// stopped run's source until it is resumed or abandoned with +// DropWriteFence. The quiesced swap refuses if the fence is no longer +// there. A table // under writes is captured instead: the run creates the slot with the row // that owns it, applies the stream to the shadow beside the copy, catches // up to a marker past every change committed so far, verifies live, and diff --git a/pkg/schemachange/fence.go b/pkg/schemachange/fence.go index e6d283c6..1762175f 100644 --- a/pkg/schemachange/fence.go +++ b/pkg/schemachange/fence.go @@ -49,16 +49,45 @@ $pgsprite$` // a CREATE the other has in flight. const writeFenceLockSQL = "SELECT pg_advisory_xact_lock($1, hashtext(quote_ident($2) || '.' || quote_ident($3)))" +// writeFenceTriggerType is pg_trigger.tgtype for the fence as installed: a +// statement-level BEFORE trigger on INSERT (4), DELETE (8), UPDATE (16), and +// TRUNCATE (32), the BEFORE bit being 2 and the row-level bit unset. +const writeFenceTriggerType = 2 + 4 + 8 + 16 + 32 + +// triggerEnabledAlways is pg_trigger.tgenabled for a trigger that fires in +// every session, whatever its session_replication_role. +const triggerEnabledAlways = "A" + +// writeFenceStateSQL reads the enablement of the engine's fence on one +// relation: the row exists only for a trigger of the fence's name, shape, +// and function in the engine's schema. Every catalog name is qualified, so +// the read does not depend on the session's search_path (CO-9). +const writeFenceStateSQL = ` +SELECT tg.tgenabled::pg_catalog.text +FROM pg_catalog.pg_trigger tg +JOIN pg_catalog.pg_proc p ON p.oid OPERATOR(pg_catalog.=) tg.tgfoid +JOIN pg_catalog.pg_namespace pn ON pn.oid OPERATOR(pg_catalog.=) p.pronamespace +WHERE tg.tgrelid OPERATOR(pg_catalog.=) $1::pg_catalog.oid + AND tg.tgname::pg_catalog.text OPERATOR(pg_catalog.=) $2::pg_catalog.text + AND p.proname::pg_catalog.text OPERATOR(pg_catalog.=) $2::pg_catalog.text + AND pn.nspname::pg_catalog.text OPERATOR(pg_catalog.=) $3::pg_catalog.text + AND tg.tgtype::pg_catalog.int4 OPERATOR(pg_catalog.=) $4::pg_catalog.int4 + AND NOT tg.tgisinternal` + // installWriteFence puts the write fence on the source, in one bounded // transaction as the engine role and only while the caller holds the // table lock. The trigger is a statement-level BEFORE trigger on INSERT, // UPDATE, DELETE, and TRUNCATE, so it fires once per statement and costs // the run nothing per row; installing it waits, under lock_timeout, for // any write already in flight to finish, which is the point at which the -// table is quiesced. The install is idempotent, so a resumed run installs -// over the fence the stopped run left. The trigger is created as the -// engine role, not the owner: the function lives in the engine's schema, -// and the owner needs no access to that schema for the trigger to fire. +// table is quiesced. The trigger is enabled ALWAYS, so it also fires for a +// session running under session_replication_role = replica, which skips +// every ordinary trigger. The install is idempotent, so a resumed run +// installs over the fence the stopped run left, and a fence that run left +// disabled, or enabled for origin sessions only, comes back enabled +// ALWAYS. The trigger is created as the engine role, not the owner: the +// function lives in the engine's schema, and the owner needs no access to +// that schema for the trigger to fire. func installWriteFence(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, target preflight.CopySwapTarget, opts Options) error { if err := requireTableLock(lock, target.Schema(), target.Table()); err != nil { return err @@ -98,12 +127,45 @@ func installWriteFence(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.Tab if _, err := tx.Exec(ctx, create); err != nil { return fmt.Errorf("install write fence on %s.%s: %w", target.Schema(), target.Table(), err) } + // INV: CO-1 — a session in replica role fires no ORIGIN trigger, so the + // fence must be ALWAYS to refuse its writes too. + enable := "ALTER TABLE " + source + " ENABLE ALWAYS TRIGGER " + pgx.Identifier{WriteFenceTrigger}.Sanitize() + if _, err := tx.Exec(ctx, enable); err != nil { + return fmt.Errorf("enable write fence always on %s.%s: %w", target.Schema(), target.Table(), err) + } if err := tx.Commit(ctx); err != nil { return fmt.Errorf("commit write fence install: %w", err) } return nil } +// confirmWriteFence refuses the swap of a quiesced proof unless the source +// still carries the engine's fence, enabled ALWAYS. It runs under the swap's +// ACCESS EXCLUSIVE lock, so the trigger cannot be disabled or dropped +// between this read and the rename. The fence is what makes the proof's +// quiescence true rather than assumed; a source whose fence was disabled or +// dropped since the pass may have taken writes the pass never saw, and the +// run stops there for a fresh install and a fresh pass. +func confirmWriteFence(ctx context.Context, tx pgx.Tx, built BuiltShadow) error { + schema, table := built.Schema(), built.SourceTable() + var enabled string + err := tx.QueryRow(ctx, writeFenceStateSQL, built.SourceOID(), WriteFenceTrigger, preflight.EngineSchema, writeFenceTriggerType).Scan(&enabled) + if errors.Is(err, pgx.ErrNoRows) { + // INV: CO-1 + return refuse(CauseCutoverUnfenced, nil, "source %s.%s carries no %s trigger calling %s.%s, so the quiesced proof no longer holds", + schema, table, WriteFenceTrigger, preflight.EngineSchema, WriteFenceTrigger) + } + if err != nil { + return fmt.Errorf("read write fence on %s.%s: %w", schema, table, err) + } + if enabled != triggerEnabledAlways { + // INV: CO-1 + return refuse(CauseCutoverUnfenced, nil, "the write fence on %s.%s has tgenabled %q, not %q, so a write could pass it", + schema, table, enabled, triggerEnabledAlways) + } + return nil +} + // DropWriteFence takes the write fence off the source of a run that is // being abandoned, so the table takes writes again. A run that completes // needs no call: the fence goes to the retained source with the swap and diff --git a/pkg/schemachange/refusal.go b/pkg/schemachange/refusal.go index b9fa0379..0c631fd2 100644 --- a/pkg/schemachange/refusal.go +++ b/pkg/schemachange/refusal.go @@ -85,6 +85,12 @@ const ( // the stream — and no drain to apply those changes under the swap lock // (CO-1). CauseCutoverUndrained RefusalCause = "cutover-undrained" + // CauseCutoverUnfenced means the swap of a quiesced proof found the + // source without the engine's write fence enabled ALWAYS under the + // swap lock (CO-1): the trigger was disabled, dropped, or replaced + // since it was installed, so a write may have reached the source + // after the pass that proved the shadow equal to it. + CauseCutoverUnfenced RefusalCause = "cutover-unfenced" // CauseDrainDeferred means the shadow refused a batch the swap's drain // had to apply under the swap lock (CO-6): a unique-key move the // changes since the live pass cannot be ordered into the shadow. No @@ -218,6 +224,7 @@ func RefusalCauses() []RefusalCause { CauseIdentityHandoff, CauseCutoverUnverified, CauseCutoverUndrained, + CauseCutoverUnfenced, CauseDrainDeferred, CauseRelationReplaced, CauseSchemaDrift, @@ -259,7 +266,7 @@ func (c RefusalCause) Invariant() string { case CauseShadowOwner, CauseForeignRelation, CauseGrantsDiffer, CauseIdentityHandoff, CauseSchemaDrift, CauseFidelityDrift, CauseIndexInvalid, CauseNameTaken: return "ST-5" - case CauseCutoverUnverified, CauseCutoverUndrained: + case CauseCutoverUnverified, CauseCutoverUndrained, CauseCutoverUnfenced: return "CO-1" case CauseDrainDeferred: return "CO-6" @@ -296,8 +303,8 @@ func (c RefusalCause) Class() refusal.Class { case CauseLockLost, CauseLockHeldElsewhere, CauseLockUnconfirmed, CauseSourceShape, CauseShadowOwner, CauseForeignRelation, CauseIdentityHandoff, CauseDrainDeferred, CauseRelationReplaced, CauseSchemaDrift, CauseFidelityDrift, CauseIndexInvalid, - CauseNameTaken, CauseOutcomeAmbiguous, CauseShadowUnrecorded, CauseResumeShadowGone, - CauseResumeSlotForeign, CauseResumeSlotHeld: + CauseNameTaken, CauseOutcomeAmbiguous, CauseCutoverUnfenced, CauseShadowUnrecorded, + CauseResumeShadowGone, CauseResumeSlotForeign, CauseResumeSlotHeld: return refusal.ClassEnvironmental case CauseStatementConversion: return refusal.ClassCapabilityBoundary diff --git a/pkg/schemachange/refusal_test.go b/pkg/schemachange/refusal_test.go index 546623f2..712a2f56 100644 --- a/pkg/schemachange/refusal_test.go +++ b/pkg/schemachange/refusal_test.go @@ -39,6 +39,7 @@ func TestEveryRefusalCauseNamesItsInvariant(t *testing.T) { CauseNameTaken: "ST-5", CauseCutoverUnverified: "CO-1", CauseCutoverUndrained: "CO-1", + CauseCutoverUnfenced: "CO-1", CauseDrainDeferred: "CO-6", CauseSwapMismatch: "ST-6", CauseOutcomeAmbiguous: "LK-4", @@ -87,6 +88,7 @@ func TestEveryRefusalCauseNamesItsClass(t *testing.T) { CauseIdentityHandoff: refusal.ClassEnvironmental, CauseCutoverUnverified: refusal.ClassInvariantViolation, CauseCutoverUndrained: refusal.ClassInvariantViolation, + CauseCutoverUnfenced: refusal.ClassEnvironmental, CauseDrainDeferred: refusal.ClassEnvironmental, CauseRelationReplaced: refusal.ClassEnvironmental, CauseSchemaDrift: refusal.ClassEnvironmental, diff --git a/pkg/schemachange/runner.go b/pkg/schemachange/runner.go index 38ca6273..95c79a59 100644 --- a/pkg/schemachange/runner.go +++ b/pkg/schemachange/runner.go @@ -130,10 +130,18 @@ func (o RunOptions) livePoll() time.Duration { // no WAL and applies no captured change, so the copy and the // verification pass are the whole proof of data equality (CO-1). The // Runner enforces that itself with a write fence on the source — a -// statement trigger that refuses INSERT, UPDATE, DELETE, and TRUNCATE -// with SQLSTATE WriteFenceSQLState — installed before the copy starts -// and in place until the swap, across any stop and resume in between; -// a writer that reaches the table gets an error, never a silent loss. +// statement trigger, enabled ALWAYS so that it fires in every session +// whatever its session_replication_role, that refuses INSERT, UPDATE, +// DELETE, and TRUNCATE with SQLSTATE WriteFenceSQLState — installed +// before the copy starts and in place until the swap, across any stop +// and resume in between. A writer that reaches the table while the +// fence is in place gets an error. The swap confirms, under its +// ACCESS EXCLUSIVE lock, that the fence is still there and enabled +// ALWAYS, and refuses as CauseCutoverUnfenced when it is not, leaving +// the next Run to install it again and verify again. The fence is not +// tamper-proof: a role that owns the table can disable it, write, and +// put it back between the verification pass and the swap, and that +// write is not seen. // - A live run, for a target proven to decode WAL. The source keeps // taking writes throughout: the run owns a replication slot under the // target's derived name, decodes the source's changes from it, and @@ -410,7 +418,7 @@ func (r *Runner) swapQuiesced(ctx context.Context, pool *pgxpool.Pool, lock *dbc return report, err } } - swapped, err := Cutover(ctx, pool, lock, ready, nil, r.opts.Cutover) + swapped, err := Cutover(ctx, pool, lock, ready, nil, r.opts.Cutover.withWriteFence()) if err != nil { return report, err } diff --git a/pkg/schemachange/runner_fence_integration_test.go b/pkg/schemachange/runner_fence_integration_test.go index 1ff05096..40505549 100644 --- a/pkg/schemachange/runner_fence_integration_test.go +++ b/pkg/schemachange/runner_fence_integration_test.go @@ -222,6 +222,150 @@ func TestRunnerDropsTheShadowWhenItsFirstSaveFails(t *testing.T) { assert.Equal(t, checkpoint.PhaseDone, result.Checkpoint.Phase) } +// fenceEnabled reads pg_trigger.tgenabled of the fence on table: "A" fires +// in every session, "O" only in origin sessions, "D" never. +func (f runnerFixture) fenceEnabled(t *testing.T, table string) string { + t.Helper() + var enabled string + require.NoError(t, f.pool.QueryRow(t.Context(), ` + SELECT tg.tgenabled::text FROM pg_trigger tg + JOIN pg_class c ON c.oid = tg.tgrelid JOIN pg_namespace n ON n.oid = c.relnamespace + WHERE n.nspname = $1 AND c.relname = $2 AND tg.tgname = $3 AND NOT tg.tgisinternal`, + f.schema, table, schemachange.WriteFenceTrigger).Scan(&enabled)) + return enabled +} + +// assertReplicaWritesFenced checks that a session running under +// session_replication_role = replica, which skips every trigger not enabled +// ALWAYS, is refused by the fence on every kind of write to table. +func (f runnerFixture) assertReplicaWritesFenced(t *testing.T, table string) { + t.Helper() + conn, err := f.pool.Acquire(t.Context()) + require.NoError(t, err) + defer conn.Release() + _, err = conn.Exec(t.Context(), "SET session_replication_role = replica") + require.NoError(t, err, "the test role may set session_replication_role") + defer func() { + _, err := conn.Exec(context.WithoutCancel(t.Context()), "RESET session_replication_role") + assert.NoError(t, err) + }() + before := f.rowCount(t, table) + for kind, sql := range f.writeStatements(table) { + _, err := conn.Exec(t.Context(), sql) + var pgErr *pgconn.PgError + require.ErrorAs(t, err, &pgErr, "%s from a replica-role session should be refused by the server", kind) + assert.Equal(t, schemachange.WriteFenceSQLState, pgErr.Code, "%s is refused with the fence's SQLSTATE", kind) + } + assert.Equal(t, before, f.rowCount(t, table), "a refused write changes nothing") +} + +// A session running under session_replication_role = replica fires no +// ordinary trigger, so a fence enabled for origin sessions only would let +// its write through and the swap would drop the row. The fence is enabled +// ALWAYS, so every kind of write from such a session is refused at the +// latest point a writer could reach the source, after the verification pass +// and with the swap still ahead, and the run completes. +func TestRunnerFenceRefusesASessionInReplicaRole(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + checked := false + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + assert.Equal(t, "A", f.fenceEnabled(t, "orders"), "the fence fires in every session") + f.assertReplicaWritesFenced(t, "orders") + checked = true + } + return nil + }} + + _, err := f.run(t, "orders", `ALTER TABLE %s.orders DROP COLUMN note`, opts) + require.NoError(t, err) + + assert.True(t, checked, "the hook saw the cutover save") +} + +// The swap of a quiesced proof is gated on the fence still being there. A +// fence disabled after the verification pass, with the swap ahead, no longer +// keeps the source quiet, so the swap refuses as cutover-unfenced and leaves +// the source in place. The row stays in cutover, and the next Run installs +// the fence again, verifies again, and completes. +func TestRunnerRefusesTheSwapWhenTheFenceWasDisabled(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + const alter = `ALTER TABLE %s.orders DROP COLUMN note` + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + f.exec(t, `ALTER TABLE %s.orders DISABLE TRIGGER pgsprite_write_fence`) + } + return nil + }} + + _, err := f.run(t, "orders", alter, opts) + + require.Error(t, err) + assert.Equal(t, schemachange.CauseCutoverUnfenced, schemachange.RefusalCauseOf(err)) + assert.ErrorIs(t, err, schemachange.ErrCutoverRolledBack) + assert.True(t, f.hasColumn(t, "orders", "note"), "the source was not swapped out") + assert.True(t, f.relationExists(t, schemachange.ShadowName(f.schema, "orders")), "the shadow is still in place") + assert.Equal(t, checkpoint.PhaseCutover, f.inspectRow(t, "orders").Phase) + assert.Equal(t, "D", f.fenceEnabled(t, "orders"), "the refusal does not touch the fence") + + f.assertResumedToDone(t, alter, schemachange.RunOptions{}) + f.assertWritable(t, "orders") +} + +// A fence dropped after the verification pass is refused the same way: the +// trigger the proof relies on is gone, so the swap does not proceed, and the +// resumed run installs it again. +func TestRunnerRefusesTheSwapWhenTheFenceWasDropped(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + const alter = `ALTER TABLE %s.orders DROP COLUMN note` + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + f.exec(t, `DROP TRIGGER pgsprite_write_fence ON %s.orders`) + } + return nil + }} + + _, err := f.run(t, "orders", alter, opts) + + require.Error(t, err) + assert.Equal(t, schemachange.CauseCutoverUnfenced, schemachange.RefusalCauseOf(err)) + assert.True(t, f.hasColumn(t, "orders", "note"), "the source was not swapped out") + assert.False(t, f.hasFence(t, "orders")) + + f.assertResumedToDone(t, alter, schemachange.RunOptions{}) + f.assertWritable(t, "orders") +} + +// A fence an earlier engine left enabled for origin sessions only is not +// refused for good: the resumed run installs over it, which enables it +// ALWAYS, so the swap finds the fence it requires. +func TestRunnerResumeUpgradesAnOriginFenceToAlways(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + const alter = `ALTER TABLE %s.orders DROP COLUMN note` + stop := &stopAfter{phase: checkpoint.PhaseCopying} + + _, err := f.run(t, "orders", alter, schemachange.RunOptions{AfterCheckpoint: stop.hook}) + require.ErrorIs(t, err, errStopHere) + f.exec(t, `ALTER TABLE %s.orders ENABLE TRIGGER pgsprite_write_fence`) + require.Equal(t, "O", f.fenceEnabled(t, "orders"), "the stopped run's fence is origin-only") + + upgraded := false + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + assert.Equal(t, "A", f.fenceEnabled(t, "orders"), "the resume enabled the fence ALWAYS") + upgraded = true + } + return nil + }} + f.assertResumedToDone(t, alter, opts) + + assert.True(t, upgraded, "the hook saw the cutover save") +} + // cancelClock cancels a context whenever it is read, and tells the wall // clock's time. type cancelClock struct { From 31641e327f7e513905bab7b56dc08e12d91746e8 Mon Sep 17 00:00:00 2001 From: Kiran Muddukrishna Date: Sun, 18 Oct 2026 21:27:30 -0700 Subject: [PATCH 2/2] schemachange: confirm the fence's shape at the swap A trigger of the engine's name and function enabled ALWAYS could still have been replaced by a conditional one; the swap now also requires no WHEN clause, no column list, no arguments, and not deferrable. --- docs/engine-role.md | 33 +++++---- docs/invariants.md | 28 +++++--- docs/refusal-classes.md | 2 +- docs/schemabot-integration.md | 4 +- pkg/schemachange/cutover_swap.go | 10 +-- pkg/schemachange/doc.go | 12 ++-- pkg/schemachange/fence.go | 32 ++++++--- pkg/schemachange/refusal.go | 7 +- pkg/schemachange/runner.go | 13 ++-- .../runner_fence_integration_test.go | 67 +++++++++++++++++++ 10 files changed, 151 insertions(+), 57 deletions(-) diff --git a/docs/engine-role.md b/docs/engine-role.md index 576c23e5..428a7b01 100644 --- a/docs/engine-role.md +++ b/docs/engine-role.md @@ -100,24 +100,31 @@ nothing writes the table: it installs a statement-level `BEFORE INSERT OR UPDATE TRUNCATE` trigger, `pgsprite_write_fence`, on the source before its first checkpoint save, and every write the trigger sees is refused with SQLSTATE `55006` (`object_in_use`) and a message naming the table. The trigger is enabled `ALWAYS`, so it fires for a session running under -`session_replication_role = replica` as well. The trigger calls a function of the same name in the engine's own `pgsprite` -schema, created once per database as the engine role and owned by it, so the table's owner -needs no access to that schema for the fence to fire. The fence stays on a stopped run's source -until the run is resumed or abandoned (`DropWriteFence`); a completed run carries it to the -retained `_old` table with the swap, where it is dropped with the table or, under `KeepOld`, -taken off so the retained table can be renamed back and take writes at once. +`session_replication_role = replica` as well. The writers that run in that role are the apply +worker of a logical-replication subscription that writes into the source, and a bulk restore +run with triggers disabled; a fence enabled for origin sessions only would let their writes +through to be lost with the old table at the swap. For a subscription apply worker the +refusal means a worker that errors and retries until the run completes or is abandoned, with +the publisher retaining WAL for the subscription's slot meanwhile. The trigger calls a +function of the same name in the engine's own `pgsprite` schema, created once per database as +the engine role and owned by it, so the table's owner needs no access to that schema for the +fence to fire. The fence stays on a stopped run's source until the run is resumed or +abandoned (`DropWriteFence`); a completed run carries it to the retained `_old` table with +the swap, where it is dropped with the table or, under `KeepOld`, taken off so the retained +table can be renamed back and take writes at once. What this needs: `CREATE TRIGGER` and `ALTER TABLE … ENABLE ALWAYS TRIGGER` on a table require ownership, which the Tier 1 membership in the owning role already gives; the function needs `CREATE` on the `pgsprite` schema, which the engine owns. A pre-provisioned deployment that created the schema for the engine needs nothing further once the schema is owned by the engine -role. The fence does not stop a role that owns the table, which can disable or drop it; the -swap refuses (`cutover-unfenced`) a fence found disabled or gone under its lock, but a role that -disables the fence, writes, and restores it between the verification pass and the swap is not -seen. A deployment whose writers can hold table ownership has to quiesce the table by its own -means, and a difference the verification pass finds is reported, never repaired. Before `Run`, quiesce the table as far -as the application allows — stop or pause its writers, or route them around it — so the fence -refuses nothing in practice; the fence is the proof, not the plan. +role. The fence does not stop a role that owns the table, which can disable, drop, or replace +it. The swap refuses (`cutover-unfenced`) a fence found under its lock disabled, dropped, or +replaced by a trigger of another shape, but a role that disables the fence, writes, and +restores it between the verification pass and the swap is not seen. A deployment whose +writers can hold table ownership has to quiesce the table by its own means, and a difference +the verification pass finds is reported, never repaired. Before `Run`, quiesce the table as +far as the application allows — stop or pause its writers, or route them around it — so the +fence refuses nothing in practice; the fence is the proof, not the plan. Two cluster-level *facts* — settings, not grants — accompany Tier 3 and are checked by `preflight.CheckCopySwapEnvironment` once the privilege check has passed and the shape check diff --git a/docs/invariants.md b/docs/invariants.md index cacd8557..abf7404a 100644 --- a/docs/invariants.md +++ b/docs/invariants.md @@ -62,17 +62,23 @@ its first save, and keeps it through every stop until the swap carries it to the table, so the copy and one quiesced pass are the whole proof (fences-the-source, refuses-a-replica-role-session, leaves-the-fence-on-a-stopped-run, unfences-the-old-table-it-keeps, stops-on-a-shadow-that-differs tests). A resumed run installs -over the fence, which enables it `ALWAYS` again (resume-upgrades-an-origin-fence test). The -quiesced swap re-reads the fence under its `ACCESS EXCLUSIVE` lock and refuses the proof as -`cutover-unfenced` unless the engine's trigger is still on the source, enabled `ALWAYS` -(refuses-when-the-fence-was-disabled and refuses-when-the-fence-was-dropped tests); a role -that owns the table can still disable the fence, write, and restore it between the pass and -the swap, and that write is not seen. A live run installs no fence, so its swap checks none. -On a decoding target it runs `CheckLive` under -`DivergenceAbort` once the catch-up has passed a marker written after the copy, with the -capture's own `Catchup` as the pass's confirmed source, and hands the swap a drain that -reopens the stream from the position the capture confirmed and applies it under the swap's -lock (carries-a-live-table-from-no-row-to-done test). +over the fence, which enables it `ALWAYS` again (resume-upgrades-an-origin-fence test). + +The quiesced swap confirms the fence once more, under its `ACCESS EXCLUSIVE` lock: it refuses +the proof as `cutover-unfenced` unless the engine's trigger is still on the source, of the +shape the install gave it — no `WHEN` condition, no `UPDATE OF` column list, no arguments, not +deferrable — and enabled `ALWAYS` (refuses-when-the-fence-was-disabled, +refuses-when-the-fence-was-dropped, refuses-when-the-fence-is-origin-only, and +refuses-when-the-fence-was-replaced-by-a-conditional-one tests). The check has a stated +limit: a role that owns the table can disable the fence, write, and restore it between the +pass and the swap, and that write is not seen. A live run installs no fence, so its swap +checks none. + +On a decoding target the `Runner` runs `CheckLive` under `DivergenceAbort` once the catch-up +has passed a marker written after the copy, with the capture's own `Catchup` as the pass's +confirmed source, and hands the swap a drain that reopens the stream from the position the +capture confirmed and applies it under the swap's lock (carries-a-live-table-from-no-row-to-done +test). *Source:* [design-principles](design-principles.md#correctness-and-safety), risks-and-mitigations; Spirit's "never skip it". diff --git a/docs/refusal-classes.md b/docs/refusal-classes.md index 05d9d2d5..208b4d12 100644 --- a/docs/refusal-classes.md +++ b/docs/refusal-classes.md @@ -196,7 +196,7 @@ causes above: the verdict's `cause` is the shadow cause unchanged and its `class | `cutover-unverified` | The cutover gate was handed an empty built-shadow or verified-shadow proof, two proofs naming different relations or different relation OIDs (a shadow rebuilt under the same name after the checksum ran), or a verified watermark that does not cover the whole key space ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Only the copier and verifier mint those proofs for the same table; a caller that reaches the gate without them skipped the checksum, which no retry corrects. | | `cutover-undrained` | The cutover was handed a verified-shadow proof a live pass minted — the shadow equal to the source as of a position the stream had reached, with every change since still in the stream — and no drain to apply those changes under the swap lock ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Pass a `DrainFunc` that applies the stream through the final position under the swap lock, or stop the stream and re-verify with a quiesced `Check`; the proof is sound, the hand-off to the swap is not. | | `cutover-drain-deferred` | The shadow refused a batch the swap's drain had to apply under the swap lock: a unique-key move among the changes committed since the live verification pass that cannot be ordered into the shadow, with no copy left to change the outcome ([CO-6](invariants.md#co-6--unique-secondary-key-moves-must-converge-postgresql-specific-gap)) | `environmental` | The swap transaction is rolled back and the source stays live; the row is left in `cutover` with the position the drain reopened from, so a resume drains the same changes again once the rows the move touched have settled. A move the shadow cannot ever order is re-planned. | -| `cutover-unfenced` | The quiesced `Runner`'s swap, under its `ACCESS EXCLUSIVE` lock, did not find the engine's write fence on the source enabled `ALWAYS`: the trigger was disabled, dropped, or replaced since the run installed it ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `environmental` | Something other than the engine changed the source's triggers under the run, so a write may have reached it after the verification pass. The swap rolled back and nothing is swapped; the next `Run` installs the fence again, verifies again, and swaps, and a write that did land makes that pass stop on a divergence. | +| `cutover-unfenced` | The quiesced `Runner`'s swap, under its `ACCESS EXCLUSIVE` lock, did not find the engine's write fence on the source enabled `ALWAYS`: the trigger was disabled, dropped, or replaced by one of another shape — a `WHEN` condition, an `UPDATE OF` column list, arguments, or deferral — since the run installed it ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `environmental` | Something other than the engine changed the source's triggers under the run, so a write may have reached it after the verification pass. The swap rolled back and nothing is swapped; the next `Run` installs the fence again, verifies again, and swaps, and a write that did land makes that pass stop on a divergence. | | `cutover-relation-replaced` | The source or the shadow under its name is no longer the relation the build proved: its OID moved; or, after the swap, the derived `_old` name is borne by a relation other than the source the swap retained, or by nothing while the retained source still exists under another name ([ST-6](invariants.md#st-6--preflight-before-the-first-write)) | `environmental` | The proof describes a table that no longer exists; before the swap, drop the shadow and start the change over from preflight; after it, an operator reads the catalog to learn what took the `_old` name, or where the retained source went, before anything is dropped. | | `cutover-schema-drift` | The source's or the shadow's introspected model no longer matches the fingerprint the build recorded: a column, constraint, index, storage parameter, or column-owned sequence declaration changed since ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | Someone changed a table under the run; the shadow no longer embodies the gated statement against this source, so it is dropped and the change re-planned. | | `cutover-fidelity-drift` | The source's or the shadow's owner, grants, policies, comment, replica identity, per-column or extended statistics targets, unvalidated checks, or identity columns and their sequences no longer match the snapshot the build recorded, or the source's grants no longer match the shadow's ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | The metadata the swap would carry forward moved after it was copied; re-synchronising it under the lock is a planned capability, and until then the shadow is rebuilt. | diff --git a/docs/schemabot-integration.md b/docs/schemabot-integration.md index 2a835499..c51915fb 100644 --- a/docs/schemabot-integration.md +++ b/docs/schemabot-integration.md @@ -327,8 +327,8 @@ the slot under the row (decoding target): | `Run` outcome | What it means | Orchestrator action | | --- | --- | --- | | `*decode.SlotExistsError` (decoding target, first start) | A logical slot of the engine's derived name already exists and no row names it: a run that ended before its row landed, or a slot made by hand. The run made nothing and dropped nothing — the slot, its publication, and the absence of a row are as the run found them | Run `Store.ReapOrphanSlots`, which drops the slot once nothing holds it, and then `Run` again; a slot that stays held is an operator's to inspect | -| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a role that owns the table and disables it) to find | -| `cutover-unfenced` (quiesced target, from `Cutover`, through `Run`) | The swap, under its lock, found the engine's write fence on the source disabled, dropped, or not enabled `ALWAYS`; nothing is swapped and the row is in `cutover` | Retry `Run`: it installs the fence again, verifies again, and swaps; a write that landed while the fence was down makes the verification pass stop on a `*checksum.DivergenceError`. Find what changed the source's triggers | +| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a role that owns the table and disables it) to find. A writer in `session_replication_role = replica` — a logical-replication subscription's apply worker, a bulk restore with triggers disabled — does not bypass the fence, which is enabled `ALWAYS`: it is refused by it, and a subscription that errors and retries for the run's length is that refusal, not a divergence | +| `cutover-unfenced` (quiesced target, from `Cutover`, through `Run`) | The swap, under its lock, found the engine's write fence on the source disabled, dropped, replaced by a trigger of another shape, or not enabled `ALWAYS`; nothing is swapped and the row is in `cutover` | Retry `Run`: it installs the fence again, verifies again, and swaps; a write that landed while the fence was down makes the verification pass stop on a `*checksum.DivergenceError`. Find what changed the source's triggers | | `*checksum.UnresolvedError` (decoding target) | The live pass could not settle a chunk the source kept writing within its reread bound; nothing was found wrong and nothing was minted, and the row stays in `verifying`. The bound is `LiveOptions.Attempts` fresh rereads of one chunk (default `checksum.DefaultSettleAttempts`), each waited on for the catch-up up to `LiveOptions.SettleWait`: a chunk written again between every two of its rereads never settles, so a table whose writes land on one chunk faster than the pass can read it twice is the case, whatever the table's total write rate | Retry `Run`: the resumed row catches up again and runs a fresh pass; a chunk that never settles is a hot spot to wait out, write less often, or give more `Attempts` | | `cutover-drain-deferred` (decoding target) | The swap's drain met a unique-key move it could not order into the shadow; the swap rolled back and the row is in `cutover` with the position the drain reopened from | Retry `Run`: `InspectSwapped` answers `ErrNotSwapped`, the row goes back to `catching_up`, and the same changes are drained again once the rows the move touched have settled | | `*decode.SlotLagExceededError` (decoding target) | The slot retained more WAL than `CatchupOptions.SlotLagCeiling` while the capture ran; the run stopped with the slot in place under the row | Retry `Run` once the source's write rate allows: the stream reopens at the row's position and drains the backlog. Abandoning the run (`DropShadow`, `decode.DropSlot`, `Store.Delete`) releases the WAL | diff --git a/pkg/schemachange/cutover_swap.go b/pkg/schemachange/cutover_swap.go index 500e2c3b..801717a4 100644 --- a/pkg/schemachange/cutover_swap.go +++ b/pkg/schemachange/cutover_swap.go @@ -13,11 +13,11 @@ import ( // swapOnce is one attempt at the swap transaction: lock, confirm the write // fence when the caller requires it, drain, re-check the shape, re-gate, -// rename, hand off, recheck, commit. Any error leaves -// the transaction rolled back on the server unless the client stopped -// hearing from it, which the caller resolves by following the backend -// returned here to its exit and then inspecting the catalog. The backend -// is zero when the attempt ended before its transaction ran a statement. +// rename, hand off, recheck, commit. Any error leaves the transaction +// rolled back on the server unless the client stopped hearing from it, +// which the caller resolves by following the backend returned here to its +// exit and then inspecting the catalog. The backend is zero when the +// attempt ended before its transaction ran a statement. func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, requireFence bool, opts Options) (SwappedTable, swapBackend, error) { built := ready.built tx, err := pool.Begin(ctx) diff --git a/pkg/schemachange/doc.go b/pkg/schemachange/doc.go index 01ca0aaf..814d1f8c 100644 --- a/pkg/schemachange/doc.go +++ b/pkg/schemachange/doc.go @@ -20,12 +20,12 @@ // ALWAYS, that refuses every write until the swap, and the fence stays on a // stopped run's source until it is resumed or abandoned with // DropWriteFence. The quiesced swap refuses if the fence is no longer -// there. A table -// under writes is captured instead: the run creates the slot with the row -// that owns it, applies the stream to the shadow beside the copy, catches -// up to a marker past every change committed so far, verifies live, and -// drains the stream under the swap's lock before it renames; a stopped -// run's slot stays until it is resumed or abandoned with decode.DropSlot. +// there as installed. A table under writes is captured instead: the run +// creates the slot with the row that owns it, applies the stream to the +// shadow beside the copy, catches up to a marker past every change +// committed so far, verifies live, and drains the stream under the swap's +// lock before it renames; a stopped run's slot stays until it is resumed +// or abandoned with decode.DropSlot. // The one exception to "a shadow is only dropped on abandonment" is a // first save whose row does not land: the run drops the shadow it just // built, and the slot with it, since a shadow no row records would refuse diff --git a/pkg/schemachange/fence.go b/pkg/schemachange/fence.go index 1762175f..3c375089 100644 --- a/pkg/schemachange/fence.go +++ b/pkg/schemachange/fence.go @@ -59,9 +59,15 @@ const writeFenceTriggerType = 2 + 4 + 8 + 16 + 32 const triggerEnabledAlways = "A" // writeFenceStateSQL reads the enablement of the engine's fence on one -// relation: the row exists only for a trigger of the fence's name, shape, -// and function in the engine's schema. Every catalog name is qualified, so -// the read does not depend on the session's search_path (CO-9). +// relation: the row exists only for a trigger of the fence's name, calling +// the fence's function in the engine's schema, of the shape the install +// gives it — a statement-level BEFORE trigger on every write statement +// (tgtype), with no WHEN condition (tgqual), no UPDATE OF column list +// (tgattr), no arguments (tgnargs), and not deferrable. A trigger of the +// same name and function whose shape differs fires for fewer statements +// than the fence, so it yields no row and the swap refuses it as a fence +// that is gone. Every catalog name is qualified, so the read does not +// depend on the session's search_path (CO-9). const writeFenceStateSQL = ` SELECT tg.tgenabled::pg_catalog.text FROM pg_catalog.pg_trigger tg @@ -72,6 +78,10 @@ WHERE tg.tgrelid OPERATOR(pg_catalog.=) $1::pg_catalog.oid AND p.proname::pg_catalog.text OPERATOR(pg_catalog.=) $2::pg_catalog.text AND pn.nspname::pg_catalog.text OPERATOR(pg_catalog.=) $3::pg_catalog.text AND tg.tgtype::pg_catalog.int4 OPERATOR(pg_catalog.=) $4::pg_catalog.int4 + AND tg.tgqual IS NULL + AND tg.tgattr::pg_catalog.text OPERATOR(pg_catalog.=) '' + AND tg.tgnargs::pg_catalog.int4 OPERATOR(pg_catalog.=) 0 + AND NOT tg.tgdeferrable AND NOT tg.tgisinternal` // installWriteFence puts the write fence on the source, in one bounded @@ -140,19 +150,21 @@ func installWriteFence(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.Tab } // confirmWriteFence refuses the swap of a quiesced proof unless the source -// still carries the engine's fence, enabled ALWAYS. It runs under the swap's -// ACCESS EXCLUSIVE lock, so the trigger cannot be disabled or dropped -// between this read and the rename. The fence is what makes the proof's -// quiescence true rather than assumed; a source whose fence was disabled or -// dropped since the pass may have taken writes the pass never saw, and the -// run stops there for a fresh install and a fresh pass. +// still carries the engine's fence, of the shape the install gave it and +// enabled ALWAYS. It runs under the swap's ACCESS EXCLUSIVE lock, so the +// trigger cannot be disabled, dropped, or replaced between this read and +// the rename. The fence is what makes the proof's quiescence true rather +// than assumed; a source whose fence was disabled, dropped, or replaced by +// a trigger that fires for fewer statements since the pass may have taken +// writes the pass never saw, and the run stops there for a fresh install +// and a fresh pass. func confirmWriteFence(ctx context.Context, tx pgx.Tx, built BuiltShadow) error { schema, table := built.Schema(), built.SourceTable() var enabled string err := tx.QueryRow(ctx, writeFenceStateSQL, built.SourceOID(), WriteFenceTrigger, preflight.EngineSchema, writeFenceTriggerType).Scan(&enabled) if errors.Is(err, pgx.ErrNoRows) { // INV: CO-1 - return refuse(CauseCutoverUnfenced, nil, "source %s.%s carries no %s trigger calling %s.%s, so the quiesced proof no longer holds", + return refuse(CauseCutoverUnfenced, nil, "source %s.%s carries no %s trigger of the fence's shape calling %s.%s, so the quiesced proof no longer holds", schema, table, WriteFenceTrigger, preflight.EngineSchema, WriteFenceTrigger) } if err != nil { diff --git a/pkg/schemachange/refusal.go b/pkg/schemachange/refusal.go index 0c631fd2..7634c285 100644 --- a/pkg/schemachange/refusal.go +++ b/pkg/schemachange/refusal.go @@ -87,9 +87,10 @@ const ( CauseCutoverUndrained RefusalCause = "cutover-undrained" // CauseCutoverUnfenced means the swap of a quiesced proof found the // source without the engine's write fence enabled ALWAYS under the - // swap lock (CO-1): the trigger was disabled, dropped, or replaced - // since it was installed, so a write may have reached the source - // after the pass that proved the shadow equal to it. + // swap lock (CO-1): the trigger was disabled, dropped, or replaced by + // one of another shape since it was installed, so a write may have + // reached the source after the pass that proved the shadow equal to + // it. CauseCutoverUnfenced RefusalCause = "cutover-unfenced" // CauseDrainDeferred means the shadow refused a batch the swap's drain // had to apply under the swap lock (CO-6): a unique-key move the diff --git a/pkg/schemachange/runner.go b/pkg/schemachange/runner.go index 6ef4c306..8b0a53cc 100644 --- a/pkg/schemachange/runner.go +++ b/pkg/schemachange/runner.go @@ -141,12 +141,13 @@ func (o RunOptions) livePoll() time.Duration { // before the copy starts and in place until the swap, across any stop // and resume in between. A writer that reaches the table while the // fence is in place gets an error. The swap confirms, under its -// ACCESS EXCLUSIVE lock, that the fence is still there and enabled -// ALWAYS, and refuses as CauseCutoverUnfenced when it is not, leaving -// the next Run to install it again and verify again. The fence is not -// tamper-proof: a role that owns the table can disable it, write, and -// put it back between the verification pass and the swap, and that -// write is not seen. +// ACCESS EXCLUSIVE lock, that the fence is still there, of the shape +// the install gave it, and enabled ALWAYS, and refuses as +// CauseCutoverUnfenced when it was disabled, dropped, or replaced, +// leaving the next Run to install it again and verify again. The +// fence is not tamper-proof: a role that owns the table can disable +// it, write, and put it back between the verification pass and the +// swap, and that write is not seen. // - A live run, for a target proven to decode WAL. The source keeps // taking writes throughout: the run owns a replication slot under the // target's derived name, decodes the source's changes from it, and diff --git a/pkg/schemachange/runner_fence_integration_test.go b/pkg/schemachange/runner_fence_integration_test.go index 40505549..b70bf27e 100644 --- a/pkg/schemachange/runner_fence_integration_test.go +++ b/pkg/schemachange/runner_fence_integration_test.go @@ -11,6 +11,7 @@ import ( "github.com/stretchr/testify/require" "github.com/block/pg-sprite/pkg/checkpoint" + "github.com/block/pg-sprite/pkg/checksum" "github.com/block/pg-sprite/pkg/schemachange" ) @@ -339,6 +340,72 @@ func TestRunnerRefusesTheSwapWhenTheFenceWasDropped(t *testing.T) { f.assertWritable(t, "orders") } +// A fence left enabled for origin sessions only, by a later ENABLE TRIGGER +// after the verification pass, fires for no replica-role writer, so the +// swap refuses it as it refuses a disabled one: the proof's quiescence is +// only as strong as a fence that fires in every session. The refusal does +// not touch the fence; the resumed run installs over it. +func TestRunnerRefusesTheSwapWhenTheFenceIsOriginOnly(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + const alter = `ALTER TABLE %s.orders DROP COLUMN note` + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + f.exec(t, `ALTER TABLE %s.orders ENABLE TRIGGER pgsprite_write_fence`) + } + return nil + }} + + _, err := f.run(t, "orders", alter, opts) + + require.Error(t, err) + assert.Equal(t, schemachange.CauseCutoverUnfenced, schemachange.RefusalCauseOf(err)) + assert.True(t, f.hasColumn(t, "orders", "note"), "the source was not swapped out") + assert.Equal(t, "O", f.fenceEnabled(t, "orders"), "the refusal does not touch the fence") + + f.assertResumedToDone(t, alter, schemachange.RunOptions{}) + f.assertWritable(t, "orders") +} + +// A trigger of the fence's name and function that fires for fewer +// statements than the fence — here one with a WHEN condition that is never +// true, enabled ALWAYS — is not the fence, and the swap refuses it: a write +// made under it reached the source after the verification pass, and +// swapping would drop that row with the old table. The source keeps the +// row, and the resumed run's install replaces the trigger with the fence, +// so its pass stops on the divergence that write made. +func TestRunnerRefusesTheSwapWhenTheFenceWasReplacedByAConditionalOne(t *testing.T) { + f := newRunnerFixture(t) + f.createOrders(t) + const alter = `ALTER TABLE %s.orders DROP COLUMN note` + opts := schemachange.RunOptions{AfterCheckpoint: func(_ context.Context, cp checkpoint.Checkpoint) error { + if cp.Phase == checkpoint.PhaseCutover { + f.exec(t, ` + CREATE OR REPLACE TRIGGER pgsprite_write_fence + BEFORE INSERT OR UPDATE OR DELETE OR TRUNCATE ON %s.orders + FOR EACH STATEMENT + WHEN (false) + EXECUTE FUNCTION pgsprite.pgsprite_write_fence()`) + f.exec(t, `ALTER TABLE %s.orders ENABLE ALWAYS TRIGGER pgsprite_write_fence`) + f.exec(t, `INSERT INTO %s.orders (id, qty) VALUES (999999, 1)`) + } + return nil + }} + before := f.rowCount(t, "orders") + + _, err := f.run(t, "orders", alter, opts) + + require.Error(t, err) + assert.Equal(t, schemachange.CauseCutoverUnfenced, schemachange.RefusalCauseOf(err)) + assert.True(t, f.hasColumn(t, "orders", "note"), "the source was not swapped out") + assert.Equal(t, before+1, f.rowCount(t, "orders"), "the row the replaced trigger let through is still on the source") + assert.Equal(t, "A", f.fenceEnabled(t, "orders"), "the replaced trigger is enabled ALWAYS and is still not the fence") + + var divergence *checksum.DivergenceError + _, err = f.run(t, "orders", alter, schemachange.RunOptions{}) + require.ErrorAs(t, err, &divergence, "the resumed run's pass sees the row the pass before it did not") +} + // A fence an earlier engine left enabled for origin sessions only is not // refused for good: the resumed run installs over it, which enables it // ALWAYS, so the swap finds the fence it requires.