schemachange: enable the write fence always and check it at swap

docs/engine-role.md

@@ -99,23 +99,32 @@ A quiesced copy-and-swap run (`pkg/schemachange.Runner`) does not take the calle
 nothing writes the table: it installs a statement-level `BEFORE INSERT OR UPDATE OR DELETE OR
 TRUNCATE` trigger, `pgsprite_write_fence`, on the source before its first checkpoint save, and
 every write the trigger sees is refused with SQLSTATE `55006` (`object_in_use`) and a message
-naming the table. The trigger calls a function of the same name in the engine's own `pgsprite`
-schema, created once per database as the engine role and owned by it, so the table's owner
-needs no access to that schema for the fence to fire. The fence stays on a stopped run's source
-until the run is resumed or abandoned (`DropWriteFence`); a completed run carries it to the
-retained `_old` table with the swap, where it is dropped with the table or, under `KeepOld`,
-taken off so the retained table can be renamed back and take writes at once.
-
-What this needs: `CREATE TRIGGER` on a table requires the `TRIGGER` privilege on it or
+naming the table. The trigger is enabled `ALWAYS`, so it fires for a session running under
+`session_replication_role = replica` as well. The writers that run in that role are the apply
+worker of a logical-replication subscription that writes into the source, and a bulk restore
+run with triggers disabled; a fence enabled for origin sessions only would let their writes
+through to be lost with the old table at the swap. For a subscription apply worker the
+refusal means a worker that errors and retries until the run completes or is abandoned, with
+the publisher retaining WAL for the subscription's slot meanwhile. The trigger calls a
+function of the same name in the engine's own `pgsprite` schema, created once per database as
+the engine role and owned by it, so the table's owner needs no access to that schema for the
+fence to fire. The fence stays on a stopped run's source until the run is resumed or
+abandoned (`DropWriteFence`); a completed run carries it to the retained `_old` table with
+the swap, where it is dropped with the table or, under `KeepOld`, taken off so the retained
+table can be renamed back and take writes at once.
+
+What this needs: `CREATE TRIGGER` and `ALTER TABLE … ENABLE ALWAYS TRIGGER` on a table require
 ownership, which the Tier 1 membership in the owning role already gives; the function needs
 `CREATE` on the `pgsprite` schema, which the engine owns. A pre-provisioned deployment that
 created the schema for the engine needs nothing further once the schema is owned by the engine
-role. The fence does not stop a superuser, nor a session running under
-`session_replication_role = replica`, which fires no ordinary triggers; a deployment whose
-writers can hold either has to quiesce the table by its own means, and a difference the
-verification pass then finds is reported, never repaired. Before `Run`, quiesce the table as far
-as the application allows — stop or pause its writers, or route them around it — so the fence
-refuses nothing in practice; the fence is the proof, not the plan.
+role. The fence does not stop a role that owns the table, which can disable, drop, or replace
+it. The swap refuses (`cutover-unfenced`) a fence found under its lock disabled, dropped, or
+replaced by a trigger of another shape, but a role that disables the fence, writes, and
+restores it between the verification pass and the swap is not seen. A deployment whose
+writers can hold table ownership has to quiesce the table by its own means, and a difference
+the verification pass finds is reported, never repaired. Before `Run`, quiesce the table as
+far as the application allows — stop or pause its writers, or route them around it — so the
+fence refuses nothing in practice; the fence is the proof, not the plan.
 
 Two cluster-level *facts* — settings, not grants — accompany Tier 3 and are checked by
 `preflight.CheckCopySwapEnvironment` once the privilege check has passed and the shape check

docs/invariants.md

@@ -57,14 +57,28 @@ since under the swap lock (`cutover-undrained`); the mechanism is in
 [copy-and-swap-design.md](copy-and-swap-design.md#live-verification). The `Runner` on a
 quiesced target makes the quiescence its pass relies on true rather than assumed: it fences
 the source with a statement-level `BEFORE` trigger (`pgsprite_write_fence`, SQLSTATE `55006`)
-before its first save and keeps it through every stop until the swap carries it to the
-retained table, so the copy and one quiesced pass are the whole proof (fences-the-source,
-leaves-the-fence-on-a-stopped-run, unfences-the-old-table-it-keeps,
-stops-on-a-shadow-that-differs tests). On a decoding target it runs `CheckLive` under
-`DivergenceAbort` once the catch-up has passed a marker written after the copy, with the
-capture's own `Catchup` as the pass's confirmed source, and hands the swap a drain that
-reopens the stream from the position the capture confirmed and applies it under the swap's
-lock (carries-a-live-table-from-no-row-to-done test).
+enabled `ALWAYS`, so a session in `session_replication_role = replica` is refused too, before
+its first save, and keeps it through every stop until the swap carries it to the retained
+table, so the copy and one quiesced pass are the whole proof (fences-the-source,
+refuses-a-replica-role-session, leaves-the-fence-on-a-stopped-run,
+unfences-the-old-table-it-keeps, stops-on-a-shadow-that-differs tests). A resumed run installs
+over the fence, which enables it `ALWAYS` again (resume-upgrades-an-origin-fence test).
+
+The quiesced swap confirms the fence once more, under its `ACCESS EXCLUSIVE` lock: it refuses
+the proof as `cutover-unfenced` unless the engine's trigger is still on the source, of the
+shape the install gave it — no `WHEN` condition, no `UPDATE OF` column list, no arguments, not
+deferrable — and enabled `ALWAYS` (refuses-when-the-fence-was-disabled,
+refuses-when-the-fence-was-dropped, refuses-when-the-fence-is-origin-only, and
+refuses-when-the-fence-was-replaced-by-a-conditional-one tests). The check has a stated
+limit: a role that owns the table can disable the fence, write, and restore it between the
+pass and the swap, and that write is not seen. A live run installs no fence, so its swap
+checks none.
+
+On a decoding target the `Runner` runs `CheckLive` under `DivergenceAbort` once the catch-up
+has passed a marker written after the copy, with the capture's own `Catchup` as the pass's
+confirmed source, and hands the swap a drain that reopens the stream from the position the
+capture confirmed and applies it under the swap's lock (carries-a-live-table-from-no-row-to-done
+test).
 *Source:* [design-principles](design-principles.md#correctness-and-safety),
 risks-and-mitigations; Spirit's "never skip it".
 

docs/refusal-classes.md

@@ -196,6 +196,7 @@ causes above: the verdict's `cause` is the shadow cause unchanged and its `class
 | `cutover-unverified` | The cutover gate was handed an empty built-shadow or verified-shadow proof, two proofs naming different relations or different relation OIDs (a shadow rebuilt under the same name after the checksum ran), or a verified watermark that does not cover the whole key space ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Only the copier and verifier mint those proofs for the same table; a caller that reaches the gate without them skipped the checksum, which no retry corrects. |
 | `cutover-undrained` | The cutover was handed a verified-shadow proof a live pass minted — the shadow equal to the source as of a position the stream had reached, with every change since still in the stream — and no drain to apply those changes under the swap lock ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `invariant-violation` | Pass a `DrainFunc` that applies the stream through the final position under the swap lock, or stop the stream and re-verify with a quiesced `Check`; the proof is sound, the hand-off to the swap is not. |
 | `cutover-drain-deferred` | The shadow refused a batch the swap's drain had to apply under the swap lock: a unique-key move among the changes committed since the live verification pass that cannot be ordered into the shadow, with no copy left to change the outcome ([CO-6](invariants.md#co-6--unique-secondary-key-moves-must-converge-postgresql-specific-gap)) | `environmental` | The swap transaction is rolled back and the source stays live; the row is left in `cutover` with the position the drain reopened from, so a resume drains the same changes again once the rows the move touched have settled. A move the shadow cannot ever order is re-planned. |
+| `cutover-unfenced` | The quiesced `Runner`'s swap, under its `ACCESS EXCLUSIVE` lock, did not find the engine's write fence on the source enabled `ALWAYS`: the trigger was disabled, dropped, or replaced by one of another shape — a `WHEN` condition, an `UPDATE OF` column list, arguments, or deferral — since the run installed it ([CO-1](invariants.md#co-1--the-checksum-gate-is-non-skippable)) | `environmental` | Something other than the engine changed the source's triggers under the run, so a write may have reached it after the verification pass. The swap rolled back and nothing is swapped; the next `Run` installs the fence again, verifies again, and swaps, and a write that did land makes that pass stop on a divergence. |
 | `cutover-relation-replaced` | The source or the shadow under its name is no longer the relation the build proved: its OID moved; or, after the swap, the derived `_old` name is borne by a relation other than the source the swap retained, or by nothing while the retained source still exists under another name ([ST-6](invariants.md#st-6--preflight-before-the-first-write)) | `environmental` | The proof describes a table that no longer exists; before the swap, drop the shadow and start the change over from preflight; after it, an operator reads the catalog to learn what took the `_old` name, or where the retained source went, before anything is dropped. |
 | `cutover-schema-drift` | The source's or the shadow's introspected model no longer matches the fingerprint the build recorded: a column, constraint, index, storage parameter, or column-owned sequence declaration changed since ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | Someone changed a table under the run; the shadow no longer embodies the gated statement against this source, so it is dropped and the change re-planned. |
 | `cutover-fidelity-drift` | The source's or the shadow's owner, grants, policies, comment, replica identity, per-column or extended statistics targets, unvalidated checks, or identity columns and their sequences no longer match the snapshot the build recorded, or the source's grants no longer match the shadow's ([ST-5](invariants.md#st-5--the-swap-is-gated-on-a-fidelity-checklist-not-just-the-checksum)) | `environmental` | The metadata the swap would carry forward moved after it was copied; re-synchronising it under the lock is a planned capability, and until then the shadow is rebuilt. |

docs/schemabot-integration.md

@@ -327,7 +327,8 @@ the slot under the row (decoding target):
 | `Run` outcome | What it means | Orchestrator action |
 | --- | --- | --- |
 | `*decode.SlotExistsError` (decoding target, first start) | A logical slot of the engine's derived name already exists and no row names it: a run that ended before its row landed, or a slot made by hand. The run made nothing and dropped nothing — the slot, its publication, and the absence of a row are as the run found them | Run `Store.ReapOrphanSlots`, which drops the slot once nothing holds it, and then `Run` again; a slot that stays held is an operator's to inspect |
-| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a superuser, or `session_replication_role = replica`) to find |
+| `*checksum.DivergenceError` | The verification pass found the shadow differing from the source — under a fence nothing should have written either, and under a stream a difference the catch-up did not settle is a confirmed divergence — so the run stops rather than repair (CO-3 `DivergenceAbort`) and the row stays in `verifying` | Do not retry blindly: a retry runs the same pass over the same shadow and stops the same way. Abandon the run — `DropShadow`, `DropWriteFence` or `decode.DropSlot`, `Store.Delete` under the table lock — and run again; a difference that recurs under a fence is a writer bypassing it (a role that owns the table and disables it) to find. A writer in `session_replication_role = replica` — a logical-replication subscription's apply worker, a bulk restore with triggers disabled — does not bypass the fence, which is enabled `ALWAYS`: it is refused by it, and a subscription that errors and retries for the run's length is that refusal, not a divergence |
+| `cutover-unfenced` (quiesced target, from `Cutover`, through `Run`) | The swap, under its lock, found the engine's write fence on the source disabled, dropped, replaced by a trigger of another shape, or not enabled `ALWAYS`; nothing is swapped and the row is in `cutover` | Retry `Run`: it installs the fence again, verifies again, and swaps; a write that landed while the fence was down makes the verification pass stop on a `*checksum.DivergenceError`. Find what changed the source's triggers |
 | `*checksum.UnresolvedError` (decoding target) | The live pass could not settle a chunk the source kept writing within its reread bound; nothing was found wrong and nothing was minted, and the row stays in `verifying`. The bound is `LiveOptions.Attempts` fresh rereads of one chunk (default `checksum.DefaultSettleAttempts`), each waited on for the catch-up up to `LiveOptions.SettleWait`: a chunk written again between every two of its rereads never settles, so a table whose writes land on one chunk faster than the pass can read it twice is the case, whatever the table's total write rate | Retry `Run`: the resumed row catches up again and runs a fresh pass; a chunk that never settles is a hot spot to wait out, write less often, or give more `Attempts` |
 | `cutover-drain-deferred` (decoding target) | The swap's drain met a unique-key move it could not order into the shadow; the swap rolled back and the row is in `cutover` with the position the drain reopened from | Retry `Run`: `InspectSwapped` answers `ErrNotSwapped`, the row goes back to `catching_up`, and the same changes are drained again once the rows the move touched have settled |
 | `*decode.SlotLagExceededError` (decoding target) | The slot retained more WAL than `CatchupOptions.SlotLagCeiling` while the capture ran; the run stopped with the slot in place under the row | Retry `Run` once the source's write rate allows: the stream reopens at the row's position and drains the backlog. Abandoning the run (`DropShadow`, `decode.DropSlot`, `Store.Delete`) releases the WAL |

pkg/schemachange/cutover.go

@@ -82,6 +82,18 @@ type CutoverOptions struct {
 	// wall clock. Tests inject one to drive the retry loop without real
 	// time passing.
 	Sleep func(ctx context.Context, d time.Duration) error
+
+	// requireWriteFence makes the swap of a quiesced proof refuse unless the
+	// source still carries the engine's write fence. Only the Runner, which
+	// installs the fence, sets it; a caller of Cutover that quiesces the
+	// table by its own means has no fence to find.
+	requireWriteFence bool
+}
+
+// withWriteFence is the options with the swap held to the write fence.
+func (o CutoverOptions) withWriteFence() CutoverOptions {
+	o.requireWriteFence = true
+	return o
 }
 
 // validate refuses bounds the retry loop could not honour.
@@ -206,7 +218,7 @@ func cutoverWithRetry(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.Tabl
 	attempts := opts.lockAttempts()
 	var last error
 	for attempt := 1; attempt <= attempts; attempt++ {
-		swapped, backend, err := swapOnce(ctx, pool, lock, ready, drain, opts.Options)
+		swapped, backend, err := swapOnce(ctx, pool, lock, ready, drain, opts.requireWriteFence, opts.Options)
 		if err == nil {
 			swapped.attempts = attempt
 			return swapped, nil

pkg/schemachange/cutover_swap.go

@@ -11,13 +11,14 @@ import (
 	"github.com/block/pg-sprite/pkg/preflight"
 )
 
-// swapOnce is one attempt at the swap transaction: lock, drain, re-check
-// the shape, re-gate, rename, hand off, recheck, commit. Any error leaves
-// the transaction rolled back on the server unless the client stopped
-// hearing from it, which the caller resolves by following the backend
-// returned here to its exit and then inspecting the catalog. The backend
-// is zero when the attempt ended before its transaction ran a statement.
-func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, opts Options) (SwappedTable, swapBackend, error) {
+// swapOnce is one attempt at the swap transaction: lock, confirm the write
+// fence when the caller requires it, drain, re-check the shape, re-gate,
+// rename, hand off, recheck, commit. Any error leaves the transaction
+// rolled back on the server unless the client stopped hearing from it,
+// which the caller resolves by following the backend returned here to its
+// exit and then inspecting the catalog. The backend is zero when the
+// attempt ended before its transaction ran a statement.
+func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, requireFence bool, opts Options) (SwappedTable, swapBackend, error) {
 	built := ready.built
 	tx, err := pool.Begin(ctx)
 	if err != nil {
@@ -33,7 +34,7 @@ func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSes
 	if err != nil {
 		return SwappedTable{}, swapBackend{}, err
 	}
-	swapped, err := swapInTx(ctx, tx, lock, ready, drain, opts)
+	swapped, err := swapInTx(ctx, tx, lock, ready, drain, requireFence, opts)
 	if err != nil {
 		return SwappedTable{}, backend, err
 	}
@@ -45,7 +46,7 @@ func swapOnce(ctx context.Context, pool *pgxpool.Pool, lock *dbconn.TableLockSes
 
 // swapInTx runs every statement of the swap inside the open transaction,
 // up to but not including COMMIT.
-func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, opts Options) (SwappedTable, error) {
+func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, ready CutoverReady, drain DrainFunc, requireFence bool, opts Options) (SwappedTable, error) {
 	built := ready.built
 	if err := setGateSession(ctx, tx, built, opts); err != nil {
 		return SwappedTable{}, err
@@ -56,6 +57,14 @@ func swapInTx(ctx context.Context, tx pgx.Tx, lock *dbconn.TableLockSession, rea
 	if err := lockForSwap(ctx, tx, built); err != nil {
 		return SwappedTable{}, err
 	}
+	// A quiesced proof is only as good as the fence that kept the source
+	// quiet; a live proof is finished by the drain instead.
+	// INV: CO-1
+	if requireFence && !ready.verified.Live() {
+		if err := confirmWriteFence(ctx, tx, built); err != nil {
+			return SwappedTable{}, err
+		}
+	}
 	if drain != nil {
 		if err := drain(ctx, tx); err != nil {
 			return SwappedTable{}, fmt.Errorf("drain captured changes into %s.%s: %w", built.Schema(), built.ShadowTable(), err)