Skip to content

MOVING handoff: hold new work in the backlog while the in-flight commands drain, so the recycle happens within a round trip instead of at the window end - #3347

Merged
mgravell merged 2 commits into
mainfrom
marc/moving-handoff-minimal
Nov 12, 2026
Merged

mgravell merged 2 commits into
mainfrom
marc/moving-handoff-minimal

Conversation

@mgravell

Copy link
Copy Markdown
Member

A pared-down alternative to #3335: the same gap, the smallest mechanism that closes it, and a stated list of what it does not guarantee.

The problem

The MOVING drain waited for "no caller work" before replacing the connections. Under steady traffic that never happens: there is always a command in flight, so the drain waited out the entire announced window and recycled at the deadline, failing whatever was in flight at that instant and leaving no time to establish the replacement.

The fix

While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection the server is about to close, and the backlog processor does not flush while the hold stands. The drain then waits only for what is already written, so under load it finishes within a round trip; it is capped at half the relaxed timeout (bounded by half of what is left of the window, floor one second), after which the connection is recycled regardless. The recycle is the same dispose as before; the hold is cleared after it, and the held work is flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect, and a FailFast policy does not abandon the held work for the handoff's own disconnect.

Also: the drain budget is recomputed after the DNS poll (which can eat several seconds of the window); a second MOVING during a handoff updates the successor pin instead of being dropped; and the pin lasts the window rather than being cleared on the first establishment, so the second bridge dials the same successor.

Measured

Live, Redis Enterprise through the fault injector, endpoint rebind, single proxy policy, maintMovingEndpointType=Auto, sixteen workers issuing PING back-to-back (~900 ops/s). The harness is MovingUnderLoadScenarioTests in this PR; the main row is the same harness on a main worktree.

event main this PR
MOVING received +6.0s +5.9s
connections replaced +20.0s (window end) +6.2s
ConnectionRestored +20.1s +6.2s
completed commands 90376 97070
failed commands 16 0
"did not establish a replacement" warning yes no

What is deliberately not here, and what that gives up

  • No per-message timeout cover. Held work relies on the relaxed window. An application that has turned relaxation off and runs a timeout shorter than the hold can see held commands time out in the backlog, as it would in the backlog of any other reconnect.
  • No retire under the write lock, no requeue of late writers. A writer that passed the hold gate before the hold began and reached the lock after the recycle gets a write fault, which is what it got before. This keeps the "fail, don't reorder" behaviour for that narrow race.
  • No phase machine, heartbeat re-judge or lockless fallback. With the hold cleared after the dispose, "is this the replacement" does not arise: whatever establishes next gets the backlog. A reconnect that lands back on the retiring node is closed at window end and reconnects again, as before.
  • No handoff-own-failure shield for FailFast. If the replacement attempt itself fails, FailFast abandons the held work, which is what it means on every other reconnect.
  • Back-to-back MOVINGs are last-writer-wins on the pin; no second drain.

Tests

  • MovingUnderSteadyLoadRecyclesPromptlyAndFailsNothing (fails on main: recycle at the window end), and with relaxation disabled.
  • MovingWaitsForSlowCommandsThatFitInsideTheDrainCap, MovingWithBlockingCommandsInFlightRecyclesAtTheDrainCap.
  • MovingUnderLoadScenarioTests.HandoffUnderSteadyLoad, fault-injector tier: records the timeline above; asserts only that a MOVING arrived and a recycle happened.
  • MaintenanceNotificationTests 37/37; the thirteen Moving* tests repeated seven times, six fully green and one run with a single failure whose output was not captured, so a flake check before merge is warranted.
  • Full Build.csproj Release build with CI=true: clean.

…ands drain, so the recycle happens within a round trip instead of at the window end

The drain waited for "no caller work", which under steady traffic never comes: there is always a command in flight, so the recycle waited out the whole announced window and failed whatever was in flight at that instant. Measured live (endpoint rebind, single proxy, ~900 ops/s): recycle at +20.0s against a MOVING at +6.0s, 16 failed commands, missed-deadline warning.

While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection about to be replaced; the backlog processor does not flush while the hold stands; the drain waits only for what is already written, capped at half the relaxed timeout (bounded by half the remaining window, floor one second); the connection is then recycled as before, the hold cleared, and the held work flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect. A FailFast policy does not abandon held work for the handoff's own disconnect. Same scenario on this branch: recycle at +6.2s against a MOVING at +5.9s, 0 failed of 97k.

Deliberately a flag and nothing more: no per-message timeout cover (with relaxation off and a timeout shorter than the hold, held commands can time out in the backlog, as on any reconnect), and no retire under the write lock (a writer that reaches the lock after the recycle gets a write fault, as before). Also: the drain budget is recomputed after the DNS poll, a second MOVING during a handoff updates the successor pin rather than being dropped, and the pin lasts the window so the second bridge dials the same successor.

Tests: steady load (with and without relaxation), slow command inside the cap, blocking command past the cap; plus a fault-injector tier measurement that records the live timeline.
…ther than counting at the instant the second node relaxed; the callback is dispatched after the relaxation, so the count could be read as zero (flaked on main at ~1 in 6)
@mgravell
mgravell merged commit 510f791 into main Nov 12, 2026
8 of 11 checks passed
@mgravell
mgravell deleted the marc/moving-handoff-minimal branch November 12, 2026 13:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant