Repository navigation
MOVING handoff: hold new work in the backlog while the in-flight commands drain, so the recycle happens within a round trip instead of at the window end - #3347
Merged
Conversation
…ands drain, so the recycle happens within a round trip instead of at the window end The drain waited for "no caller work", which under steady traffic never comes: there is always a command in flight, so the recycle waited out the whole announced window and failed whatever was in flight at that instant. Measured live (endpoint rebind, single proxy, ~900 ops/s): recycle at +20.0s against a MOVING at +6.0s, 16 failed commands, missed-deadline warning. While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection about to be replaced; the backlog processor does not flush while the hold stands; the drain waits only for what is already written, capped at half the relaxed timeout (bounded by half the remaining window, floor one second); the connection is then recycled as before, the hold cleared, and the held work flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect. A FailFast policy does not abandon held work for the handoff's own disconnect. Same scenario on this branch: recycle at +6.2s against a MOVING at +5.9s, 0 failed of 97k. Deliberately a flag and nothing more: no per-message timeout cover (with relaxation off and a timeout shorter than the hold, held commands can time out in the backlog, as on any reconnect), and no retire under the write lock (a writer that reaches the lock after the recycle gets a write fault, as before). Also: the drain budget is recomputed after the DNS poll, a second MOVING during a handoff updates the successor pin rather than being dropped, and the pin lasts the window so the second bridge dials the same successor. Tests: steady load (with and without relaxation), slow command inside the cap, blocking command past the cap; plus a fault-injector tier measurement that records the live timeline.
…ther than counting at the instant the second node relaxed; the callback is dispatched after the relaxation, so the count could be read as zero (flaked on main at ~1 in 6)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A pared-down alternative to #3335: the same gap, the smallest mechanism that closes it, and a stated list of what it does not guarantee.
The problem
The
MOVINGdrain waited for "no caller work" before replacing the connections. Under steady traffic that never happens: there is always a command in flight, so the drain waited out the entire announced window and recycled at the deadline, failing whatever was in flight at that instant and leaving no time to establish the replacement.The fix
While a handoff drains, the bridge holds a caller's work in the backlog instead of writing it to the connection the server is about to close, and the backlog processor does not flush while the hold stands. The drain then waits only for what is already written, so under load it finishes within a round trip; it is capped at half the relaxed timeout (bounded by half of what is left of the window, floor one second), after which the connection is recycled regardless. The recycle is the same dispose as before; the hold is cleared after it, and the held work is flushed to the replacement by the ordinary reconnect path. Timeouts are relaxed for the hold and the reconnect, and a
FailFastpolicy does not abandon the held work for the handoff's own disconnect.Also: the drain budget is recomputed after the DNS poll (which can eat several seconds of the window); a second
MOVINGduring a handoff updates the successor pin instead of being dropped; and the pin lasts the window rather than being cleared on the first establishment, so the second bridge dials the same successor.Measured
Live, Redis Enterprise through the fault injector, endpoint rebind,
singleproxy policy,maintMovingEndpointType=Auto, sixteen workers issuing PING back-to-back (~900 ops/s). The harness isMovingUnderLoadScenarioTestsin this PR; themainrow is the same harness on amainworktree.mainWhat is deliberately not here, and what that gives up
FailFast. If the replacement attempt itself fails,FailFastabandons the held work, which is what it means on every other reconnect.MOVINGs are last-writer-wins on the pin; no second drain.Tests
MovingUnderSteadyLoadRecyclesPromptlyAndFailsNothing(fails onmain: recycle at the window end), and with relaxation disabled.MovingWaitsForSlowCommandsThatFitInsideTheDrainCap,MovingWithBlockingCommandsInFlightRecyclesAtTheDrainCap.MovingUnderLoadScenarioTests.HandoffUnderSteadyLoad, fault-injector tier: records the timeline above; asserts only that aMOVINGarrived and a recycle happened.MaintenanceNotificationTests37/37; the thirteenMoving*tests repeated seven times, six fully green and one run with a single failure whose output was not captured, so a flake check before merge is warranted.Build.csprojRelease build withCI=true: clean.