Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Remote child startup and its enclosing placement RPC used the same timeout budget. A slow but reachable node could exhaust its startup wait just as the RPC expired, preventing the ordinary timeout response from reaching the caller. The resulting transport timeout put the entire node into cooldown, affecting placement of unrelated keys.
The remote readiness wait also used a fixed allowance even when the caller had less time remaining.
Fix
Reserve up to one second of each placement RPC budget for transit and the response, using half the budget rounded up for short calls. Do not dispatch a remote start when no startup budget remains. Bound remote readiness waiting by the remaining child-start deadline.
This lets ordinary bootstrap timeouts return without unnecessarily excluding the reachable node. Genuine transport failures still trigger cooldown, and an already-supervised bootstrap continues after its caller stops waiting.
Supporting information
Why one second: this follows the response-headroom policy already used by the consuming application's repository RPC path, rather than introducing a separate 250 ms tuning convention. The outer RPC timer starts before remote execution begins, so the allowance needs to cover outbound transit, scheduling, and the return trip—not just the reply. One second is a conservative policy choice, not a production-derived latency percentile or a guarantee against transport timeouts. Capping it at half the budget keeps short calls usable: a 500 ms RPC gives startup 250 ms, while the default 3-second and 8-second RPC budgets give startup 2 and 7 seconds respectively. The overall caller deadline is not increased.
Sticky placement, capacity policy, ownership rules, public option names, and return shapes are unchanged. No persisted-state migration is needed. The backend configuration and placement suites share a single node-local in-memory storage fixture.