Skip to content

Reserve response time for remote placement - #37

Open
jeregrine wants to merge 3 commits into
mainfrom
fix/placement-deadline-headroom
Open

jeregrine wants to merge 3 commits into
mainfrom
fix/placement-deadline-headroom

Conversation

@jeregrine

@jeregrine jeregrine commented Sep 22, 2026 •

Copy link
Copy Markdown
Member

Problem

Remote child startup and its enclosing placement RPC used the same timeout budget. A slow but reachable node could exhaust its startup wait just as the RPC expired, preventing the ordinary timeout response from reaching the caller. The resulting transport timeout put the entire node into cooldown, affecting placement of unrelated keys.

The remote readiness wait also used a fixed allowance even when the caller had less time remaining.

Fix

Reserve up to one second of each placement RPC budget for transit and the response, using half the budget rounded up for short calls. Do not dispatch a remote start when no startup budget remains. Bound remote readiness waiting by the remaining child-start deadline.

This lets ordinary bootstrap timeouts return without unnecessarily excluding the reachable node. Genuine transport failures still trigger cooldown, and an already-supervised bootstrap continues after its caller stops waiting.

Supporting information

Why one second: this follows the response-headroom policy already used by the consuming application's repository RPC path, rather than introducing a separate 250 ms tuning convention. The outer RPC timer starts before remote execution begins, so the allowance needs to cover outbound transit, scheduling, and the return trip—not just the reply. One second is a conservative policy choice, not a production-derived latency percentile or a guarantee against transport timeouts. Capping it at half the budget keeps short calls usable: a 500 ms RPC gives startup 250 ms, while the default 3-second and 8-second RPC budgets give startup 2 and 7 seconds respectively. The overall caller deadline is not increased.

Sticky placement, capacity policy, ownership rules, public option names, and return shapes are unchanged. No persisted-state migration is needed. The backend configuration and placement suites share a single node-local in-memory storage fixture.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant