Skip to content

fix(pegboard-gateway): send the client close without waiting on the envoy - #5874

Open
eersnington wants to merge 1 commit into
fix/client-retry-actor-ready-timeoutfrom
fix/gateway-close-without-envoy-ack
Open

eersnington wants to merge 1 commit into
fix/client-retry-actor-ready-timeoutfrom
fix/gateway-close-without-envoy-ack

Conversation

@eersnington

@eersnington eersnington commented Oct 13, 2026 •

Copy link
Copy Markdown
Member

When an actor closes a client WebSocket and its runner shuts down right after, as in a drain, the client waited up to 30s for the close frame. On a server close, the gateway first sent the close back to the envoy and waited for it. With the envoy already gone, send_message retried until it timed out, and only then sent the close frame to the client.

  • On a server close, pegboard-gateway2 and pegboard-gateway3 send the close back to the envoy in a background task, so the client gets its close frame right away.
  • Client closes and aborts still wait for the envoy, as before.
// engine/packages/pegboard-gateway2/src/lib.rs
if matches!(lifecycle_res, Ok(LifecycleResult::ServerClose(_))) {
	tokio::spawn(close_envoy.in_current_span());
} else {
	close_envoy.await;
}

This is part 2 of 6 in a stack:

@claude

claude Bot commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Review

The change is small and the approach is sound. On a server close, the client close frame no longer waits for an envoy that may already be gone. Client closes and aborts keep the old ordering. I haven't built or tested this branch.

  1. Untracked background task. tokio::spawn(close_envoy...) is detached. If the gateway shuts down mid-spawn, in_flight_req.stop(...) may never run, which could leak in-flight request state or skew stop metrics. The send_message(.., true) retry loop is also no longer bounded by anything the caller can see. If it is bounded internally by a timeout, a short comment would help. If not, wrap it in tokio::time::timeout so a dead envoy can't pin a task indefinitely.
  2. Ordering change for stop(). For server closes, stop(Success) now races with the client close frame and anything after the handler returns. Please confirm nothing downstream depends on the request being stopped first.
  3. Duplication. The blocks in pegboard-gateway2 and pegboard-gateway3 are identical. That matches the existing structure, but a shared helper would stop them drifting apart.
  4. No test coverage. Nothing exercises the case where the envoy is gone and the client still gets its close frame promptly. A test under engine/packages/engine/tests/envoy/ that drops the envoy before the actor closes the WebSocket would pin this down, ideally with a bounded-time assertion.
  5. Style. Per CLAUDE.md, warnings should hint at the likely cause, for example "envoy may have already disconnected".

Overall this looks good, and I would address 1 and 4 before merging.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant