[fylorn]

What

A new per-upstream setting, providers[].min_speed:

providers:
  - name: local-vllm
    min_speed: { tokens_per_sec: 15, window_secs: 20 }

From the first token on, a streamed answer (SSE or Bedrock eventstream) is judged once a second. When the average speed over the last window_secs seconds is below tokens_per_sec, the gateway:

• ends the answer with a 503 error frame in the client's format (Anthropic overloaded_error, Responses response.failed / server_error, Gemini UNAVAILABLE, Chat {"error": …}) and drops the upstream connection;
• fails the request with gw.upstream.too_slow (upstream, rate, min, secs), source upstream;
• records one failure for the upstream (counts toward failures_to_pause) and stops the conversation from sticking to it this turn;
• keeps usage: whatever the upstream had not reported yet is filled with the gateway's estimates, and the request row keeps the measured speed.

Speed uses the same estimate as the live speed. Silences of 5 seconds or more are taken out of the window (the idle timeout handles those), and a second with less than half a window left after that is not judged. Non-streamed answers and WebSocket turns are not judged. Unset means off; behaviour is unchanged.

Contract

• ProviderView.min_speed?: MinSpeedView { tokens_per_sec, window_secs }, ProviderInput.min_speed?: MinSpeedInput { tokens_per_sec, window_secs }.
• Event::RequestFailed gains tokens_per_sec?: number, set only for this failure.
• New codes: gw.upstream.too_slow, config.min_speed_range (upstream, field, value, min, max). One-click repair drops min_speed when the rate is out of range and resets window_secs to 20 when the window is.
• CONTROL_API_VERSION 57 → 58. Request store schema unchanged.
• Config packs, sharing and sync carry min_speed with the upstream.
• StreamConverter::fail_with(status, …) lets converted streams end with a status-specific error; other cut-offs keep using fail.

Tests

• tw-gateway/tests/min_speed.rs (fake upstreams, the "second" scaled to 50 ms): stays fast, short slowdown, long pause mid-stream, long wait before the first token, upstream without a minimum, non-streamed answer, slowdown for a window (frame, failure, speed, estimated usage, upstream dropped, health), Chat/Responses/Gemini clients, Bedrock eventstream.
• Unit tests for the window maths, validation, repair, control-plane save/view, pack units, recorder, dialect frames and the 503 mapping.
• Config manual regenerated.

🤖 Generated with Claude Code