Skip to content

Timeouts for long requests: routes, deadlines, cancellation and keepalive of streams #266

Closed

Description

@dex4er

The timeouts of Pyjoyment follow Mojolicious: inactivity_timeout of the servers (30 seconds) and of the user agent (40 seconds), keep_alive_timeout, connect_timeout and request_timeout of the user agent, graceful_timeout and heartbeat_timeout of Pyjo.Server.Prefork, and the helper c.inactivity_timeout(...) for one connection. They fit requests that answer in seconds. A request that waits for an LLM, for a tool of an agent or for a long job can wait for minutes before its first byte, and the infrastructure in front of the application has timeouts of its own.

Scope

  • Timeouts of a route, as an option of a route or of a group of routes (r.post('/chat', inactivity_timeout=600)), in the place of a call of the helper in every action, set before the action runs.
  • A deadline of a request, the most time from the request to the end of the response, which Mojolicious does not have: when it passes, the action is cancelled and the request gets a response of its own (504) if nothing was sent yet.
  • Cancellation: check what happens to an action, a coroutine of async def or a chain of promises, when its connection closes by an inactivity timeout or by the client, and cancel it, so that the request to an LLM that nobody reads anymore stops too; the same for the requests of the user agent of the action.
  • Keepalive of streams: an option that writes a comment of SSE (or a ping of WebSocket) at an interval while the stream waits, so that proxies and load balancers with an idle timeout keep the connection, and the inactivity timeout of the server does not close it.
  • The user agent for long requests: the defaults that fit the APIs of LLMs, which answer slowly or stream for minutes, and a documented way to set them for one request.
  • Documentation: a section of the cookbook with the timeouts of the infrastructure that the application has to fit in, as nginx and Caddy, the load balancers of AWS and Google Cloud, Cloud Run, Kubernetes, API Gateway and AWS Lambda, and the settings of Pyjoyment for each; the FAQ "Inactivity timeout" links it.

Steps

Validation

  • Tests of each timeout with short values, on Pyjo.Reactor.Asyncio and Pyjo.Reactor.Poll
  • A test that a cancelled action stops its request of the user agent
  • The existing tests of the timeouts of Mojolicious still pass

Related: #265 (LLM friendliness), #264 (Lambda streaming).

Activity

  1. added this to the 0.4.0 milestone on Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions