Repository navigation
Closed
Labels
Milestone
Description
The timeouts of Pyjoyment follow Mojolicious: inactivity_timeout of the servers (30 seconds) and of the user agent (40 seconds), keep_alive_timeout, connect_timeout and request_timeout of the user agent, graceful_timeout and heartbeat_timeout of Pyjo.Server.Prefork, and the helper c.inactivity_timeout(...) for one connection. They fit requests that answer in seconds. A request that waits for an LLM, for a tool of an agent or for a long job can wait for minutes before its first byte, and the infrastructure in front of the application has timeouts of its own.
Scope
- Timeouts of a route, as an option of a route or of a group of routes (
r.post('/chat', inactivity_timeout=600)), in the place of a call of the helper in every action, set before the action runs. - A deadline of a request, the most time from the request to the end of the response, which Mojolicious does not have: when it passes, the action is cancelled and the request gets a response of its own (504) if nothing was sent yet.
- Cancellation: check what happens to an action, a coroutine of
async defor a chain of promises, when its connection closes by an inactivity timeout or by the client, and cancel it, so that the request to an LLM that nobody reads anymore stops too; the same for the requests of the user agent of the action. - Keepalive of streams: an option that writes a comment of SSE (or a ping of WebSocket) at an interval while the stream waits, so that proxies and load balancers with an idle timeout keep the connection, and the inactivity timeout of the server does not close it.
- The user agent for long requests: the defaults that fit the APIs of LLMs, which answer slowly or stream for minutes, and a documented way to set them for one request.
- Documentation: a section of the cookbook with the timeouts of the infrastructure that the application has to fit in, as nginx and Caddy, the load balancers of AWS and Google Cloud, Cloud Run, Kubernetes, API Gateway and AWS Lambda, and the settings of Pyjoyment for each; the FAQ "Inactivity timeout" links it.
Steps
- Check the cancellation of an action whose connection closes, on both reactors: the action ran to its end with its request (Pyjoyment, Pyjo.Promise: async actions and their requests are cancelled when the connection closes #611)
- Timeouts of a route
- The deadline of a request
- Cancellation of the action and of its requests of the user agent
- Keepalive of SSE and WebSocket streams
- The user agent for long requests
- The section of the cookbook
Validation
- Tests of each timeout with short values, on
Pyjo.Reactor.AsyncioandPyjo.Reactor.Poll - A test that a cancelled action stops its request of the user agent
- The existing tests of the timeouts of Mojolicious still pass
Activity