Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
103 changes: 103 additions & 0 deletions docs/guides/cookbook.rst
Original file line number Diff line number Diff line change
Expand Up @@ -935,6 +935,72 @@ such as seen in `Istio
examples, visit the `Envoy documentation
<https://www.envoyproxy.io/docs/envoy/latest/start/start>`_.

.. _cookbook-long-requests:

Long requests
~~~~~~~~~~~~~

A request that waits for a model of language, for a tool of an agent or for
a long job can be silent for minutes before its first byte, and can stream
for minutes after it. Each layer in front of the application has timeouts of
its own, between two reads or for the whole response, and the shortest one
ends the request, with an error that the application never sees.

The application gives such a route timeouts of its own with the values
``inactivity_timeout``, ``request_timeout`` and ``heartbeat`` of its stash,
as :doc:`routing` describes: a longer inactivity timeout of the connection,
the most time to the end of the response, after which the request gets a
``504``, and a comment of Server-Sent Events or a ping of a WebSocket while
the stream waits, more often than the shortest idle timeout in front of it.
An action that is a coroutine function is cancelled when its connection
closes before its response ends, as when a proxy in front of it gives up,
with the requests of :mod:`Pyjo.UserAgent` that it waits for. A
``request_timeout`` below the timeout of the whole response of a layer in
front of the application ends the request with the ``504`` of the
application, which it logs, rather than with an error of that layer.

* nginx waits ``proxy_read_timeout`` between two reads from the
application, 60 seconds by default, and buffers the response, which holds
back a stream of events until ``proxy_buffering off`` or the header
``X-Accel-Buffering: no`` of the response; ingress-nginx of Kubernetes has
them as the annotations ``nginx.ingress.kubernetes.io/proxy-read-timeout``
and ``nginx.ingress.kubernetes.io/proxy-buffering``.

.. code-block:: nginx

location / {
proxy_pass http://myapp;
proxy_read_timeout 600s;
proxy_buffering off;
}

* Caddy passes a stream of ``text/event-stream`` on at once, and waits for
the headers of the response as long as ``response_header_timeout`` of its
``transport http``.
* The Application Load Balancer of AWS closes a connection that is idle for
its idle timeout, 60 seconds by default, which a heartbeat keeps from it,
and CloudFront waits its origin response timeout for the first byte and
between two reads, 30 seconds by default.
* The Application Load Balancer of Google Cloud waits the timeout of its
backend service, 30 seconds by default, for the whole response of HTTP,
also of a stream of events, which a heartbeat does not extend, and for a
WebSocket that is idle.
* Cloud Run ends a request after the request timeout of the service, 5
minutes by default, which ``gcloud run deploy --timeout`` changes, also
for a stream of events and a WebSocket.
* Kubernetes has no timeouts of its own for the requests of a Service; the
Ingress controller in front of it, or a route of the Gateway API, with
``timeouts.request`` and ``timeouts.backendRequest`` of an HTTPRoute, has
them.
* Amazon API Gateway ends a request after the timeout of its integration,
and passes a stream on only with the response transfer mode ``STREAM`` of
a REST API, with the limits of its documentation.
* AWS Lambda ends an invocation after the timeout of the function, 3
seconds by default and 15 minutes at most, and streams a response with
``PYJO_LAMBDA_INVOKE_MODE=RESPONSE_STREAM``, as :ref:`cookbook-aws-lambda`
describes, whose ``request_timeout`` below the timeout of the function
gives the client a ``504`` of the application.

FastCGI
~~~~~~~

Expand Down Expand Up @@ -1040,6 +1106,8 @@ the command :mod:`Pyjoyment.Command.wsgi` for it.
:language: python
:caption: myapp.py

.. _cookbook-aws-lambda:

AWS Lambda
~~~~~~~~~~

Expand Down Expand Up @@ -2147,6 +2215,41 @@ It is also strongly recommended to respect every sites ``robots.txt`` file as
well as terms of service, and to wait a little before reopening connections to
the same host, or the operators might be forced to block your access.

Long requests of the user agent
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A service that answers slowly, as the API of a model of language, which can
be silent for minutes before its first byte, gets a user agent of its own,
whose :attr:`~Pyjo.UserAgent.Pyjo_UserAgent.inactivity_timeout` waits that
long, and whose :attr:`~Pyjo.UserAgent.Pyjo_UserAgent.request_timeout`
bounds each of its requests. A request with a time of its own races with
:meth:`Pyjo.Promise.Pyjo_Promise.timeout`, and
:meth:`Pyjo.Promise.Pyjo_Promise.cancel` closes its connection when the time
passes first, as nothing awaits its response anymore, and does nothing when
the request has ended.

.. code-block:: python

import Pyjo.Promise
import Pyjo.UserAgent

# A user agent of its own for a slow service
llm = Pyjo.UserAgent.new(inactivity_timeout=300, request_timeout=600)


# One request with a time of its own
async def complete(prompt):
promise = llm.post_p('https://api.example.com/v1/complete', json={'prompt': prompt})
try:
tx = await Pyjo.Promise.race(promise, Pyjo.Promise.timeout(120, 'Too slow'))
finally:
promise.cancel()
return tx.result.json()

An action of :mod:`Pyjoyment` that awaits such a request is cancelled when
its client goes away, with the request, as :ref:`cookbook-long-requests`
describes.

Concurrent blocking requests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Expand Down
5 changes: 4 additions & 1 deletion docs/guides/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -271,7 +271,10 @@ with the attributes
applications you can also use the helper ``inactivity_timeout`` of
:mod:`Pyjoyment.Plugin.DefaultHelpers` to change it on demand for each
connection individually. This timeout always applies, so you might have to
tweak it for applications that take a long time to process a request.
tweak it for applications that take a long time to process a request. A
request that takes minutes, as one that waits for a model of language, needs
the timeouts of its route and of the infrastructure in front of the
application too, which :ref:`cookbook-long-requests` describes.

What does "Premature connection close" mean?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Expand Down
42 changes: 42 additions & 0 deletions t/docs/guides_cookbook.py
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,12 @@ def raw(c):
c.on('message', lambda c, msg: c.send(msg))


# A slow service, which never answers
@r.post('/v1/slow')
def slow(c):
c.render_later()


@r.any('/*rest', {'rest': ''})
def page(c):
host = c.req.headers.host
Expand Down Expand Up @@ -815,6 +821,42 @@ def drain(content, offset=None):
tx = ua.start(tx)
is_ok(tx.result.json('/body'), 'Hello World!', 'written')

with subtest('Long requests of the user agent'):
import Pyjo.Promise
import Pyjo.UserAgent

# A user agent of its own for a slow service
llm = Pyjo.UserAgent.new(inactivity_timeout=300, request_timeout=600)

# One request with a time of its own
async def complete(prompt):
promise = llm.post_p('https://api.example.com/v1/complete', json={'prompt': prompt})
try:
tx = await Pyjo.Promise.race(promise, Pyjo.Promise.timeout(120, 'Too slow'))
finally:
promise.cancel()
return tx.result.json()

with llm, remote(llm):
got = []
Pyjo.Promise.new(complete('Hello')).then(got.append).wait()
is_ok(got[0]['host'], 'api.example.com', 'host of the service')
is_ok(got[0]['body'], '{"prompt":"Hello"}', 'request')

# The same with a time that passes first, on a service that does not answer
async def slow(prompt):
promise = llm.post_p('https://api.example.com/v1/slow', json={'prompt': prompt})
try:
tx = await Pyjo.Promise.race(promise, Pyjo.Promise.timeout(0.1, 'Too slow'))
finally:
promise.cancel()
return tx.result.json()

reason = []
Pyjo.Promise.new(slow('Hello')).catch(reason.append).wait()
like_ok(str(reason[0]), r'^Too slow', 'the time passed first')
is_deeply_ok(llm._connections, {}, 'connection of the request closed')

with subtest('Non-blocking'):
# Concurrent non-blocking requests
with Pyjo.UserAgent.new() as ua, remote(ua):
Expand Down
Loading