katipo

An HTTP/HTTP2/HTTP3 client library for Erlang built around libcurl-multi and libevent.

Status

Usage

{ok, _} = application:ensure_all_started(katipo).
Pool = api_server,
{ok, _} = katipo_pool:start(Pool, 2, [{pipelining, multiplex}]).
Url = <<"https://example.com">>.
ReqHeaders = [{<<"User-Agent">>, <<"katipo">>}].
Opts = #{headers => ReqHeaders,
         body => <<"0d5cb3c25b0c5678d5297efa448e1938">>,
         connecttimeout_ms => 5000,
         proxy => <<"http://127.0.0.1:9000">>,
         ssl_verifyhost => false,
         ssl_verifypeer => false},
{ok, #{status := 200,
       headers := RespHeaders,
       cookiejar := CookieJar,
       body := RespBody}} = katipo:post(Pool, Url, Opts).

Or passing the entire request as a map

{ok, _} = application:ensure_all_started(katipo).
Pool = api_server,
{ok, _} = katipo_pool:start(Pool, 2, [{pipelining, multiplex}]).
ReqHeaders = [{<<"User-Agent">>, <<"katipo">>}].
Req = #{url => <<"https://example.com">>,
        method => post,
        headers => ReqHeaders,
        body => <<"0d5cb3c25b0c5678d5297efa448e1938">>,
        connecttimeout_ms => 5000,
        proxy => <<"http://127.0.0.1:9000">>,
        ssl_verifyhost => false,
        ssl_verifypeer => false},
{ok, #{status := 200,
       headers := RespHeaders,
       cookiejar := CookieJar,
       body := RespBody}} = katipo:req(Pool, Req).

Why

We wanted a compatible and high-performance HTTP client so took advantage of the 25+ years of development that has gone into libcurl. To allow large numbers of simultaneous connections libevent is used along with the libcurl-multi interface.

Documentation

API

-type method() :: get | post | put | head | options | patch | delete.
katipo_pool:start(Name :: atom(), size :: pos_integer(), PoolOptions :: proplist()).
katipo_pool:stop(Name :: atom()).

katipo:req(Pool :: atom(), Req :: map()).
katipo:Method(Pool :: atom(), URL :: binary()).
katipo:Method(Pool :: atom(), URL :: binary(), ReqOptions :: map()).

%% Asynchronous: return {ok, Ref} immediately. The response is delivered to the
%% calling process (or the `reply_to` option) as a {katipo_response, Ref, Response}
%% or {katipo_error, Ref, Error} message. Use await/1,2 to block for it.
katipo:async_req(Pool :: atom(), Req :: map()).
katipo:async_Method(Pool :: atom(), URL :: binary()).
katipo:async_Method(Pool :: atom(), URL :: binary(), ReqOptions :: map()).
katipo:await(Ref :: reference()).
katipo:await(Ref :: reference(), Timeout :: timeout()).
katipo:cancel(Pool :: atom(), Ref :: reference()).
katipo:update_flow(Pool :: atom(), Ref :: reference(), Credits :: pos_integer()).

Request options

Option	Type	Default	Notes
headers	[{binary(), iodata()}]	[]
cookiejar	opaque (returned in response)	[]
body	iodata()	<<>>
connecttimeout_ms	pos_integer()	30000	docs
followlocation	boolean()	false	docs
ssl_verifyhost	boolean()	true	docs
ssl_verifypeer	boolean()	true	docs
capath	binary()	undefined	docs
cacert	binary()	undefined	docs
ca_cache_timeout	integer()	86400	docs curl >= 7.87.0 (0=disable, -1=forever)
timeout_ms	pos_integer()	30000	docs
dns_cache_timeout	integer()	60	docs (0=disable, -1=forever)
maxredirs	non_neg_integer()	9	docs
proxy	binary()	undefined	docs
tcp_fastopen	boolean()	false	docs curl >= 7.49.0
pipewait	boolean()	true	docs curl >= 7.43.0
interface	binary()	undefined	docs
unix_socket_path	binary()	undefined	docs curl >= 7.40.0
doh_url	binary()	undefined	docs curl >= 7.62.0
http_version	curl_http_version_none curl_http_version_1_0 curl_http_version_1_1 curl_http_version_2_0 curl_http_version_2tls curl_http_version_2_prior_knowledge curl_http_version_3	curl_http_version_none	docs HTTP/3 requires curl >= 7.66.0
sslversion	sslversion_default sslversion_tlsv1 sslversion_tlsv1_0 sslversion_tlsv1_1 sslversion_tlsv1_2 sslversion_tlsv1_3	sslversion_default	docs
sslcert	binary()	undefined	docs
sslkey	binary()	undefined	docs
sslkey_blob	binary() (DER format)	undefined	docs curl >= 7.71.0
keypasswd	binary()	undefined	docs
http_auth	basic digest ntlm negotiate	undefined	docs
stream	boolean()	false	async only; deliver the body incrementally (see Streaming responses)
stream_window	pos_integer() | infinity	infinity	chunk credits before the transfer pauses awaiting update_flow/3
username	binary()	undefined	docs
password	binary()	undefined	docs
userpwd	binary()	undefined	docs
verbose	boolean()	false	docs
reply_to	pid()	self()	Async only (async_*): process that receives the response message. Accepted but ignored on synchronous requests.

Responses

{ok, #{status := pos_integer(),
       headers := headers(),
       cookiejar := cookiejar(),
       body := body()}}

{error, #{code := atom(), message := binary()}}

The code atom is drawn from the exported t:katipo:error_code/0 union (curl error names plus bad_opts, await_timeout, worker_died, overload, and admission_timeout).

Asynchronous requests

async_req/2 and the async_Method wrappers return {ok, Ref} immediately and deliver the result as a message instead of blocking:

{ok, Ref} = katipo:async_get(Pool, <<"https://example.com">>),
receive
    {katipo_response, Ref, #{status := Status}} -> Status;
    {katipo_error, Ref, Error} -> {error, Error}
end.

await/1,2 blocks for that message (default 30s), and cancel/2 aborts an in-flight request (best-effort). The pool argument to cancel/2 and update_flow/3 is retained for compatibility and ignored — the Ref itself routes to the worker holding the request. The message goes to the calling process by default; set the reply_to option to redirect it to another process. If the worker's port dies while a request is in flight, both sync and async callers receive {error, #{code => worker_died}}; if an async request cannot be handed to a worker at all (it died and is being restarted), async_req itself returns {error, #{code => worker_died}} and no message is delivered.

Streaming responses

Pass stream => true to any async function to receive the body incrementally instead of as one buffered binary — useful for large downloads, where buffering costs roughly twice the body size across the port boundary:

{ok, Ref} = katipo:async_get(Pool, Url, #{stream => true}),
receive {katipo_headers, Ref, #{status := Status, headers := Headers}} -> ok end,
collect(Ref) %% ->
%%   {katipo_chunk, Ref, Bin}      zero or more, in arrival order
%%   {katipo_done, Ref, #{status := Status, cookiejar := Jar}}  terminal

A {katipo_error, Ref, ErrorMap} message is terminal and can arrive at any point, including mid-body (e.g. a timeout). cancel/2 works as usual; after it takes effect no further messages arrive. Streaming is async-only — the synchronous functions reject stream => true — and await/1,2 does not apply.

By default chunks arrive as fast as the transfer produces them. To bound the consumer's exposure, pass stream_window => N: after N outstanding chunk messages the transfer pauses client-side (curl stops reading the socket, so backpressure reaches the server through TCP or the HTTP/2/3 stream window) until katipo:update_flow(Pool, Ref, Credits) grants more. A window of N bounds buffered data to roughly N × 16KB (curl's write-buffer size) plus one pipe's worth. The request timer keeps running while a transfer is paused, so a consumer that stops granting credits eventually receives operation_timedout. Use a reasonably current libcurl when combining stream_window with HTTP/2/3 multiplexing — older releases could stall other streams sharing the connection while one stream is paused.

Pool Options

Option	Type	Default	Note
pipelining	nothing http1 multiplex	nothing	HTTP pipelining CURLMOPT_PIPELINING
max_total_connections	non_neg_integer()	0 (no limit)	docs
max_concurrent_streams	non_neg_integer()	100	docs curl >= 7.67.0

Katipo adds one Erlang-side pool option of its own: {max_in_flight, N} (default infinity) caps concurrently in-flight requests per worker; a full pool rejects requests fast with {error, #{code => overload}}.

Observability

Katipo uses OpenTelemetry for tracing and metrics.

Tracing

Each HTTP request creates a span with the following attributes:

Attribute	Description
http.request.method	HTTP method (GET, POST, etc.)
url.full	Request URL (query string, fragment and userinfo are stripped for security)
server.address	Target host
http.response.status_code	Response status code (on success)

Metrics

The following metrics are recorded:

Metric	Type	Description
http.client.requests	Counter	Number of HTTP requests (with result and http.response.status_code attributes)
http.client.duration	Histogram	Total request duration (ms)
http.client.curl_time	Histogram	Curl total time (ms)
http.client.namelookup_time	Histogram	DNS lookup time (ms)
http.client.connect_time	Histogram	Connection time (ms)
http.client.appconnect_time	Histogram	SSL/TLS handshake time (ms)
http.client.pretransfer_time	Histogram	Pre-transfer time (ms)
http.client.redirect_time	Histogram	Redirect processing time (ms)
http.client.starttransfer_time	Histogram	Time to first byte (ms)

All histogram metrics include the http.request.method attribute for filtering by HTTP method.

Enabling OpenTelemetry Export

The OpenTelemetry API is a no-op by default. To export telemetry data add the OpenTelemetry SDK and an exporter to your release:

%% In rebar.config
{deps, [
    {opentelemetry, "1.5.0"},
    {opentelemetry_experimental, "0.5.1"},
    {opentelemetry_exporter, "1.8.0"}
]}.

Configure the exporter in your sys.config:

[
 {opentelemetry, [
   {span_processor, batch},
   {traces_exporter, otlp}
 ]},
 {opentelemetry_experimental, [
   {readers, [
     #{module => otel_metric_reader,
       config => #{exporter => {opentelemetry_exporter, #{}}}}
   ]}
 ]},
 {opentelemetry_exporter, [
   {otlp_endpoint, "http://localhost:4318"}
 ]}
].

Migration from metrics library

If you were using the previous metrics library integration, note the following breaking changes:

• The mod_metrics application environment option has been removed
• The return_metrics request option has been removed
• The metrics field is no longer included in response maps

To access timing metrics, configure an OpenTelemetry exporter as shown above. The histogram metrics provide the same timing data (DNS lookup, connect time, TLS handshake, etc.) that was previously available via return_metrics.

System dependencies

• libevent-dev
• libcurl4-openssl-dev
• make
• curl
• libssl-dev
• gcc

Testing

The official Erlang Docker image has everything needed to build and test Katipo.

Local httpbin Setup

The test suite uses a local httpbin instance running behind Caddy (for HTTPS/HTTP2/HTTP3 support).

Start the httpbin container:

cd test/http3-httpbin
docker-compose up -d

This starts:

• httpbin: A local instance of the httpbin.org API
• Caddy: Reverse proxy providing HTTPS with auto-generated self-signed certificates on port 8443

Run the tests (requires httpbin to be running):

rebar3 ct

To run with coverage:

rebar3 ct --cover
rebar3 cover --verbose

Stop the containers when done:

cd test/http3-httpbin
docker-compose down

Feature Availability

Some features are only available with newer versions of libcurl. You can check availability at runtime:

katipo:tcp_fastopen_available().    %% curl >= 7.49.0
katipo:unix_socket_path_available(). %% curl >= 7.40.0
katipo:doh_url_available().          %% curl >= 7.62.0
katipo:sslkey_blob_available().      %% curl >= 7.71.0
katipo:http3_available().            %% curl >= 7.66.0