Skip to content

Nginx using debian trixie - #1200

Merged
MDA2AV merged 2 commits into
MDA2AV:mainfrom
joanhey:nginx-trixie
Aug 17, 2026
Merged

Nginx using debian trixie#1200
MDA2AV merged 2 commits into
MDA2AV:mainfrom
joanhey:nginx-trixie

Conversation

@joanhey

@joanhey joanhey commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

Description

Using debian trixie, we can simplify the docker file, as use Openssl 3.5 and don't need to compile a third ssl lib.

Also update nginx to v1.31.3 and add h2c tests.


PR Commands — comment on this PR to trigger (requires collaborator approval):

Command Description
/benchmark -f <framework> Run every test the framework subscribes to
/benchmark -f <framework> -t <test> Run one test only
/benchmark -f <framework> --save Run and save results (updates the leaderboard on merge)
/benchmark -f <framework> -t <test> --save Run one test and save results
/benchmark -f <framework> --compare <other> Measure the deltas against another framework instead of this one
/benchmark-multiple -f <fw1>,<fw2>,... Benchmark several frameworks in one run — takes -t and --save too; saved results land in a single commit
/benchmark-multiple --save No -f needed: benchmark and save every framework the PR touches
/benchmark-test -t <test> Benchmark all enabled frameworks subscribed to <test> and save the results

For /benchmark, always specify -f <framework>; the flags combine in any order. Results come back as a comment with a per-profile table of RPS, p99, CPU and memory — one table per framework on multi runs. A new benchmark comment while a run is in flight queues behind it (one deep) instead of cancelling it. For multi-framework PRs (dependency bumps, same-language refactors) prefer /benchmark-multiple, which runs everything in a single job and commits all saved results together, so no run overwrites another. --compare works on single-framework runs only.

What the deltas are measured against. By default, this framework's own results published on main - answering "did this change help?". When you are tuning a variant or a successor entry, --compare re-bases them on another entry instead:

/benchmark -f genhttp-11 --compare genhttp

The reply states which baseline it used, and profiles the other framework does not run show n/a rather than a delta.


Run benchmarks locally

You can validate and benchmark your framework locally with the lite script — no CPU pinning, fixed connection counts, all load generators run in Docker.

./scripts/validate.sh <framework>
./scripts/benchmark-lite.sh <framework> baseline
./scripts/benchmark-lite.sh --load-threads 4 <framework>

Requirements: Docker Engine on Linux. Load generators (gcannon, h2load, h2load-h3, wrk, ghz) are built as self-contained Docker images on first run.

@joanhey

joanhey commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

FAIL [h2c-only]: port 8082 also answered HTTP/1.1 with 200 — dual-serving lets the benchmark measure h1 throughput instead of h2c. The h2c listener must refuse HTTP/1.1 requests.

This break the RFC, I'll open a issue about that.

@joanhey

joanhey commented Aug 17, 2026

Copy link
Copy Markdown
Contributor Author

/benchmark -f nginx

@github-actions

Copy link
Copy Markdown
Contributor

👋 Benchmark request received. A collaborator will review and approve the run.

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark Results

Framework: nginx | Test: all tests

Test Conn RPS CPU Mem Δ RPS Δ Mem
baseline 512 3,082,514 6654.3% 3.5GiB +0.6% ~0%
baseline 4096 3,250,187 6642.6% 3.5GiB -2.9% ~0%
pipelined 512 4,092,563 6598.0% 3.5GiB -0.5% ~0%
pipelined 4096 3,987,610 6369.1% 3.5GiB -2.1% ~0%
limited-conn 512 2,245,438 6291.6% 3.5GiB -1.5% ~0%
limited-conn 4096 2,626,549 6382.4% 3.5GiB -0.3% ~0%
json 4096 1,737,721 6489.0% 3.5GiB -4.0% ~0%
json-tls 4096 1,333,698 6558.4% 3.6GiB -6.7% ~0%
static 1024 1,139,901 6622.0% 3.5GiB -0.4% ~0%
static 4096 1,141,526 6538.6% 3.5GiB ~0% ~0%
static 6800 1,137,321 6462.5% 3.5GiB -1.5% ~0%
static-tls 1024 732,412 6592.4% 3.5GiB -2.2% ~0%
static-tls 4096 706,711 6534.1% 3.6GiB -3.1% ~0%
static-tls 6800 676,988 6362.1% 3.7GiB -2.8% ~0%
baseline-h2 256 3,295,384 6496.4% 3.5GiB -6.1% ~0%
baseline-h2 1024 3,332,251 6566.8% 3.6GiB -2.6% +2.9%
static-h2 256 796,590 6599.3% 3.6GiB -2.9% +2.9%
static-h2 1024 801,719 6393.5% 3.8GiB -1.4% +8.6%
baseline-h3 64 4,904,296 2997.4% 3.5GiB -0.2% ~0%
static-h3 64 366,674 5089.6% 3.6GiB -4.1% ~0%
Full log
.
23Main benchmark duration is started for thread #17.
.
38.

44Main benchmark duration is started for thread #.50
.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Certificate: RSA 2048 bits
Negotiated Group: x25519
Resumption: no
Application protocol: h3

finished in 5.01s, 4840449.20 req/s, 369.24MB/s
requests: 24202246 total, 24206342 started, 24202246 done, 24202246 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 24202246 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.80GB (1935903840) total, 1.58GB (1694157220) headers (space savings 40.68%), 23.08MB (24202246) data
UDP datagram: 758518 sent, 1980155 received
                 min         max         median     p95        p99        mean         sd        +/- sd
request     :      197us      3.40ms       801us     1.13ms     1.45ms      826us       171us    78.81%
connect     :     1.98ms      5.62ms      3.07ms     5.21ms     5.62ms     3.19ms       827us    76.56%
TTFB        :     3.15ms      8.51ms      4.75ms     7.58ms     8.51ms     5.00ms      1.14ms    75.00%
req/s       :   68337.07    78649.11    75928.39   78542.87   78649.11   75627.00     2474.67    73.44%
min RTT     :       22us       310us        25us      243us      310us       58us        74us    87.50%
smoothed RTT:      322us       616us       500us      588us      616us      490us        66us    65.63%
packets sent:      10710       12332       11902      12322      12332   11853.84      390.34    73.44%
packets recv:      27674       32101       31149      32027      32101   30940.92     1108.27    82.81%
packets lost:          0           0           0          0          0       0.00        0.00   100.00%
GRO packets :          1           1           1          1          1       1.00        0.00   100.00%
[info] CPU 3289.6% | Mem 3.6GiB

[run 3/3]
starting benchmark...
progress: Warm-up started for thread #100% of clients started
54.
.

44.
46.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Certificate: RSA 2048 bits
Negotiated Group: x25519
Resumption: no
Application protocol: h3

finished in 5.01s, 4893848.60 req/s, 373.32MB/s
requests: 24469243 total, 24473339 started, 24469243 done, 24469243 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 24469243 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 1.82GB (1957263600) total, 1.60GB (1712847010) headers (space savings 40.68%), 23.34MB (24469243) data
UDP datagram: 766672 sent, 1989054 received
                 min         max         median     p95        p99        mean         sd        +/- sd
request     :      189us      4.27ms       786us     1.14ms     1.44ms      822us       176us    79.78%
connect     :     1.66ms      6.71ms      2.91ms     5.35ms     6.71ms     3.12ms      1.04ms    79.69%
TTFB        :     2.78ms     10.44ms      4.63ms     8.27ms    10.44ms     4.95ms      1.42ms    76.56%
req/s       :   60875.90    85024.22    77735.60   80527.86   85024.22   76461.24     4488.19    92.19%
min RTT     :       21us       297us        24us      287us      297us       75us        95us    81.25%
smoothed RTT:      345us       820us       485us      612us      820us      492us        78us    75.00%
packets sent:       9529       13319       12185      12624      13319   11981.25      706.73    92.19%
packets recv:      24476       34191       31520      32815      34191   31079.97     1945.03    78.13%
packets lost:          0           0           0          0          0       0.00        0.00   100.00%
GRO packets :          1           1           1          1          1       1.00        0.00   100.00%
[info] CPU 3132.6% | Mem 3.6GiB

=== Best: 4904296 req/s (CPU: 2997.4%, Mem: 3.5GiB) ===
[info] saved results/baseline-h3/64/nginx.json
httparena-bench-nginx
httparena-bench-nginx

==============================================
=== nginx / static-h3 / 64c (tool=h2load-h3) ===
==============================================
[info] waiting for server...
[info] server ready

[run 1/3]
starting benchmark...
24Main benchmark duration is started for thread #22.
.
.
.
32.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Certificate: RSA 2048 bits
Negotiated Group: x25519
Resumption: no
Application protocol: h3
63. Stopping all clients.
22. Stopping all clients.

finished in 5.01s, 367358.40 req/s, 5.59GB/s
requests: 1836792 total, 1840888 started, 1836792 done, 1836792 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1837038 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 27.97GB (30033325249) total, 121.23MB (127123620) headers (space savings 40.01%), 27.83GB (29887223852) data
UDP datagram: 2372304 sent, 21445666 received
                 min         max         median     p95        p99        mean         sd        +/- sd
request     :     2.78ms    988.56ms      9.31ms    26.00ms    56.12ms    12.56ms     14.33ms    95.31%
connect     :     1.90ms      9.31ms      3.32ms     5.22ms     9.31ms     3.68ms      1.11ms    85.94%
TTFB        :     5.94ms     14.96ms      7.08ms    11.71ms    14.96ms     7.91ms      2.01ms    85.94%
req/s       :    1612.00     7346.70     6784.57    7185.01    7346.70    5739.61     1805.14    76.56%
min RTT     :       10us       826us        12us      117us      826us       41us       132us    96.88%
smoothed RTT:      134us      2.42ms       575us     1.92ms     2.42ms      735us       580us    82.81%
packets sent:       7386       50321       45361      49692      50321   37069.25    14636.67    75.00%
packets recv:      92364      433095      397240     421006     433095  335089.53   105707.52    76.56%
packets lost:          0         109          22         81        109      27.16       23.26    76.56%
GRO packets :          1           1           1          1          1       1.00        0.00   100.00%
[info] CPU 5089.6% | Mem 3.6GiB

[run 2/3]
starting benchmark...
.

.Main benchmark duration is started for thread #11.

TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Certificate: RSA 2048 bits
Negotiated Group: x25519
Resumption: no
Application protocol: h3
46.
51.

finished in 5.01s, 361761.60 req/s, 5.51GB/s
requests: 1808808 total, 1812904 started, 1808808 done, 1808808 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1809046 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 27.54GB (29575572821) total, 119.39MB (125186317) headers (space savings 40.01%), 27.41GB (29431635316) data
UDP datagram: 2519686 sent, 21412964 received
                 min         max         median     p95        p99        mean         sd        +/- sd
request     :     1.87ms    843.40ms      9.56ms    25.18ms    47.90ms    12.51ms     12.76ms    95.04%
connect     :     1.59ms      7.54ms      3.34ms     5.89ms     7.54ms     3.68ms      1.18ms    71.88%
TTFB        :     4.18ms     14.96ms      7.30ms    12.19ms    14.96ms     7.86ms      2.32ms    68.75%
req/s       :    2260.10     7094.30     6735.72    7014.43    7094.30    5652.17     1615.14    75.00%
min RTT     :       10us       487us        13us       64us      487us       29us        69us    95.31%
smoothed RTT:       84us      2.95ms       632us     2.32ms     2.95ms      907us       707us    76.56%
packets sent:      10081       57477       49801      56289      57477   39372.09    17094.63    65.63%
packets recv:     129358      422554      398404     418380     422554  334578.56    96246.82    73.44%
packets lost:          0         130          29         81        130      33.58       24.54    71.88%
GRO packets :          1           1           1          1          1       1.00        0.00   100.00%
[info] CPU 5039.6% | Mem 3.7GiB

[run 3/3]
starting benchmark...
45.
TLS Protocol: TLSv1.3
Cipher: TLS_AES_128_GCM_SHA256
Server Temp Key: X25519 253 bits
Certificate: RSA 2048 bits
Negotiated Group: x25519
Resumption: no
Application protocol: h3
. Stopping all clients.
26. Stopping all clients.

finished in 5.01s, 360135.00 req/s, 5.48GB/s
requests: 1800675 total, 1804771 started, 1800675 done, 1800675 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 1800914 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 27.42GB (29443077435) total, 118.85MB (124624493) headers (space savings 40.01%), 27.29GB (29300107208) data
UDP datagram: 2237386 sent, 21151451 received
                 min         max         median     p95        p99        mean         sd        +/- sd
request     :     1.77ms    715.87ms      9.58ms    25.04ms    43.55ms    12.43ms     12.75ms    95.07%
connect     :     1.67ms      5.08ms      3.18ms     4.29ms     5.08ms     3.15ms       802us    59.38%
TTFB        :     4.27ms     11.49ms      6.64ms     9.17ms    11.49ms     6.86ms      1.38ms    75.00%
req/s       :    2448.41     7104.29     6743.71    7043.06    7104.29    5626.70     1543.76    75.00%
min RTT     :       11us       380us        13us       22us      380us       20us        45us    98.44%
smoothed RTT:      144us      2.24ms       661us     1.83ms     2.24ms      787us       495us    71.88%
packets sent:      10770       50183       45485      49138      50183   34961.16    15069.05    64.06%
packets recv:     144354      416160      396117     415609     416160  330492.42    90998.25    76.56%
packets lost:          0          97          25         74         97      30.94       22.33    70.31%
GRO packets :          1           1           1          1          1       1.00        0.00   100.00%
[info] CPU 4906.9% | Mem 3.7GiB

=== Best: 366674 req/s (CPU: 5089.6%, Mem: 3.6GiB) ===
[info] saved results/static-h3/64/nginx.json
httparena-bench-nginx
httparena-bench-nginx
[info] skip: nginx does not subscribe to gateway-64
[info] skip: nginx does not subscribe to gateway-h3
[info] skip: nginx does not subscribe to production-stack
[info] skip: nginx does not subscribe to unary-grpc
[info] skip: nginx does not subscribe to unary-grpc-tls
[info] skip: nginx does not subscribe to stream-grpc
[info] skip: nginx does not subscribe to stream-grpc-tls
[info] skip: nginx does not subscribe to echo-ws
[info] skip: nginx does not subscribe to echo-ws-pipeline
[info] skip: nginx does not subscribe to echo-ws-limited
[info] rebuilding site/data/*.json
[purged stale] zix-grpc.json (zix-grpc)
[purged stale] zix-ws.json (zix-ws)
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/frameworks.json
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/results/nginx.json - 20 new, 20 total
[updated] /home/diogo/actions-runner/_work/HttpArena/HttpArena/site/data/current.json
[info] done
[info] restoring loopback MTU to 65536

@MDA2AV
MDA2AV merged commit b706ed4 into MDA2AV:main Aug 17, 2026
4 checks passed
@joanhey
joanhey deleted the nginx-trixie branch August 17, 2026 23:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants