Skip to content

Add outgoing-datagram queues to QuicDatagrams/Connection - #3983

Closed
jesup wants to merge 1 commit into
users/jesup/datagram_enginefrom
users/jesup/add_queue_to_connection
Closed

jesup wants to merge 1 commit into
users/jesup/datagram_enginefrom
users/jesup/add_queue_to_connection

Conversation

@jesup

@jesup jesup commented Sep 14, 2026

Copy link
Copy Markdown
Member

No description provided.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Negotiated-size validation and expiry handling currently risk protocol violations, stalled producers, and lost outcomes.

Get a fresh assessment by requesting another Copilot review.

Pull request overview

Integrates per-session outgoing QUIC datagram queues with connection packet scheduling and timers.

Changes:

  • Adds round-robin session queue scheduling and queue-management APIs.
  • Integrates max-age expiry with connection timers.
  • Adds tests for scheduling, teardown, and expiry.
File summaries
File Description
neqo-transport/src/quic_datagrams.rs Implements per-session queue handling and scheduling.
neqo-transport/src/connection/mod.rs Exposes queue APIs and integrates expiry timers.
neqo-transport/src/connection/tests/datagram.rs Tests queue fairness, teardown, and expiry.
Review details

Suppressed comments (1)

neqo-transport/src/connection/mod.rs:1233

  • This schedules the callback at timestamp + max_age, but DatagramQueue::expire_old removes an item only when age > max_age (datagram_queue.rs:250). At an exactly-on-time callback the item remains; if congestion prevents sending it, next_delay selects the same instant again, hits the earliest > now debug assertion below, and otherwise creates a zero-delay callback loop. Make the expiry predicate and scheduled deadline agree (for example, expire at >= max_age).
        if let Some(dgram_time) = self
            .quic_datagrams
            .next_datagram_expiry(self.datagram_default_max_age())
        {
            qtrace!("[{self}] Datagram expiry timer {dgram_time:?}");
            delays.push(dgram_time);
  • Files reviewed: 3/3 changed files
  • Comments generated: 4
  • Review effort level: Balanced

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread neqo-transport/src/quic_datagrams.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/quic_datagrams.rs Outdated
Comment thread neqo-transport/src/quic_datagrams.rs Outdated
@codecov

codecov Bot commented Sep 14, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 98.95288% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.10%. Comparing base (1a3c09a) to head (2260742).

Additional details and impacted files
@@                       Coverage Diff                       @@
##           users/jesup/datagram_engine    #3983      +/-   ##
===============================================================
+ Coverage                        97.07%   97.10%   +0.03%     
===============================================================
  Files                              114      114              
  Lines                            40743    40916     +173     
  Branches                         40743    40916     +173     
===============================================================
+ Hits                             39550    39731     +181     
+ Misses                            1179     1171       -8     
  Partials                            14       14              
Flag Coverage Δ
linux 97.16% <98.95%> (+0.03%) ⬆️
macos 95.27% <97.38%> (+0.12%) ⬆️
windows 95.36% <97.38%> (+<0.01%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Components Coverage Δ
neqo-common 99.31% <ø> (ø)
neqo-http3 95.33% <ø> (ø)
neqo-qpack 96.97% <ø> (ø)
neqo-transport 97.94% <98.95%> (+0.04%) ⬆️
neqo-udp 95.37% <ø> (ø)
mtu 89.13% <ø> (ø)
🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Sep 14, 2026 •

Copy link
Copy Markdown
Contributor

Merging this PR will degrade performance by 2.4%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 10 regressed benchmarks
✅ 88 untouched benchmarks

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Mode Benchmark BASE HEAD Efficiency
❌ WallTime 1-conn/1-100mb-resp (aka. Download) 46.9 ms 49.3 ms -4.95%
❌ Simulation simulated/pacing-false/same-seed 72.7 ms 75.9 ms -4.21%
❌ WallTime 1-conn/1-100mb-req (aka. Upload) 47.2 ms 48.8 ms -3.33%
❌ Simulation simulated/pacing-true/same-seed 72.8 ms 75.1 ms -3.03%
❌ WallTime neqo-neqo-cubic 20.2 ms 20.8 ms -2.82%
❌ WallTime 1-conn/10_000-parallel-1b-resp (aka. RPS) 66.8 ms 68.7 ms -2.76%
❌ WallTime walltime/1000-streams/each-1-bytes 3.6 ms 3.7 ms -2.32%
❌ WallTime walltime/pacing-false/varying-seeds 2 ms 2 ms -2.15%
❌ WallTime walltime/pacing-false/same-seed 1.9 ms 2 ms -2.1%
❌ WallTime walltime/pacing-true/varying-seeds 2 ms 2 ms -2.07%
⚡ WallTime neqo-s2n 50.5 ms 48.8 ms +3.59%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing users/jesup/add_queue_to_connection (2260742) with users/jesup/datagram_engine (1a3c09a)

Open in CodSpeed

@jesup
jesup force-pushed the users/jesup/datagram_engine branch from 5cf2dbe to 662dce4 Compare September 14, 2026 05:31
@jesup
jesup force-pushed the users/jesup/add_queue_to_connection branch from b0a92f6 to 3aed8a2 Compare September 14, 2026 05:31
@jesup
jesup force-pushed the users/jesup/datagram_engine branch from 662dce4 to 1c52106 Compare September 14, 2026 17:56
@jesup
jesup force-pushed the users/jesup/add_queue_to_connection branch from 3aed8a2 to 01af9c9 Compare September 14, 2026 17:56
@jesup
jesup force-pushed the users/jesup/datagram_engine branch from 1c52106 to f4f263a Compare September 14, 2026 19:42
@jesup
jesup force-pushed the users/jesup/add_queue_to_connection branch from 01af9c9 to b01f019 Compare September 14, 2026 19:42
@jesup
jesup force-pushed the users/jesup/datagram_engine branch from f4f263a to c8bef30 Compare September 14, 2026 23:44
@jesup
jesup force-pushed the users/jesup/datagram_engine branch from af19e15 to b31aa87 Compare September 17, 2026 15:03
Comment thread neqo-transport/src/connection/tests/datagram.rs Outdated
Comment thread neqo-transport/src/connection/tests/datagram.rs
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/connection/mod.rs
Comment thread neqo-transport/src/quic_datagrams.rs Outdated
Comment thread neqo-transport/src/quic_datagrams.rs Outdated
Comment thread neqo-transport/src/quic_datagrams.rs
Comment thread neqo-transport/src/quic_datagrams.rs
Comment thread neqo-transport/src/quic_datagrams.rs Outdated
Comment thread neqo-transport/src/datagram_queue.rs
@github-actions

Copy link
Copy Markdown
Contributor

Performance profiles for profiler.firefox.com

Benchmarks (14)
  • neqo-bin-main: pr · base
  • neqo-common-decoder: pr · base
  • neqo-http3-streams_simulated: pr · base
  • neqo-http3-streams_walltime: pr · base
  • neqo-transport-frame_decode: pr · base
  • neqo-transport-min_bandwidth: pr · base
  • neqo-transport-pacer: pr · base
  • neqo-transport-packet_codec: pr · base
  • neqo-transport-range_tracker: pr · base
  • neqo-transport-rx_stream_orderer: pr · base
  • neqo-transport-send_streams: pr · base
  • neqo-transport-sent_packets: pr · base
  • neqo-transport-transfer_simulated: pr · base
  • neqo-transport-transfer_walltime: pr · base
Comparisons (5)

@mxinden mxinden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two blockers. Otherwise this is ready to merge from my end.

Comment thread neqo-transport/src/connection/mod.rs
Comment thread neqo-transport/src/connection/mod.rs Outdated
Comment thread neqo-transport/src/datagram_queue.rs Outdated
New Connection methods: enqueue_datagram, set_datagram_high_water_mark,
set_datagram_max_age, datagram_queue_capacity, drop_session_datagrams.
QuicDatagrams gains a BTreeMap<StreamId, DatagramQueue> and
round-robins across sessions in write_frames one datagram at a time,
so a datagram that doesn't fit is left queued (with its original
timestamp) rather than round-tripped back through enqueue.

Hooks the datagram-expiry deadline into next_delay and expiry itself
into process_timer, the same way every other subsystem's timer works.
Expiry - on the timer or from a shrunken max age - also fires
OutgoingDatagramSpaceAvailable for any queue it unblocks: shedding by age
is how one of these queues is expected to drain, and nothing else would
ever revisit a queue that emptied without sending.

The caller, not this layer, has to keep what it enqueues within the peer's
max_datagram_frame_size: only the path MTU is applied at packet-build time.
send_datagram is the checked path.

QuicDatagrams also keeps a simpler count-bounded queue
(send_datagram/remaining_datagram_queue_capacity) for callers that
don't need per-session tracking, priority, or age.
@jesup
jesup force-pushed the users/jesup/datagram_engine branch from 6b42740 to 1a3c09a Compare September 25, 2026 15:05
@jesup
jesup force-pushed the users/jesup/add_queue_to_connection branch from b18e376 to 2260742 Compare September 25, 2026 15:05
@github-actions

Copy link
Copy Markdown
Contributor

Failed Interop Tests

QUIC Interop Runner, client vs. server, differences relative to users/jesup/datagram_engine at 1a3c09a.

neqo-pr as clientneqo-pr as server
neqo-pr vs. go-x-net: BP BA
neqo-pr vs. haproxy: BP BA
neqo-pr vs. kwik: Z
neqo-pr vs. linuxquic: ⚠️L1
neqo-pr vs. lsquic: L1 C1
neqo-pr vs. msquic: baseline result missing
neqo-pr vs. mvfst: A
neqo-pr vs. neqo: Z A
neqo-pr vs. nginx: BP BA
neqo-pr vs. ngtcp2: Z L1 ⚠️C1 CM
neqo-pr vs. picoquic: Z A
neqo-pr vs. quic-go: A
neqo-pr vs. quic-zig: ⚠️L1
neqo-pr vs. quiche: BP BA
neqo-pr vs. s2n-quic: CM
neqo-pr vs. tquic: S BP BA
neqo-pr vs. xquic: S R Z A L1 C1
aioquic vs. neqo-pr: ⚠️L1 CM
go-x-net vs. neqo-pr: 🚀BP ⚠️BA CM
kwik vs. neqo-pr: BP BA CM
msquic vs. neqo-pr: CM
mvfst vs. neqo-pr: Z L1 C1 CM
neqo vs. neqo-pr: Z A
openssl vs. neqo-pr: LR M A CM
quic-go vs. neqo-pr: CM
quiche vs. neqo-pr: CM
quinn vs. neqo-pr: V2 CM
s2n-quic vs. neqo-pr: ⚠️B BA CM
tquic vs. neqo-pr: CM
xquic vs. neqo-pr: M CM
All results

Succeeded Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr as server

Unsupported Interop Tests

QUIC Interop Runner, client vs. server

neqo-pr as client

neqo-pr as server

@github-actions

Copy link
Copy Markdown
Contributor

Client/server transfer results

Performance differences relative to 1a3c09a.

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server Mean±σ Min–Max Median±MAD MiB/s±σ ΔMedian
google-neqo-cubic 70.4 ± 0.4 69.6 – 71.4 70.4 ± 0.4 454.4 ± 2.6 💔 +1.1 (+1.6%)
neqo-neqo-cubic 19.7 ± 0.2 19.4 – 20.3 19.7 ± 0.2 1620.8 ± 15.3 💔 +0.4 (+1.9%)
neqo-neqo-newreno 20.0 ± 0.1 19.7 – 20.5 20.0 ± 0.1 1597.7 ± 11.7 💔 +0.5 (+2.3%)
neqo-neqo-newreno-nopacing 19.6 ± 0.2 19.3 – 20.2 19.6 ± 0.2 1630.1 ± 16.3 💔 +0.4 (+2.0%)
quiche-neqo-cubic 37.8 ± 0.5 37.2 – 39.2 37.7 ± 0.4 846.2 ± 10.7 💚 -0.5 (-1.4%)

Table above only shows statistically significant changes. See all results below.

All results

Transfer of 33554432 bytes over loopback, min. 100 runs. All unit-less numbers are in milliseconds.

Client vs. server Mean±σ Min–Max Median±MAD MiB/s±σ ΔMedian
google-google 134.7 ± 0.5 133.6 – 135.9 134.6 ± 0.5 237.6 ± 0.8
google-neqo-cubic 70.4 ± 0.4 69.6 – 71.4 70.4 ± 0.4 454.4 ± 2.6 💔 +1.1 (+1.6%)
neqo-google-cubic 240.6 ± 59.5 174.1 – 481.8 227.3 ± 32.4 133.0 ± 32.9 +1.1 (+0.5%)
neqo-neqo-cubic 19.7 ± 0.2 19.4 – 20.3 19.7 ± 0.2 1620.8 ± 15.3 💔 +0.4 (+1.9%)
neqo-neqo-cubic-nopacing 19.5 ± 0.2 19.2 – 20.0 19.5 ± 0.2 1640.1 ± 15.3 +0.2 (+0.8%)
neqo-neqo-newreno 20.0 ± 0.1 19.7 – 20.5 20.0 ± 0.1 1597.7 ± 11.7 💔 +0.5 (+2.3%)
neqo-neqo-newreno-nopacing 19.6 ± 0.2 19.3 – 20.2 19.6 ± 0.2 1630.1 ± 16.3 💔 +0.4 (+2.0%)
neqo-quiche-cubic 33.4 ± 0.3 32.7 – 33.8 33.4 ± 0.3 958.8 ± 8.0 +0.2 (+0.6%)
neqo-s2n-cubic 39.0 ± 0.2 38.7 – 39.3 38.9 ± 0.2 821.3 ± 3.6 -0.0 (-0.0%)
quiche-neqo-cubic 37.8 ± 0.5 37.2 – 39.2 37.7 ± 0.4 846.2 ± 10.7 💚 -0.5 (-1.4%)
quiche-quiche 39.2 ± 0.2 38.8 – 39.9 39.2 ± 0.2 815.5 ± 3.8
s2n-neqo-cubic 113.2 ± 0.4 112.3 – 114.3 113.2 ± 0.3 282.6 ± 1.0 +0.2 (+0.1%)
s2n-s2n ⚠️ 166.4 ± 26.7 134.9 – 260.4 159.5 ± 0.8 192.3 ± 30.8

Download data for profiler.firefox.com or download performance comparison data.

@mxinden mxinden left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the follow-ups!

@github-actions

Copy link
Copy Markdown
Contributor

Benchmark results

No significant performance differences relative to 1a3c09a.

All results
transfer/1-conn/1-100mb-req (aka. Upload)/mtu-1500
       time:   [48.608 ms 48.650 ms 48.702 ms]
       thrpt:  [2.0052 GiB/s 2.0073 GiB/s 2.0091 GiB/s]
Found 3 outliers among 100 measurements (3.00%)
2 (2.00%) high mild
1 (1.00%) high severe
transfer/1-conn/1-100mb-resp (aka. Download)/mtu-1500
       time:   [49.947 ms 49.993 ms 50.040 ms]
       thrpt:  [1.9516 GiB/s 1.9534 GiB/s 1.9552 GiB/s]
Found 2 outliers among 100 measurements (2.00%)
1 (1.00%) low mild
1 (1.00%) high mild
transfer/1-conn/1-1b-resp (aka. HPS)/mtu-1500
       time:   [3.0032 ms 3.0039 ms 3.0047 ms]
       thrpt:  [332.81   B/s 332.90   B/s 332.98   B/s]
Found 6 outliers among 100 measurements (6.00%)
3 (3.00%) high mild
3 (3.00%) high severe
transfer/1-conn/10_000-parallel-1b-resp (aka. RPS)/mtu-1500
       time:   [66.316 ms 66.576 ms 66.843 ms]
       thrpt:  [149.61 Kelem/s 150.20 Kelem/s 150.79 Kelem/s]
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
streams-flow-controlled/walltime/1-streams/each-4194304-bytes
       time:   [10.191 ms 10.193 ms 10.196 ms]
       thrpt:  [392.33 MiB/s 392.42 MiB/s 392.52 MiB/s]
streams-flow-controlled/walltime/10-streams/each-1048576-bytes
       time:   [27.698 ms 27.709 ms 27.720 ms]
       thrpt:  [360.75 MiB/s 360.89 MiB/s 361.03 MiB/s]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
streams/walltime/1-streams/each-1000-bytes
       time:   [132.15 µs 132.69 µs 133.38 µs]
       thrpt:  [7.1499 MiB/s 7.1873 MiB/s 7.2167 MiB/s]
Found 2 outliers among 100 measurements (2.00%)
2 (2.00%) high mild
streams/walltime/1000-streams/each-1-bytes
       time:   [3.8164 ms 3.8267 ms 3.8365 ms]
       thrpt:  [254.55 KiB/s 255.20 KiB/s 255.88 KiB/s]
streams/walltime/1000-streams/each-1000-bytes
       time:   [10.940 ms 10.947 ms 10.954 ms]
       thrpt:  [87.064 MiB/s 87.119 MiB/s 87.173 MiB/s]
Found 1 outliers among 100 measurements (1.00%)
1 (1.00%) high mild
transfer/walltime/pacing-false/same-seed
       time:   [1.9871 ms 1.9881 ms 1.9893 ms]
       thrpt:  [1.9636 GiB/s 1.9648 GiB/s 1.9658 GiB/s]
transfer/walltime/pacing-false/varying-seeds
       time:   [2.0309 ms 2.0319 ms 2.0330 ms]
       thrpt:  [1.9214 GiB/s 1.9225 GiB/s 1.9234 GiB/s]
transfer/walltime/pacing-true/same-seed
       time:   [2.1155 ms 2.1172 ms 2.1192 ms]
       thrpt:  [1.8432 GiB/s 1.8450 GiB/s 1.8465 GiB/s]
transfer/walltime/pacing-true/varying-seeds
       time:   [2.0329 ms 2.0341 ms 2.0354 ms]
       thrpt:  [1.9191 GiB/s 1.9204 GiB/s 1.9215 GiB/s]

Instructions per cycle

Criterion reported no significant timing changes.

All benchmarks
Benchmark IPC before IPC after ΔIPC
transfer/1-conn/1-100mb-resp (aka. Download)/mtu-1500 3.16 3.24 +2.6%
streams/walltime/1000-streams/each-1-bytes 3.26 3.29 +1.0%
transfer/1-conn/10_000-parallel-1b-resp (aka. RPS)/mtu-1500 3.18 3.16 -0.9%
transfer/1-conn/1-100mb-req (aka. Upload)/mtu-1500 3.15 3.17 +0.8%
streams-flow-controlled/walltime/1-streams/each-4194304-bytes 3.01 3.00 -0.4%
streams/walltime/1-streams/each-1000-bytes 2.55 2.54 -0.3%
transfer/walltime/pacing-false/same-seed 2.88 2.89 +0.3%
transfer/walltime/pacing-true/varying-seeds 2.89 2.89 -0.1%
transfer/walltime/pacing-true/same-seed 2.88 2.88 -0.1%
transfer/1-conn/1-1b-resp (aka. HPS)/mtu-1500 2.24 2.24 -0.1%
streams-flow-controlled/walltime/10-streams/each-1048576-bytes 3.17 3.17 +0.1%
transfer/walltime/pacing-false/varying-seeds 2.87 2.88 +0.0%
streams/walltime/1000-streams/each-1000-bytes 3.08 3.08 -0.0%
Profiles for profiler.firefox.com (62)

Download data for profiler.firefox.com or download performance comparison data.

@jesup

jesup commented Sep 29, 2026

Copy link
Copy Markdown
Member Author

Landed as part of #4003 (squash-merged in 9d7e35b).

@jesup jesup closed this Sep 29, 2026
@jesup
jesup deleted the users/jesup/add_queue_to_connection branch September 30, 2026 03:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants