Skip to content

Commit 9864ce0

Browse files
committed
Add English GitHub comment draft for issue #842 reply
1 parent 9e48570 commit 9864ce0

1 file changed

Lines changed: 92 additions & 0 deletions

File tree

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
# GitHub Comment for Issue #842
2+
3+
> Copy the content below (starting from the first `---`) to reply to
4+
> https://github.com/F-Stack/f-stack/issues/842
5+
6+
---
7+
8+
Hi @winstonzhao,
9+
10+
Thanks for the detailed report. We reproduced your test scenario and performed a thorough analysis.
11+
12+
While this client-side bulk-receive pattern is not F-Stack's primary use case (F-Stack is designed for high-throughput server-side workloads where it consistently outperforms the kernel stack), we wanted to verify whether the 3.75x latency gap you observed still exists.
13+
14+
## Test Environment
15+
16+
| Role | Host | IP | NIC |
17+
| --- | --- | --- | --- |
18+
| Python echo server (sender) | f-stack-client | 9.134.211.87 | eth1 |
19+
| F-Stack client (receiver) | local | 9.134.214.176 | DPDK NIC (igb_uio) |
20+
| Kernel client (receiver) | local | 9.134.213.67 | eth1 (virtio-pci) |
21+
22+
- **F-Stack 1.26** + FreeBSD 15.0 + DPDK 24.11.6 LTS
23+
- Python server sends 1,000,000 timestamp messages (`str(time.time_ns()) * 200`, ~3800 bytes each, ~3.8 GB total)
24+
- Clients compiled with `-O2`
25+
26+
## Configuration
27+
28+
We used the same optimized configuration you reported in the issue:
29+
30+
```ini
31+
[dpdk]
32+
idle_sleep=0
33+
pkt_tx_delay=0
34+
35+
[freebsd.sysctl]
36+
net.inet.tcp.delayed_ack=0
37+
net.inet.tcp.sendspace=1677721
38+
net.inet.tcp.recvspace=1677721
39+
net.inet.tcp.sendbuf_max=16777216
40+
net.inet.tcp.recvbuf_max=16777216
41+
net.inet.tcp.sendbuf_auto=1
42+
net.inet.tcp.recvbuf_auto=1
43+
```
44+
45+
## Results
46+
47+
Each test was run 3 times; the median is reported:
48+
49+
| Test | Stack | Config | Median Time | vs Kernel | Result |
50+
| --- | --- | --- | --- | --- | --- |
51+
| T1 | Linux Kernel | default | 9.286s | — | ✅ baseline |
52+
| T3 | F-Stack | optimized (as above) | 9.587s | +3.2% | ✅ success |
53+
| T4 | F-Stack | recvspace=8192, others optimized | 9.417s | +1.4% | ✅ success |
54+
| T2 | F-Stack | delayed_ack=1, idle_sleep=20, pkt_tx_delay=100, recvspace=8192 | N/A | — | ❌ connection reset |
55+
| T5 | F-Stack | delayed_ack=1, others optimized | N/A | — | ❌ connection reset |
56+
57+
**The 3.75x latency gap reported in the issue does not reproduce in our environment.** With the optimized configuration, the F-Stack TCP client receiving 1M messages (3.8 GB) took 9.587s, matching the Linux kernel's 9.286s (only 3.2% gap, within measurement noise).
58+
59+
## Root Cause Analysis
60+
61+
Parameter isolation tests (T4, T5) identified `net.inet.tcp.delayed_ack=1` as the key configuration causing connection failure:
62+
63+
1. With `delayed_ack=1`, ACKs are delayed by up to 40 ms (FreeBSD `TCPTV_DELACK`, `tcp_timer.h`).
64+
2. In `tcp_output.c`, the `TF_DELACK` flag suppresses window-update ACKs from being sent immediately:
65+
```c
66+
if (recwin > 0 && !(tp->t_flags & TF_NEEDSYN) &&
67+
!(tp->t_flags & TF_DELACK) && // TF_DELACK blocks window update
68+
!TCPS_HAVERCVDFIN(tp->t_state)) {
69+
```
70+
3. In a bulk-receive scenario, the receive window cannot be updated in time → window exhaustion → the peer stops sending → eventually RST or timeout.
71+
72+
Setting `delayed_ack=0` resolves this completely. `recvspace=8192` does not affect performance (T4 confirmed — 9.417s with small buffer, still matching kernel).
73+
74+
This is expected FreeBSD TCP stack behavior (not an F-Stack-specific bug); the Linux kernel's delayed ACK implementation has a more aggressive "quick ACK" path and does not block window updates the same way.
75+
76+
## Recommendations
77+
78+
1. **Upgrade to the latest F-Stack** (currently the dev branch, based on DPDK 24.11.6 LTS and FreeBSD 15.0) and retest with `delayed_ack=0`.
79+
2. If the issue persists after upgrading, please **open a new issue** with:
80+
- Exact F-Stack, DPDK, and FreeBSD versions (`git log -1`, `pkg-config --modversion libdpdk`)
81+
- The full `config.ini` (sanitized of sensitive IPs)
82+
- Complete test program source code (client + server)
83+
- `netstat -sp tcp` output before and after the test
84+
- A packet capture (pcap) of the failing connection if possible
85+
86+
## Additional Finding
87+
88+
During testing we observed that F-Stack's `ff_epoll` does not reliably deliver `EPOLLOUT` on connect completion (the kqueue `EVFILT_WRITE` → `EPOLLOUT` translation is unstable). This is a pre-existing `ff_epoll` implementation issue, not the root cause of #842, but it affects client-side development. Using the kqueue native API (`ff_kqueue`/`ff_kevent`) avoids this. We plan to address it in a future fix.
89+
90+
---
91+
92+
*Full investigation report (in Chinese): `docs/issue_842_latency_spec/zh_cn/`*

0 commit comments

Comments
 (0)