tests: integration: Adjust allocation patterns for Ubuntu 26.04 - #12346
tests: integration: Adjust allocation patterns for Ubuntu 26.04#12346cosmo0920 wants to merge 1 commit into
Conversation
cosmo0920
changed the title
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
Included review availability: Your plan provides up to 8 included reviews per hour; 7 remain after this review. 📝 WalkthroughWalkthroughThe forward integration test now generates a larger allocation-reproduction payload, distinguishes connection-registration failures, and uses revised memory-limit increments. ChangesForward allocation-failure reproduction
Estimated code review effort: 2 (Simple) | ~10 minutes Merge Risk: ⚪ Minimal · up to This change updates a localized integration test to use a larger malformed payload and clearer allocation-failure classification without changing production behavior. No actionable merge-blocking risk remains beyond normal checks and review. Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
aa8aa8a to
d4a478f
Compare
1 participant
This PR is leveraged with Fable5 investigation and plan and orchestrated Opus-5 agent(s).
Investigation
The test works by sandwiching fluent-bit's heap with
RLIMIT_DATA: it measures baselineVmData, then restarts fluent-bit withlimit = baseline + deltaand sends a truncated ~29 KiB forward payload. The payload keeps a msgpack object incomplete, forcingmsgpack_unpacker_reserve_bufferinreceiver_to_unpackerfw_prot.c:1233 to repeatedly grow its buffer (doubling from 1 KiB up to ~32–64 KiB). The test needs a limit where connection setup succeeds but that buffer growth malloc fails, producing "could not allocate msgpack unpacker buffer".The failure log shows the two regimes are separated by at most 16 KiB (27496 KiB → "connection allocation failed", 27512 KiB → "nothing failed"), and no probed delta lands in a working window. This machine runs Ubuntu 26.04.1 with glibc 2.43; on older glibc the heap-growth granularity left a gap where the ~32–64 KiB unpacker expansion was the first allocation to hit the limit. With glibc 2.43's malloc (different top-chunk padding/arena behavior), once the connection-setup allocations succeed, the arena already has enough slack to serve the unpacker growth too — the target window is narrower than the test's step size, or doesn't exist at all. The test's tiny 29 KiB payload makes it inherently fragile: the allocation it wants to fail is the same order of magnitude as malloc's growth granularity.
Plan
MALLOC_ARENA_MAX,MALLOC_TOP_PAD_,MALLOC_MMAP_THRESHOLD_) and/or a much larger truncated payload (hundreds of KiB–MiB) so the unpacker's expansion allocation dwarfs malloc granularity and a wide limit window opens.test_in_forward_001.py, keeping the test's intent (allocation failure closes connection, then recovery), then verify with the single test and the full in_forward scenario file.Root cause
On Ubuntu 26.04 (glibc 2.43) the test's target failure was structurally unreachable, not just flaky. The old 29229-byte payload only ever forced the msgpack unpacker to grow to 32–64 KiB, and glibc serves allocations that small from the ~128 KiB of arena top slack that already exists at startup — the kernel is never asked for memory, so
RLIMIT_DATAis never consulted. Meanwhilefw_conn_add()does a single 1000 KiBflb_mallocfor the connection buffer (buffer_chunk_size), which pinned the too-tight/too-loose boundary at exactly baseline+1000 KiB. A 1 KiB-step sweep confirmed the window between the two regimes was empty: no limit value could ever make the unpacker allocation fail first. glibc tunables (MALLOC_ARENA_MAXetc.) moved the boundary by only 1 KiB and were rejected.Fix (test file only, test_in_forward_001.py)
{"log": "R"*100000}with declared length 16, truncated/zero-padded to 800000 bytes. Peak unpacker allocation becomes ~1 MiB — roughly 8× any plausible glibc arena slack — while staying underbuffer_chunk_size(1024000) soconn->bufis never realloc'd. This opens a measured ~1 MiB-wide trigger window (delta ∈ [1096, ~2160] KiB), which is granularity-independent across glibc 2.35/2.39/2.43.[1152, 1408, 1664, 1920, 2176, 2688, 3200]; the first four all land inside the window.could not register new connection, the coroutine-stack allocation failing) is now reported as its own too-tight status instead of masquerading as "allocation failure was not triggered".Verification
test_in_forward_001.pyfile: 72 passed, no regressions.test_in_opentelemetry_non_matching_route_drops_all_signals[profiles-...]failed once mid-run but passes standalone and in the final clean run — worth keeping an eye on separately.Enter
[N/A]in the box, if an item is not applicable to your change.Testing
Before we can approve your change; please submit the following in a comment:
If this is a change to packaging of containers or native binaries then please confirm it works for all targets.
ok-package-testlabel to test for all targets (requires maintainer to do).Documentation
Backporting
Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.
Summary by CodeRabbit