Bound native NIO buffer retention after large socket writes

[stopachka]

Large WebSocket writes leave native NIO copy buffers cached on long-lived IO threads. Corretto 26 does not limit their cached size by default, and these buffers are excluded from both the direct-buffer MXBean and MaxDirectMemorySize. This fixes a confirmed source of native growth implicated in the September 28 host-memory outages.

Set jdk.nio.maxCachedBufferSize=131072 so temporary buffers larger than 128 KiB are freed after I/O. Ordinary Java socket buffers remain reusable and large messages remain supported. Also enable native heap trimming every 60 seconds to return freed glibc pages. Both defaults precede JAVA_OPTS for explicit overrides; the operational notes describe rollback and rollout checks.

Evidence:

• Both failed hosts reached about 120.7 GiB RSS on 123.1 GiB machines with a 90 GiB heap cap. Their final reported heap pressure was below 50%.
• A bounded production allocation trace caught a 36,233,528-byte request through Util.getTemporaryDirectBuffer → SocketChannelImpl.write → Undertow.flushSenders. It maps to the exact repeated 36,237,312-byte resident growth unit observed independently.
• A read-only census found 4.97 GiB in temporary NIO caches on the surviving host, including 4.96 GiB on 32 IO threads. The newer host already retained 1.67 GiB. Separate earlier trimming reclaimed 4.63 GiB of freed allocator pages.

Validation on the exact cached Linux Corretto 26 runtime:

• Eight completed gathering writes retained 276.47 MiB of large native buffers by default. With the 128 KiB cap they retained none of those buffers while writer threads remained alive; all payload lengths and CRC32 checks passed. Idle RSS fell from 473.5 to 199.6 MiB.
• Nonblocking gathering writes with a slow reader and partial/zero writes passed. Cache inspection confirmed 128 KiB buffers remain reusable while 8 MiB buffers are released.
• The actual Compose shell command applies the cap and trim interval, preserves JAVA_OPTS heap options, and accepts both rollback overrides.
• Clojure CI passed build, lint, and all five test shards on 3a57848451595b0d438d6f391041092e6b897784.

The cap applies per cached buffer, not to in-flight or all native memory. Backpressured large writes do more allocation/free work; local timing under emulation is not a production latency guarantee. The failed processes were unavailable for a cache census, so the evidence does not retrospectively account for every byte of their RSS. Observe memory and request/reactivity latency through large-message traffic and a complete backup cycle after rollout.

This PR is not deployed. Instance-count configuration is unchanged from main.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Advanced

Run ID: d1a426bb-72bc-47bb-b174-823dadaea6ae

📥 Commits

Reviewing files that changed from the base of the PR and between 4b69e56 and 3a57848.

📒 Files selected for processing (2)

• server/docker-compose.yml
• server/infra/README.md

🚧 Files skipped from review as they are similar to previous changes (1)

• server/infra/README.md

Included review availability: This review used your included allowance. Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

The web service command sets an NIO cached-buffer limit and enables native-heap trimming once per minute with info-level logging. The infrastructure README documents these settings, observed memory measurements, monitoring, limits, and JVM option overrides.

Changes

Native memory trimming

Layer / File(s)	Summary
Configure and document native memory settings server/docker-compose.yml, server/infra/README.md	The web service command sets the NIO cached-buffer limit and enables periodic native-heap trimming before JAVA_OPTS. The README documents the settings, observed memory measurements, monitoring guidance, limits, and overrides.

Priority: ⬆️ High

Estimated code review effort: 2 (Simple) | ~8 minutes

Change: Bug fix

Merge Risk: ⚪ Minimal · up to 3a578

The production runtime accepts the new memory settings, and the documented override order is preserved. No unresolved change-specific risk remains; the change is ready to merge subject to normal checks.

Architecture Summary

Architecture risk: 🔵 Low · up to 3a578

The change affects 1 system.

Changed systems: server

Architecture concerns
No architecture-level concerns identified.

Review details

Systems and components

• observed — server (service) was modified; 2 changed files map to changed impact.

Before / after behavior

• observed — Modified behavior in server/docker-compose.yml: The web startup command adds the NIO cached-buffer limit, periodic native-heap trimming, and trim logging before JAVA_OPTS; the existing Java options and application launch arguments remain.
• observed — Modified behavior in server/infra/README.md: Added documentation describing the 128 KiB per-buffer NIO cache cap, its 1,024-entry-per-thread limit and restart requirement, observed native memory use, and the separate once-per-minute glibc trimming setting. It also documents monitoring guidance, the cache-retention tradeoff, test limitations, and JVM option overrides.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Analyzed 0 functions across 0…
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Title check	✅ Passed	The title clearly and concisely describes the main change: limiting native NIO buffer retention after large socket writes.
Description check	✅ Passed	The description directly explains the NIO buffer retention issue, the JVM configuration changes, validation evidence, operational considerations, and rollout guidance.

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}