feat(rocksdb): add optional direct I/O for reads and flush/compaction - #3594
feat(rocksdb): add optional direct I/O for reads and flush/compaction#3594bocharov wants to merge 1 commit into
Conversation
|
Hi @bocharov, Thank you for your pull request. Please review our Contributing Guide. Please make sure you understand your changes and explain your reasoning in this pull request. Low-quality pull requests may be closed. |
Add two opt-in RocksDB options, both off by default: rocksdb.use_direct_reads rocksdb.use_direct_io_for_flush_and_compaction They map to the corresponding RocksDB DBOptions and let an operator bypass the OS page cache. With buffered reads the page cache grows with the working set and, under a hard memory limit (e.g. a cgroup memory limit), is accounted against that limit and can lead to the process being OOM-killed; direct reads keep the process memory bounded to the block cache (size it to the working set when enabling). Both default off, so existing behavior is unchanged. They are applied when the DB is opened, so they are read-only at runtime. Documented in kvrocks.conf; covered by a cppunit test (defaults, file parsing, read-only) and a gocase test (defaults + read-only via a running server).
f879da7 to
981104d
Compare
|
Hello @bocharov. Please disclose in the PR which AI tools and models you used. Thank you. |
|
@bocharov Want to know how would you use those configurations? |
There was a problem hiding this comment.
Pull request overview
Adds opt-in RocksDB direct I/O configuration for reads and flush/compaction.
Changes:
- Adds two read-only configuration options, disabled by default.
- Maps them to RocksDB options during database initialization.
- Adds configuration parsing and runtime immutability tests.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
src/config/config.h |
Stores direct I/O settings. |
src/config/config.cc |
Registers read-only configuration fields. |
src/storage/storage.cc |
Applies settings to RocksDB options. |
kvrocks.conf |
Documents the new options. |
tests/cppunit/config_test.cc |
Tests defaults, parsing, and immutability. |
tests/gocase/unit/config/config_test.go |
Tests server-visible defaults and rejected updates. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| options.use_direct_reads = config_->rocks_db.use_direct_reads; | ||
| options.use_direct_io_for_flush_and_compaction = config_->rocks_db.use_direct_io_for_flush_and_compaction; |
| // Optionally bypass the OS page cache with O_DIRECT. This bounds the process's memory footprint | ||
| // to the RocksDB block cache instead of letting the (buffered-read) page cache grow, which is | ||
| // useful under a hard memory limit; reads then rely on the block cache, so size it accordingly. |
| # Use O_DIRECT for reads (bypassing the OS page cache). With buffered reads the page cache | ||
| # grows with the working set and, under a hard memory limit (e.g. a cgroup limit), is counted | ||
| # against that limit and can trigger the process to be killed. Direct reads keep memory usage | ||
| # bounded to the RocksDB block cache, at the cost of losing the page cache as a read cache — so | ||
| # size rocksdb.block_cache_size to the hot working set when enabling this. |
|
@git-hulk Honestly: the driver was bounding cgroup-accounted page cache on memory-capped kvrocks pods — but after deploying, our OOM turned out to be output buffers, which O_DIRECT doesn't touch, and that one workload has since moved off kvrocks. We still run kvrocks for other use cases, so the option remains relevant to us where RSS is page-cache-dominated — but I no longer have a proven production win for it. Happy to close unless you'd still like it in. @jihuayu Built with Claude Code (Anthropic Claude), reviewed and verified by me. |
Summary
Add two opt-in RocksDB options, both off by default:
rocksdb.use_direct_readsrocksdb.use_direct_io_for_flush_and_compactionThey map to the corresponding RocksDB
DBOptionsand let an operator useO_DIRECTto bypass the OS page cache.Motivation
With buffered reads, the OS page cache grows with the working set. Under a hard memory limit (for example a cgroup memory limit), that page cache is accounted against the limit and can lead to the process being OOM-killed even when the block cache and heap are well within budget. Enabling direct reads keeps the process's resident memory bounded to the RocksDB block cache.
The trade-off is that reads no longer benefit from the OS page cache, so
rocksdb.block_cache_sizeshould be sized to the working set when enabling this.Notes
no, so existing behavior is unchanged.CONFIG SETis rejected rather than silently ignored).Tests
Config.DirectIO: defaults off, parsed from the config file, and read-only at runtime.TestConfigDirectIO: through a running server, defaultsnoandCONFIG SETrejected.