Repository navigation
feat(remote_cxl): prefetch hints (read and write) through a region extension, so the memory node can prepare KV before transfers - #130
Conversation
d7af561 to
f3692fe
Compare
A remote CXL agent can load one out-of-tree region extension, registered
under the maru.region_extensions entry-point group and named with
--region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a
package that provides one can turn it on for the installed service with
a drop-in). It serves memory that is slow until a device has brought a
page in (a CXL-SSD with a DRAM cache in front of its flash):
- validate_access may refuse a region before it is mapped, mapped hears
it once the NIC has it (raising keeps the region out of use), and
retire is asked before a region goes back; while it says no, the
region stays here unregistered and the sweep asks again.
- hint takes read_hint / read_hint_end messages (op "extension") from
any authenticated connection, once the agent has checked the epoch, the
action and that every range lies in a region it maps and some
connection holds, under a matching handle. The check runs under the
agent's lock and the extension is called after it, so no other request
waits for the extension. A connection may keep at most
--max-hint-requests (4096) hint requests open.
- The optional connection_closed(conn) is called for every connection
that closes, so the extension can end what its hints started.
- The hello advertises region_extension {"version": 1, "actions": [...]}
and RegionMapClient keeps the hint actions it advertises; anything else
means no hints.
An agent that fails to load its extension exits. Without one, nothing
changes.
… bytes early A serving engine knows which stored keys a request will load well before its worker starts the load. Read hints carry that to the memory side, where a region extension can bring the bytes into a faster tier. Maru's KV data path stays as it is: a hint is advice that never gates a READ or WRITE, and one that is late or lost costs only the time it would have saved. - read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on. MaruHandler.hint_read / end_read_hint queue a hint and its end; one thread per backend probes maru-server for where the hits are (a probe with locations, which takes no lease) and sends read_hint and read_hint_end to every agent whose hello advertises them, in messages of at most 4096 ranges. At most 1024 hints wait; ends are never dropped. The thread keeps at most 4096 hinted requests open and ends the oldest past that. - A client that loads with read_hints on hints each load as its READs start, under a name of its own, and ends that hint after the load (for engines that name no request to their storage). - The vLLM scheduler connector hints a request when it first finds its asynchronous load, or on arrival with maru_read_hint_on_arrival, and ends the hint when vLLM reports the load done (or the request finishes first). Only the scheduler gets read_hints, so vLLM loads are not hinted twice. - The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since vLLM's connector API is first called at the next step boundary. It is installed only if the method takes (self, request) and passes on only a result whose first item is a request with an id; otherwise hints on arrival are off with a warning.
A region extension may front memory whose contents a device keeps in a write-back cache (a CXL-SSD). Two things about such memory reach it now. Fresh regions. A region its owner allocated in this session holds no KV until the owner writes to it, and the owner writes only after a successful map reply. Until the first one, its maps carry fresh=true, retries included; after it (an agent restart, a remap, a region given back) they do not. The agent takes fresh only with the region's own handle and only on the map that maps the region there first, and passes it to RegionExtension.mapped(fresh=) before the reply leaves, so a device can drop the region's stale contents before any WRITE reaches it. fresh is the owner's word, to be used only on the region's own bytes. An agent that does not know fresh ignores it. Memory generations. A write-back cache loses what it held on a reset without a reboot of its node, and Maru already treats a changed agent boot as a reboot: owners give their regions back and readers load misses. An extension may tell the memory's generation (optional memory_generation(), local state, outside the agent's lock), and the agent advertises "<node boot id>:<generation>" as its boot. - The agent reads the generation at start (a failure stops it), on a thread every --memory-generation-poll-s (100 ms) and on every hello, map and sweep, one caller at a time. When it changes, every mapping is deregistered from the NIC, so RDMA through an old registration fails, and dropped as a release does. Map replies now carry the boot, and a worker takes it over its connection's. - A generation the extension does not tell for --memory-generation-grace-s (2 s) is unknown: every mapping goes, the boot reads <boot id>:unknown and maps are refused until it answers. The boot after the n-th such period is <boot id>:<generation>~<n>, so nothing mapped before one is trusted after it. - A map whose boot changed while it ran fails and its registration goes.
…'s pages Without them, a store's RDMA WRITEs into memory with a slow tier behind a cache make the device fill each written page from the slow tier first, page by page. On a CXL-SSD, a build that stored without write hints wrote at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other settings differed too). - write_hints (vLLM: maru_write_hints, which follows maru_read_hints unless set) makes a store, once it has its pages and before it posts the WRITEs, queue a write_hint on the hint thread: the pages, in write order, under a token for the store. Fire and forget; no end. With vLLM only the worker sends them (it stores; the scheduler hints loads). - The agent takes write_hint with the read hints' checks; it keeps no request open for it. A region extension says which hint actions it takes (optional hint_actions; read_hint and read_hint_end without it), the hello advertises those, Maru sends only those and the agent refuses others. A write hint gives no leave to change the pages. - The hint thread class is MemoryHints now that it sends both kinds.
The memory side brings hinted requests' bytes in in hint order, which is not the order their loads run in, so loads read chunks still being brought in while others' wait ready. A worker now tells the memory side when the load of a request starts. - read_promotes (vLLM: on in the worker with maru_read_hints and maru_async_load) makes MaruHandler.promote_read(request_id) queue a read_promote on the hint thread: no ranges, no end, fire and forget. The vLLM worker calls it once per remote_cxl load, before its first batch is read. - The worker did not send the request's hint (its scheduler did, on another connection), so it sends the promote to every agent of the pool whose extension takes it; the extension matches it to a hint by request id alone. - The agent takes read_promote like read_hint_end and keeps no request open for it; it advertises and accepts it only if the region extension lists it in hint_actions.
A queued write hint reaches the memory side about when the store's WRITEs do, so most WRITEs still landed on pages the device had to fill from its slow tier first. Stores run off the request path (vLLM's write-behind thread), so they can afford a short wait. - When an agent's region extension takes write_wait, a store sends its pages for that agent in a write_wait (the store's token, the ranges in write order, timeout_ms) and waits for the answer, then posts the WRITEs whatever it is. The waits of one store share write_wait_ms (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an agent that does not answer costs the reply timeout on top. Other agents still get a queued write hint. Waits are off unless write hints are on. - Waits live in waits.py (MemoryWaits): each thread has its own connection per agent, so one store's wait never holds up another's, a hint or a region map. An agent that cannot be reached is not asked again for 5 s; its stores send write hints meanwhile. - The agent checks a write_wait like a write hint, plus timeout_ms (1 to 10000), keeps no request open for it, and calls the extension outside its lock on the requesting connection's own thread; the extension's answer (a map, or None) is the reply. Only that connection waits. It is advertised and accepted only if the extension lists it in hint_actions.
… hint leaves vLLM
…ead sends the hint
7eef3b6 to
94365de
Compare
seohui-XCENA
left a comment
There was a problem hiding this comment.
Read the diff and ran the remote_cxl and vLLM unit tests on 94365de (1188 passed). In one case, hint code can break a store or load, which the hints-are-advice design is meant to rule out (inline on client.py L878). The rest are suggestions. I reproduced the first two inline issues with unit tests on the PR's fixtures, not on a device.
Nits:
_MAX_HINT_RANGES(service.pyL96) and the10_000inconfig.pyL171 repeathints.MAX_HINT_RANGESandregion_map.MAX_WRITE_WAIT_MS. Both sides must agree, soregion_map.pycould hold them.- The agent CLI accepts
--memory-generation-poll-s≤ 0 (a busy loop) and--max-hint-requests≤ 0 (every read hint is refused). It also doesn't close the loaded extension whenRegionMapServicefails to start. shutdown()doesn't stop_arrival_pool. A request that arrives afterwards can create a new scheduler handler through_ensure_handler().
…wait backs off Hints and waits are advice, but the store's write preparation and the load's read hint ran after the transfer count went up and before the try that brings it down. If either raised, the transfer never ended: the pages stayed sending, the store failed, and close() waited for the RDMA timeout. Both now run inside that try, and any exception from hint code is logged and ignored. One way they raised: the hint connections read the agent key lazily, so a key file changed after start made load_key raise. They now reuse the key the data path read at start. MemoryWaits also catches a failing connect and backs off instead of raising. A write wait that fails or times out now backs the agent off for UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do not each pay the reply timeout.
hello() reads boot without the lock, and boot was built from three fields the generation watch changes one by one. When the generation became known again, a concurrent hello could advertise the boot from before the unknown period. The boot is now one string, rebuilt under the lock whenever the generation state changes (known again, changed, unknown), so a reader sees the boot from before or after a change, never a mix.
…he extension closed on a failed start - MAX_HINT_RANGES moves to maru_remote.region_map beside MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the write_wait_ms config check all use these two, so both sides agree. - --max-hint-requests and --memory-generation-poll-s must be above zero (zero refused every read hint or polled in a busy loop). - When RegionMapService fails to start, the loaded region extension is closed before the agent exits.
…new handler shutdown() closed the scheduler's MaruHandler but left the arrival-hint pool running, so a request that arrived afterwards could create a new handler through _ensure_handler(). The scheduler connector now has its own shutdown(): it marks itself shut down, stops the arrival pool (waiting for a hint under way), then closes the handler. Arrival hints are submitted under the hint lock, so none lands on a stopped pool.
|
Thanks for the careful read. All three nits are fixed:
Re-requesting your review. |
seohui-XCENA
left a comment
There was a problem hiding this comment.
Re-checked on 373abf6 and ran the unit tests (2112 passed). The earlier items are all fixed. Two new ones, both reproduced with unit tests on the PR's fixtures, not on a device. The first is a one-line fix that I'd like in before merge. The second is optional.
…on is unknown map() checks the generation before mapping, and _establish compares the boot it read at the start with the boot at the end. If the generation turned unknown between the check and the first boot read, both boots read ":unknown", the comparison passed, and the map succeeded. Its registration then outlived the unknown period: the sweep kept it and the recovery released nothing, so the owner could RDMA into memory whose state was unknown. _establish now also refuses a map whose generation is unknown at the end. A boot that is known at the end already differs from an unknown one read at the start, so that case was covered.
With several remote_cxl pools, the scheduler's lookup hint was routed by the first key it named. When vLLM's own prefix cache held a chunk or more, that key was not the request's first chunk key, which routes the load, so the hint went to another pool and was dropped there. The hint is now routed by the request's first chunk key, as the load and the arrival hint are.
MemoryHints takes its locate and send callables from the backend and holds the background thread, the queue, the drop policy and the end bookkeeping, none of which depends on RDMA or agents. It now lives in maru_handler/hints.py, so another backend (the cxl backend of #137) uses it without depending on the remote_cxl package. The shared wire limits stay in maru_remote.region_map, which maru_common already imports.
seohui-XCENA
left a comment
There was a problem hiding this comment.
Re-checked on 35e481a: both items are fixed, and the repros from my last review pass. Unit tests: 2163 passed; the 107 failures are CUDA-kernel tests that fail the same way on main (c55266b) here.
One thing from the merge with #141: the scheduler's read hint (and the arrival hint) names the bare chunk keys, but on remote_cxl a chunk is now stored as key + slice suffix at TP>1 and as key_c{n}_g{gid} with several KV cache groups. _hint_locate stops at the first miss, so in either setup no read hint is sent. In a two-pool unit test with two slices, no agent got a hint, while the same test at TP=1 does; the several-groups case follows from the key names. Load-time hints and write hints name the real keys and are unaffected. Nothing breaks; the prefetch just doesn't happen.
Could you either state the limit here (e.g. in read_hints.md and the maru_read_hints row: scheduler read hints apply at TP=1 with one KV cache group for now) and handle it in a follow-up PR, or have the hint name the objects the load reads in this PR? Either is fine with me.
…r now Since #141, remote_cxl stores a chunk under its key plus a slice suffix (TP > 1) or a group suffix (several KV cache groups). The scheduler's read hint names the bare chunk keys, finds no stored chunk and sends nothing. Loads, the hints a worker sends as a load starts, read promotes and write hints are unaffected. A follow-up makes the hint name the objects the load reads.
|
@seohui-XCENA Thanks for catching the interaction with #141. I took the first option: 60c677a states the limit in A follow-up PR will make the scheduler's hint name the objects the load reads (every slice and group key of each chunk). |
🤔 Background & Motivation (Why)
Maru can place KV on remote memory that has a front cache, such as a CXL-SSD (a DRAM cache in front of flash). But Maru tells the memory node only which regions are mapped, so the device cannot bring KV into its DRAM before a READ arrives: a READ of KV still on flash waits for flash. Maru does know which KV a request will read between the request's arrival and its READ (a median of 14.3 ms in the serving replay below). This PR sends that to the memory node as prefetch hints: the KV a load will read (read hints) and the pages a store will write (write hints), so that a CXL-SSD can bring them from flash into its DRAM cache before the transfer. Maru moves no KV for a hint; the device moves its own pages, through a region extension: a hook on the remote CXL agent, implemented by a separately installed package. This PR adds the hooks only; Maru's KV data path is unchanged, and memory nodes backed by plain DRAM or CXL DRAM are unaffected. In the serving replay, read hints cut the mean time to first token (TTFT) of turns that reuse KV from 1.18× to 1.004× of the same path with all KV in device DRAM.
Scope:
remote_cxlonly. With the localcxlbackend, where the GPU node maps the CXL-SSD itself and there is no agent, the hint settings (read_hints,write_hints,read_promotes, where a read promote tells the memory node that a hinted load starts now so its KV goes first, and their vLLM forms such asmaru_read_hints) are rejected at startup, andMaruHandler.hint_read,promote_readandend_read_hintdo nothing. The existing handler plugin hookon_batch_retrieveruns at load time, too late to prepare anything. A follow-up PR, stacked on this one, will allow these settings on thecxlbackend and pass the calls to handler plugins (maru.handler_plugins).Figure 1 covers reads; writes and resets are in the table below.
Figure 1. Where a read hint leaves vLLM (schematic). With prefetch hints (b), Maru's wrapper runs on vLLM's input thread and calls the scheduler's Maru connector there; its hint thread sends the read hint before the request enters the input queue, so the device prefetches while the request waits (median 14.3 ms in the serving replay). Mean per-request increase in KV load time over a paired run with all KV in device DRAM: +41 ms without read hints, +1.2 ms with them. Path (b) is opt-in (
maru_read_hint_on_arrival) and these numbers used it; the default path is compared in the measurement paragraph below. The memory node strip is simplified: a hint reaches the device through the remote CXL agent and the region extension.What goes wrong without this PR, on a CXL-SSD used as
remote_cxlmemory:Serving replay: one CXL-SSD, vLLM 0.22.1, Llama-3.1-8B, 48 conversations × 6 turns, 33.6 GiB of KV behind a 16 GiB device DRAM cache. Metric: mean TTFT of the turns that reuse earlier KV, relative to the same path with all KV in device DRAM. With this PR and
maru_read_hint_on_arrival(path (b) in Figure 1): 1.004× [0.993, 1.015] (5 pairs). With the default path, where the read hint leaves when the scheduler first looks the request up: 1.005× [0.979, 1.031] (3 pairs), the same within the intervals; the device's prefetch rate (measured on an idle device) was close to the RDMA READ rate of the 100GbE link, so a prefetch that starts with the load still kept ahead of the READs. No run without the PR was made on this build. Brackets are 95% intervals over paired runs.Replaces #124, which reached the same TTFT (1.004×) but made every RDMA READ ask the memory node first: a round trip per READ, with device policy in the data path.
🏗️ Design Changes
Before: the memory node hears only which regions are mapped.
flowchart LR subgraph gpuB["GPU node"] mB["Maru (vLLM scheduler + worker)"] end subgraph memB["memory node"] agB["remote CXL agent"] devB[("CXL-SSD")] end mB -->|"map regions"| agB mB ==>|"RDMA READ / WRITE"| devBAfter: Maru sends prefetch hints (read and write) to the agent, which checks them and hands them to the region extension (the agent's hook, implemented by a package installed on the memory node) that drives the device. Thick line: KV bytes; solid: control messages; dotted: device commands.
flowchart LR subgraph gpuA["GPU node"] mA["Maru (vLLM scheduler + worker)"] end subgraph memA["memory node"] agA["remote CXL agent<br/>(checks every hinted range)"] extA["region extension<br/>(separate package)"] devA[("CXL-SSD")] end mA -->|"map regions, read and write hints"| agA agA -->|"checked hints, region maps"| extA extA -.->|"prefetch, write-back,<br/>memory generation"| devA mA ==>|"RDMA READ / WRITE (unchanged)"| devA classDef pkg fill:#fbe3cc,stroke:#c06000,color:#111 class extA pkg📝 Implementation Details
Review in three parts; each part passes the unit tests on its own. The full design is in
docs/source/design_doc/read_hints.md.Please look at:
maru_read_hint_on_arrivalwraps vLLM'sEngineCore.preprocess_add_request, because the connector API is first called at the next step boundary. It is installed only if the method takes(self, request); otherwise hints fall back to the scheduler's first lookup, with a warning. Checked against vLLM 0.22.1 and 0.32.0 sources; run only on 0.22.1. On the measured 100GbE link it gave the same TTFT as the default path (above); a link faster than the device's prefetch, where the earlier start is expected to shorten the load, was not measured.probewithlocationsgives the scheduler the read handles of its hits without a lease; the hint only names them, and every READ still leases.retireanswers False), areleaseof it errors and a new map of it is refused; agent shutdown waits up to 30 s for the extension.maru_write_hintsfollowsmaru_read_hints, so turning on read hints also turns on store waits: up to 50 ms (maru_write_wait_ms), plus the reply timeout if the agent does not answer. Stores run on the write-behind thread (on by default forremote_cxl).✅ Tests
🔗 Related Issues (optional)
🌿 Related PRs (optional)
🌿 Related Branches (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
FIXED
IMPORTANT NOTES