Skip to content

feat(handler): add read hint callbacks for local CXL plugins - #137

Merged
youngrok-XCENA merged 27 commits into
mainfrom
feat/cxl-read-hints
Nov 19, 2026
Merged

youngrok-XCENA merged 27 commits into
mainfrom
feat/cxl-read-hints

Conversation

@youngrok-XCENA

@youngrok-XCENA youngrok-XCENA commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

Let a local CXL handler plugin prepare stored KV before a request loads it. Previously, read hints were accepted only by the remote CXL backend. A local plugin's retrieval hook ran when loading had already begun.

The scheduler identifies the KV a request will reuse. The worker loads that KV into GPU memory. This PR delivers the scheduler's early read hint and the worker's load-start notification to an independently installed plugin.

The scheduler's read hints, which remote_cxl already sent, also had a defect: they named whole-chunk keys, while workers store one object per TP slice, per layer in layerwise format, and per KV cache group. Unless TP=1, the format is chunkwise and there is one KV cache group, the hint's keys matched no stored object and no hint reached storage. The hints this PR gives to local plugins are built the fixed way.

🏗️ Design Changes

%%{init: {"themeVariables": {"fontSize": "13px"}, "flowchart": {"nodeSpacing": 16, "rankSpacing": 24}}}%%
flowchart LR
    S[Scheduler] -->|read_hint · end| P[Plugin]
    W[Worker] -->|read_promote| P
    K[(CXL KV)] ==>|load| W
Loading

Figure 1. Thin arrows are the added plugin notifications; the thick arrow is the existing KV load. Previously, the retrieval hook notified the plugin only at load time; the scheduler's hint now provides an earlier preparation opportunity. Handlers and queues are omitted. Delivery is asynchronous, so promote may precede hint. The end notification follows load completion, an earlier request finish, or handler close unless close times out.

📝 Implementation Details

  • Allow read_hints and read_promotes on the cxl backend and notify the plugin when a load of a hinted request starts, for models with one or several KV cache groups.
  • Plugins opt in through hint_actions and on_memory_hint(handler, message). Messages carry the request ID and, for a read hint, the leading stored byte ranges.
  • Reuse the background hint queue and isolate callback exceptions. Enabling hints without an accepting plugin logs a warning and sends no hints.
  • The vLLM scheduler hints the stored objects its workers load (each TP slice, each layer in layerwise format, each needed KV cache group object), chunk by chunk, using the workers' load rule. It builds the keys on its own hint thread, which also sends the read_hint_end notifications, so an end never overtakes its hint.
  • Closing a cxl handler sends the queued hints, then read_hint_end for every open request, then calls plugin on_close, including on a metadata_only handler (the one a vLLM scheduler uses). Previously that handler returned before on_close and sent no end for open requests. If the queued hints take longer than 5 s, every message still queued is dropped, the ends of open requests included, and close waits without a time limit for the hint thread's call under way (a metadata lookup or a plugin callback), so no hint reaches a plugin after on_close. Plugin callbacks must return promptly.
  • Plugin implementations and dependencies remain in separate packages. Hints are advisory; KV transfer and ownership follow the existing paths.
  • Write hints and memory-generation handling remain on the remote_cxl backend.

✅ Tests

  • Unit tests: 918 passed, 199 skipped, 1 deselected on this branch, using the command below.
  • ruff check ., ruff format --check ., git diff --check.
  • Integration tests
  • Manual tests
  • No tests needed (reason: not applicable)
CUDA_VISIBLE_DEVICES= python -m pytest -q \
  tests/unit/test_cxl_read_hints.py tests/unit/test_plugin_loader.py \
  tests/unit/test_vllm*.py \
  --deselect=tests/unit/test_vllm_remote_cxl_tp_groups.py::test_the_scheduler_looks_up_in_the_workers_namespace

The deselected test also fails on unchanged main because the installed vLLM 0.22.1 lacks its required API. Validation used CPU tensors and fake components; it does not establish device performance.

🔗 Related Issues (optional)

  • None.

🌿 Related PRs (optional)

  • None.

🌿 Related Branches (optional)

  • None required.

📦 Release Note (for auto-generation / write in English)

NEW

  • Optional read hint callbacks for local CXL handler plugins.

CHANGED

  • CXL handlers accept read hints and read promotes.

FIXED

  • vLLM scheduler read hints previously named whole-chunk keys and reached storage only with TP=1, chunkwise format and one KV cache group; they now name each TP slice, layer and KV cache group object the workers load.

IMPORTANT NOTES

  • Receivers must tolerate a promote arriving before its hint.
  • When a handler's close passes the 5 s hint timeout, on_close may follow open requests that got no read_hint_end; release per-request state in on_close.

A remote CXL agent can load one out-of-tree region extension, registered
under the maru.region_extensions entry-point group and named with
--region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a
package that provides one can turn it on for the installed service with
a drop-in). It serves memory that is slow until a device has brought a
page in (a CXL-SSD with a DRAM cache in front of its flash):

- validate_access may refuse a region before it is mapped, mapped hears
  it once the NIC has it (raising keeps the region out of use), and
  retire is asked before a region goes back; while it says no, the
  region stays here unregistered and the sweep asks again.
- hint takes read_hint / read_hint_end messages (op "extension") from
  any authenticated connection, once the agent has checked the epoch, the
  action and that every range lies in a region it maps and some
  connection holds, under a matching handle. The check runs under the
  agent's lock and the extension is called after it, so no other request
  waits for the extension. A connection may keep at most
  --max-hint-requests (4096) hint requests open.
- The optional connection_closed(conn) is called for every connection
  that closes, so the extension can end what its hints started.
- The hello advertises region_extension {"version": 1, "actions": [...]}
  and RegionMapClient keeps the hint actions it advertises; anything else
  means no hints.

An agent that fails to load its extension exits. Without one, nothing
changes.
… bytes early

A serving engine knows which stored keys a request will load well before
its worker starts the load. Read hints carry that to the memory side,
where a region extension can bring the bytes into a faster tier. Maru's
KV data path stays as it is: a hint is advice that never gates a READ or
WRITE, and one that is late or lost costs only the time it would have
saved.

- read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on.
  MaruHandler.hint_read / end_read_hint queue a hint and its end; one
  thread per backend probes maru-server for where the hits are (a probe
  with locations, which takes no lease) and sends read_hint and
  read_hint_end to every agent whose hello advertises them, in messages
  of at most 4096 ranges. At most 1024 hints wait; ends are never
  dropped. The thread keeps at most 4096 hinted requests open and ends
  the oldest past that.
- A client that loads with read_hints on hints each load as its READs
  start, under a name of its own, and ends that hint after the load
  (for engines that name no request to their storage).
- The vLLM scheduler connector hints a request when it first finds its
  asynchronous load, or on arrival with maru_read_hint_on_arrival, and
  ends the hint when vLLM reports the load done (or the request
  finishes first). Only the scheduler gets read_hints, so vLLM loads are
  not hinted twice.
- The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since
  vLLM's connector API is first called at the next step boundary. It is
  installed only if the method takes (self, request) and passes on only
  a result whose first item is a request with an id; otherwise hints on
  arrival are off with a warning.
A region extension may front memory whose contents a device keeps in a
write-back cache (a CXL-SSD). Two things about such memory reach it now.

Fresh regions. A region its owner allocated in this session holds no KV
until the owner writes to it, and the owner writes only after a
successful map reply. Until the first one, its maps carry fresh=true,
retries included; after it (an agent restart, a remap, a region given
back) they do not. The agent takes fresh only with the region's own
handle and only on the map that maps the region there first, and passes
it to RegionExtension.mapped(fresh=) before the reply leaves, so a
device can drop the region's stale contents before any WRITE reaches
it. fresh is the owner's word, to be used only on the region's own
bytes. An agent that does not know fresh ignores it.

Memory generations. A write-back cache loses what it held on a reset
without a reboot of its node, and Maru already treats a changed agent
boot as a reboot: owners give their regions back and readers load
misses. An extension may tell the memory's generation (optional
memory_generation(), local state, outside the agent's lock), and the
agent advertises "<node boot id>:<generation>" as its boot.

- The agent reads the generation at start (a failure stops it), on a
  thread every --memory-generation-poll-s (100 ms) and on every hello,
  map and sweep, one caller at a time. When it changes, every mapping is
  deregistered from the NIC, so RDMA through an old registration fails,
  and dropped as a release does. Map replies now carry the boot, and a
  worker takes it over its connection's.
- A generation the extension does not tell for
  --memory-generation-grace-s (2 s) is unknown: every mapping goes, the
  boot reads <boot id>:unknown and maps are refused until it answers.
  The boot after the n-th such period is <boot id>:<generation>~<n>, so
  nothing mapped before one is trusted after it.
- A map whose boot changed while it ran fails and its registration goes.
…'s pages

Without them, a store's RDMA WRITEs into memory with a slow tier behind a
cache make the device fill each written page from the slow tier first,
page by page. On a CXL-SSD, a build that stored without write hints wrote
at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with
the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other
settings differed too).

- write_hints (vLLM: maru_write_hints, which follows maru_read_hints
  unless set) makes a store, once it has its pages and before it posts
  the WRITEs, queue a write_hint on the hint thread: the pages, in write
  order, under a token for the store. Fire and forget; no end. With vLLM
  only the worker sends them (it stores; the scheduler hints loads).
- The agent takes write_hint with the read hints' checks; it keeps no
  request open for it. A region extension says which hint actions it
  takes (optional hint_actions; read_hint and read_hint_end without it),
  the hello advertises those, Maru sends only those and the agent
  refuses others. A write hint gives no leave to change the pages.
- The hint thread class is MemoryHints now that it sends both kinds.
The memory side brings hinted requests' bytes in in hint order, which is
not the order their loads run in, so loads read chunks still being
brought in while others' wait ready. A worker now tells the memory side
when the load of a request starts.

- read_promotes (vLLM: on in the worker with maru_read_hints and
  maru_async_load) makes MaruHandler.promote_read(request_id) queue a
  read_promote on the hint thread: no ranges, no end, fire and forget.
  The vLLM worker calls it once per remote_cxl load, before its first
  batch is read.
- The worker did not send the request's hint (its scheduler did, on
  another connection), so it sends the promote to every agent of the
  pool whose extension takes it; the extension matches it to a hint by
  request id alone.
- The agent takes read_promote like read_hint_end and keeps no request
  open for it; it advertises and accepts it only if the region extension
  lists it in hint_actions.
A queued write hint reaches the memory side about when the store's WRITEs
do, so most WRITEs still landed on pages the device had to fill from its
slow tier first. Stores run off the request path (vLLM's write-behind
thread), so they can afford a short wait.

- When an agent's region extension takes write_wait, a store sends its
  pages for that agent in a write_wait (the store's token, the ranges in
  write order, timeout_ms) and waits for the answer, then posts the
  WRITEs whatever it is. The waits of one store share write_wait_ms
  (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an
  agent that does not answer costs the reply timeout on top. Other agents
  still get a queued write hint. Waits are off unless write hints are on.
- Waits live in waits.py (MemoryWaits): each thread has its own
  connection per agent, so one store's wait never holds up another's, a
  hint or a region map. An agent that cannot be reached is not asked
  again for 5 s; its stores send write hints meanwhile.
- The agent checks a write_wait like a write hint, plus timeout_ms (1 to
  10000), keeps no request open for it, and calls the extension outside
  its lock on the requesting connection's own thread; the extension's
  answer (a map, or None) is the reply. Only that connection waits.
  It is advertised and accepted only if the extension lists it in
  hint_actions.
…wait backs off

Hints and waits are advice, but the store's write preparation and the
load's read hint ran after the transfer count went up and before the try
that brings it down. If either raised, the transfer never ended: the
pages stayed sending, the store failed, and close() waited for the RDMA
timeout. Both now run inside that try, and any exception from hint code
is logged and ignored.

One way they raised: the hint connections read the agent key lazily, so
a key file changed after start made load_key raise. They now reuse the
key the data path read at start. MemoryWaits also catches a failing
connect and backs off instead of raising.

A write wait that fails or times out now backs the agent off for
UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do
not each pay the reply timeout.
hello() reads boot without the lock, and boot was built from three
fields the generation watch changes one by one. When the generation
became known again, a concurrent hello could advertise the boot from
before the unknown period. The boot is now one string, rebuilt under
the lock whenever the generation state changes (known again, changed,
unknown), so a reader sees the boot from before or after a change,
never a mix.
…he extension closed on a failed start

- MAX_HINT_RANGES moves to maru_remote.region_map beside
  MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the
  write_wait_ms config check all use these two, so both sides agree.
- --max-hint-requests and --memory-generation-poll-s must be above zero
  (zero refused every read hint or polled in a busy loop).
- When RegionMapService fails to start, the loaded region extension is
  closed before the agent exits.
…new handler

shutdown() closed the scheduler's MaruHandler but left the arrival-hint
pool running, so a request that arrived afterwards could create a new
handler through _ensure_handler(). The scheduler connector now has its
own shutdown(): it marks itself shut down, stops the arrival pool
(waiting for a hint under way), then closes the handler. Arrival hints
are submitted under the hint lock, so none lands on a stopped pool.
…er plugins

A handler of the cxl backend (direct mapping) now takes read_hints and
read_promotes. It gives the hint messages remote_cxl sends to a remote CXL
agent's region extension (read_hint with the leading hit keys' bytes,
read_hint_end, read_promote), in the same form, to the handler plugins whose
hint_actions name them, through the new on_memory_hint hook, on one hint
thread. The vLLM connector lets maru_read_hints and
maru_read_hint_on_arrival reach the cxl backend (write hints stay with
remote_cxl) and promotes each deferred cxl load as it starts.
youngrok-XCENA added a commit that referenced this pull request Nov 5, 2026
MemoryHints takes its locate and send callables from the backend and holds
the background thread, the queue, the drop policy and the end bookkeeping,
none of which depends on RDMA or agents. It now lives in
maru_handler/hints.py, so another backend (the cxl backend of #137) uses it
without depending on the remote_cxl package. The shared wire limits stay in
maru_remote.region_map, which maru_common already imports.
youngrok-XCENA added a commit that referenced this pull request Nov 6, 2026
…tension, so the memory node can prepare KV before transfers (#130)

* feat(remote_cxl_agent): region extensions with a checked read hint relay

A remote CXL agent can load one out-of-tree region extension, registered
under the maru.region_extensions entry-point group and named with
--region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a
package that provides one can turn it on for the installed service with
a drop-in). It serves memory that is slow until a device has brought a
page in (a CXL-SSD with a DRAM cache in front of its flash):

- validate_access may refuse a region before it is mapped, mapped hears
  it once the NIC has it (raising keeps the region out of use), and
  retire is asked before a region goes back; while it says no, the
  region stays here unregistered and the sweep asks again.
- hint takes read_hint / read_hint_end messages (op "extension") from
  any authenticated connection, once the agent has checked the epoch, the
  action and that every range lies in a region it maps and some
  connection holds, under a matching handle. The check runs under the
  agent's lock and the extension is called after it, so no other request
  waits for the extension. A connection may keep at most
  --max-hint-requests (4096) hint requests open.
- The optional connection_closed(conn) is called for every connection
  that closes, so the extension can end what its hints started.
- The hello advertises region_extension {"version": 1, "actions": [...]}
  and RegionMapClient keeps the hint actions it advertises; anything else
  means no hints.

An agent that fails to load its extension exits. Without one, nothing
changes.

* feat(remote_cxl): read hints, so the memory side can prepare a load's bytes early

A serving engine knows which stored keys a request will load well before
its worker starts the load. Read hints carry that to the memory side,
where a region extension can bring the bytes into a faster tier. Maru's
KV data path stays as it is: a hint is advice that never gates a READ or
WRITE, and one that is late or lost costs only the time it would have
saved.

- read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on.
  MaruHandler.hint_read / end_read_hint queue a hint and its end; one
  thread per backend probes maru-server for where the hits are (a probe
  with locations, which takes no lease) and sends read_hint and
  read_hint_end to every agent whose hello advertises them, in messages
  of at most 4096 ranges. At most 1024 hints wait; ends are never
  dropped. The thread keeps at most 4096 hinted requests open and ends
  the oldest past that.
- A client that loads with read_hints on hints each load as its READs
  start, under a name of its own, and ends that hint after the load
  (for engines that name no request to their storage).
- The vLLM scheduler connector hints a request when it first finds its
  asynchronous load, or on arrival with maru_read_hint_on_arrival, and
  ends the hint when vLLM reports the load done (or the request
  finishes first). Only the scheduler gets read_hints, so vLLM loads are
  not hinted twice.
- The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since
  vLLM's connector API is first called at the next step boundary. It is
  installed only if the method takes (self, request) and passes on only
  a result whose first item is a request with an id; otherwise hints on
  arrival are off with a warning.

* docs: read hints design and configuration

* feat(remote_cxl_agent): fresh region maps and memory generations

A region extension may front memory whose contents a device keeps in a
write-back cache (a CXL-SSD). Two things about such memory reach it now.

Fresh regions. A region its owner allocated in this session holds no KV
until the owner writes to it, and the owner writes only after a
successful map reply. Until the first one, its maps carry fresh=true,
retries included; after it (an agent restart, a remap, a region given
back) they do not. The agent takes fresh only with the region's own
handle and only on the map that maps the region there first, and passes
it to RegionExtension.mapped(fresh=) before the reply leaves, so a
device can drop the region's stale contents before any WRITE reaches
it. fresh is the owner's word, to be used only on the region's own
bytes. An agent that does not know fresh ignores it.

Memory generations. A write-back cache loses what it held on a reset
without a reboot of its node, and Maru already treats a changed agent
boot as a reboot: owners give their regions back and readers load
misses. An extension may tell the memory's generation (optional
memory_generation(), local state, outside the agent's lock), and the
agent advertises "<node boot id>:<generation>" as its boot.

- The agent reads the generation at start (a failure stops it), on a
  thread every --memory-generation-poll-s (100 ms) and on every hello,
  map and sweep, one caller at a time. When it changes, every mapping is
  deregistered from the NIC, so RDMA through an old registration fails,
  and dropped as a release does. Map replies now carry the boot, and a
  worker takes it over its connection's.
- A generation the extension does not tell for
  --memory-generation-grace-s (2 s) is unknown: every mapping goes, the
  boot reads <boot id>:unknown and maps are refused until it answers.
  The boot after the n-th such period is <boot id>:<generation>~<n>, so
  nothing mapped before one is trusted after it.
- A map whose boot changed while it ran fails and its registration goes.

* docs: fresh region maps and memory resets

* feat(remote_cxl): write hints, so the memory side can prepare a store's pages

Without them, a store's RDMA WRITEs into memory with a slow tier behind a
cache make the device fill each written page from the slow tier first,
page by page. On a CXL-SSD, a build that stored without write hints wrote
at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with
the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other
settings differed too).

- write_hints (vLLM: maru_write_hints, which follows maru_read_hints
  unless set) makes a store, once it has its pages and before it posts
  the WRITEs, queue a write_hint on the hint thread: the pages, in write
  order, under a token for the store. Fire and forget; no end. With vLLM
  only the worker sends them (it stores; the scheduler hints loads).
- The agent takes write_hint with the read hints' checks; it keeps no
  request open for it. A region extension says which hint actions it
  takes (optional hint_actions; read_hint and read_hint_end without it),
  the hello advertises those, Maru sends only those and the agent
  refuses others. A write hint gives no leave to change the pages.
- The hint thread class is MemoryHints now that it sends both kinds.

* feat(remote_cxl): promote a hinted load when it starts

The memory side brings hinted requests' bytes in in hint order, which is
not the order their loads run in, so loads read chunks still being
brought in while others' wait ready. A worker now tells the memory side
when the load of a request starts.

- read_promotes (vLLM: on in the worker with maru_read_hints and
  maru_async_load) makes MaruHandler.promote_read(request_id) queue a
  read_promote on the hint thread: no ranges, no end, fire and forget.
  The vLLM worker calls it once per remote_cxl load, before its first
  batch is read.
- The worker did not send the request's hint (its scheduler did, on
  another connection), so it sends the promote to every agent of the
  pool whose extension takes it; the extension matches it to a hint by
  request id alone.
- The agent takes read_promote like read_hint_end and keeps no request
  open for it; it advertises and accepts it only if the region extension
  lists it in hint_actions.

* feat(remote_cxl): a store waits briefly for its pages to be prepared

A queued write hint reaches the memory side about when the store's WRITEs
do, so most WRITEs still landed on pages the device had to fill from its
slow tier first. Stores run off the request path (vLLM's write-behind
thread), so they can afford a short wait.

- When an agent's region extension takes write_wait, a store sends its
  pages for that agent in a write_wait (the store's token, the ranges in
  write order, timeout_ms) and waits for the answer, then posts the
  WRITEs whatever it is. The waits of one store share write_wait_ms
  (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an
  agent that does not answer costs the reply timeout on top. Other agents
  still get a queued write hint. Waits are off unless write hints are on.
- Waits live in waits.py (MemoryWaits): each thread has its own
  connection per agent, so one store's wait never holds up another's, a
  hint or a region map. An agent that cannot be reached is not asked
  again for 5 s; its stores send write hints meanwhile.
- The agent checks a write_wait like a write hint, plus timeout_ms (1 to
  10000), keeps no request open for it, and calls the extension outside
  its lock on the requesting connection's own thread; the extension's
  answer (a map, or None) is the reply. Only that connection waits.
  It is advertised and accepted only if the extension lists it in
  hint_actions.

* docs: write hints, store waits and read promotes

* docs: call read and write hints prefetch hints, and show where a read hint leaves vLLM

* docs: shorter panel titles in the prefetch hint figure; say which thread sends the hint

* fix(remote_cxl): hint code never fails a store or load, and a failed wait backs off

Hints and waits are advice, but the store's write preparation and the
load's read hint ran after the transfer count went up and before the try
that brings it down. If either raised, the transfer never ended: the
pages stayed sending, the store failed, and close() waited for the RDMA
timeout. Both now run inside that try, and any exception from hint code
is logged and ignored.

One way they raised: the hint connections read the agent key lazily, so
a key file changed after start made load_key raise. They now reuse the
key the data path read at start. MemoryWaits also catches a failing
connect and backs off instead of raising.

A write wait that fails or times out now backs the agent off for
UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do
not each pay the reply timeout.

* fix(remote_cxl_agent): a hello never advertises a half-changed boot

hello() reads boot without the lock, and boot was built from three
fields the generation watch changes one by one. When the generation
became known again, a concurrent hello could advertise the boot from
before the unknown period. The boot is now one string, rebuilt under
the lock whenever the generation state changes (known again, changed,
unknown), so a reader sees the boot from before or after a change,
never a mix.

* fix(remote_cxl_agent): shared hint limits, positive CLI values, and the extension closed on a failed start

- MAX_HINT_RANGES moves to maru_remote.region_map beside
  MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the
  write_wait_ms config check all use these two, so both sides agree.
- --max-hint-requests and --memory-generation-poll-s must be above zero
  (zero refused every read hint or polled in a busy loop).
- When RegionMapService fails to start, the loaded region extension is
  closed before the agent exits.

* fix(vllm): scheduler shutdown stops the arrival hints and creates no new handler

shutdown() closed the scheduler's MaruHandler but left the arrival-hint
pool running, so a request that arrived afterwards could create a new
handler through _ensure_handler(). The scheduler connector now has its
own shutdown(): it marks itself shut down, stops the arrival pool
(waiting for a hint under way), then closes the handler. Arrival hints
are submitted under the hint lock, so none lands on a stopped pool.

* fix(remote_cxl_agent): a map never succeeds while the memory generation is unknown

map() checks the generation before mapping, and _establish compares the
boot it read at the start with the boot at the end. If the generation
turned unknown between the check and the first boot read, both boots read
":unknown", the comparison passed, and the map succeeded. Its registration
then outlived the unknown period: the sweep kept it and the recovery
released nothing, so the owner could RDMA into memory whose state was
unknown.

_establish now also refuses a map whose generation is unknown at the end.
A boot that is known at the end already differs from an unknown one read
at the start, so that case was covered.

* fix(vllm): a lookup hint goes to the pool its load reads

With several remote_cxl pools, the scheduler's lookup hint was routed by
the first key it named. When vLLM's own prefix cache held a chunk or more,
that key was not the request's first chunk key, which routes the load, so
the hint went to another pool and was dropped there. The hint is now
routed by the request's first chunk key, as the load and the arrival hint
are.

* refactor(handler): MemoryHints moves out of the remote_cxl package

MemoryHints takes its locate and send callables from the backend and holds
the background thread, the queue, the drop policy and the end bookkeeping,
none of which depends on RDMA or agents. It now lives in
maru_handler/hints.py, so another backend (the cxl backend of #137) uses it
without depending on the remote_cxl package. The shared wire limits stay in
maru_remote.region_map, which maru_common already imports.
Base automatically changed from feat/remote-cxl-read-hints to main November 6, 2026 15:44
@youngrok-XCENA youngrok-XCENA changed the title feat(handler): read hints of the cxl backend go to handler plugins, so a device plugin can prefetch before the load feat(maru_vllm): add local read hints and complete prefix-cache stores Nov 11, 2026
@youngrok-XCENA youngrok-XCENA changed the title feat(maru_vllm): add local read hints and complete prefix-cache stores feat(handler): add read hint callbacks for local CXL plugins Nov 11, 2026
@youngrok-XCENA
youngrok-XCENA marked this pull request as ready for review November 11, 2026 15:35
@seohui-XCENA

Copy link
Copy Markdown
Collaborator

Reviewed the current head, 1f58cc7. I found two issues in the local CXL hint path:

  1. [P2] Read hints look up bare chunk keys even when the stored KV objects have TP or layer suffixes.

    In _local_hint_locate(), the hint keys go directly to batch_lookup_kv(). The scheduler's hit check accounts for TP slice suffixes and layerwise completion markers, but the hint still contains bare chunk keys.

    For example, TP stores objects such as kv_…_h0-4 and kv_…_h4-8, while layerwise storage uses kv_…_L0, kv_…_L1, etc. Looking up kv_… misses in both cases, and the leading-miss check drops the hint. I reproduced both cases: the scheduler reported a cache hit and selected an asynchronous load, but no read hint reached the plugin.

    Could the hint keys be expanded to the actual objects the workers will load? For layerwise storage, the hint needs the KV objects' ranges rather than the _DONE marker.

  2. [P2] A scheduler's hint plugin receives no cleanup callback when its handler closes.

    In close(), the local hint queue is closed, then the metadata_only branch returns before plugin on_close() dispatch. Scheduler handlers are metadata-only. Closing MemoryHints also does not send read_hint_end for every request still open.

    If the scheduler shuts down with a hinted request outstanding, a plugin that keeps pins, reservations, or external service state for that request can therefore miss its cleanup notification. I reproduced a connected metadata-only handler delivering a read hint, then closing without invoking the plugin's on_close().

    Could metadata-only hint receivers get a cleanup callback too, with an explicit policy for ending outstanding hints during shutdown?

Validation: the existing local hint, plugin loader, and test_vllm*.py tests passed locally (851 passed, 105 skipped), as did Ruff on the changed implementation files and git diff --check. Three additional regression cases failed at the expected assertions: TP hint delivery, layerwise hint delivery, and scheduler plugin cleanup. These checks used Python 3.12, vLLM 0.23.0, CPU tensors, and fake components; they do not measure device performance.

The scheduler hinted bare chunk keys, while TP workers store one object
per KV slice (and, layerwise, per layer; with several KV cache groups,
per group). The lookup of those keys missed, so a cxl hint plugin got no
read hint outside TP=1 chunkwise with one group. The scheduler now names
the objects the workers load, chunk by chunk, from the same load rule
the worker uses (_group_load_items), and builds the keys on one hint
thread that also carries the ends, so the scheduler's path stays short
and an end never passes its hint.

A metadata-only handler (a vLLM scheduler's) returned from close()
before its plugins' on_close, and closing its hint queue sent no end for
the requests still open. close() now ends every open hint, then calls
on_close while the RPC connection is live.
@youngrok-XCENA

Copy link
Copy Markdown
Collaborator Author

@seohui-XCENA Replying to your review (#137 (comment)). Both are fixed in 6427d2f.

  1. [P2] Read hints look up bare chunk keys even when the stored KV objects have TP or layer suffixes.

The scheduler now hints the objects the workers load instead of bare chunk keys: every TP slice of a chunk, each layer's object in layerwise format (not the _DONE marker), and, with several KV cache groups, the group objects the hit needs. The worker's load rule moved into one helper that both sides use, so the two lists cannot drift. Keys go chunk by chunk, so an arrival hint still stops at the first chunk with an object missing. Building the keys can take a few milliseconds at high TP in layerwise format, so it runs on one connector hint thread that also sends the ends, which keeps the scheduler's path short and an end behind its hint. The remote_cxl scheduler hint shares this path and is fixed too.

  1. [P2] A scheduler's hint plugin receives no cleanup callback when its handler closes.

On close, a cxl handler now sends the queued hints, then read_hint_end for every request still open, then calls the plugins' on_close while the RPC connection is live, including on a metadata-only (scheduler) handler. Remote agents keep ending a closed connection's requests themselves, so their behavior is unchanged.

Your three cases are now regression tests (TP and layerwise hint delivery through a real MaruHandler and plugin, and the close order), plus a multi-group key test. On the previous head, only TP=1 chunkwise passes. Could you take another look?

# Conflicts:
#	docs/source/api_reference/plugins.md
#	docs/source/design_doc/read_hints.md

@seohui-XCENA seohui-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The hint shutdown ordering needs to be addressed before merging. I reproduced callbacks running after on_close when the hint thread exceeds its shutdown timeout.

Separately, the multi-group CXL load path does not emit read_promote. This can be addressed here or documented as a limitation for follow-up.

The existing relevant tests passed: 325 passed, 27 skipped. Additional targeted tests reproduced both issues.

Comment thread maru_handler/handler.py Resolved
Comment thread maru_vllm/connector.py Resolved
- MemoryHints.close() returns False when its thread is still busy past
  the timeout, and MaruHandler.close() went on to on_close regardless,
  so a hint or end could reach a plugin after on_close. Past the timeout
  the handler now abandons the queue (the messages still queued are
  dropped) and waits for the plugin call under way before on_close.
- _group_load_job (several KV cache groups) sent no read_promote; it now
  promotes the hinted request before its first read, as the one-group
  load does.

@seohui-XCENA seohui-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified the fixes at 9c63aae. Both review findings are addressed:

  • Handler shutdown waits for the hint thread to exit before plugin teardown, with the timeout/drop behavior documented.
  • Multi-group loads now emit read_promote before their first read.

Local validation: 360 passed, 27 skipped across the relevant hint, plugin, and connector tests, including the added regression cases. This validation used CPU tensors and mocked components; it does not establish GPU/CXL device performance.

At approval time, Python 3.13/3.14, Resource Manager, lint, and Jenkins checks have passed. The Python 3.12 rerun is still in progress following the earlier Nexus dependency-download timeout.

@youngrok-XCENA
youngrok-XCENA merged commit ca23e6a into main Nov 19, 2026
14 of 18 checks passed
@youngrok-XCENA
youngrok-XCENA deleted the feat/cxl-read-hints branch November 19, 2026 08:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants