Repository navigation
feat(handler): add read hint callbacks for local CXL plugins - #137
Conversation
A remote CXL agent can load one out-of-tree region extension, registered
under the maru.region_extensions entry-point group and named with
--region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a
package that provides one can turn it on for the installed service with
a drop-in). It serves memory that is slow until a device has brought a
page in (a CXL-SSD with a DRAM cache in front of its flash):
- validate_access may refuse a region before it is mapped, mapped hears
it once the NIC has it (raising keeps the region out of use), and
retire is asked before a region goes back; while it says no, the
region stays here unregistered and the sweep asks again.
- hint takes read_hint / read_hint_end messages (op "extension") from
any authenticated connection, once the agent has checked the epoch, the
action and that every range lies in a region it maps and some
connection holds, under a matching handle. The check runs under the
agent's lock and the extension is called after it, so no other request
waits for the extension. A connection may keep at most
--max-hint-requests (4096) hint requests open.
- The optional connection_closed(conn) is called for every connection
that closes, so the extension can end what its hints started.
- The hello advertises region_extension {"version": 1, "actions": [...]}
and RegionMapClient keeps the hint actions it advertises; anything else
means no hints.
An agent that fails to load its extension exits. Without one, nothing
changes.
… bytes early A serving engine knows which stored keys a request will load well before its worker starts the load. Read hints carry that to the memory side, where a region extension can bring the bytes into a faster tier. Maru's KV data path stays as it is: a hint is advice that never gates a READ or WRITE, and one that is late or lost costs only the time it would have saved. - read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on. MaruHandler.hint_read / end_read_hint queue a hint and its end; one thread per backend probes maru-server for where the hits are (a probe with locations, which takes no lease) and sends read_hint and read_hint_end to every agent whose hello advertises them, in messages of at most 4096 ranges. At most 1024 hints wait; ends are never dropped. The thread keeps at most 4096 hinted requests open and ends the oldest past that. - A client that loads with read_hints on hints each load as its READs start, under a name of its own, and ends that hint after the load (for engines that name no request to their storage). - The vLLM scheduler connector hints a request when it first finds its asynchronous load, or on arrival with maru_read_hint_on_arrival, and ends the hint when vLLM reports the load done (or the request finishes first). Only the scheduler gets read_hints, so vLLM loads are not hinted twice. - The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since vLLM's connector API is first called at the next step boundary. It is installed only if the method takes (self, request) and passes on only a result whose first item is a request with an id; otherwise hints on arrival are off with a warning.
A region extension may front memory whose contents a device keeps in a write-back cache (a CXL-SSD). Two things about such memory reach it now. Fresh regions. A region its owner allocated in this session holds no KV until the owner writes to it, and the owner writes only after a successful map reply. Until the first one, its maps carry fresh=true, retries included; after it (an agent restart, a remap, a region given back) they do not. The agent takes fresh only with the region's own handle and only on the map that maps the region there first, and passes it to RegionExtension.mapped(fresh=) before the reply leaves, so a device can drop the region's stale contents before any WRITE reaches it. fresh is the owner's word, to be used only on the region's own bytes. An agent that does not know fresh ignores it. Memory generations. A write-back cache loses what it held on a reset without a reboot of its node, and Maru already treats a changed agent boot as a reboot: owners give their regions back and readers load misses. An extension may tell the memory's generation (optional memory_generation(), local state, outside the agent's lock), and the agent advertises "<node boot id>:<generation>" as its boot. - The agent reads the generation at start (a failure stops it), on a thread every --memory-generation-poll-s (100 ms) and on every hello, map and sweep, one caller at a time. When it changes, every mapping is deregistered from the NIC, so RDMA through an old registration fails, and dropped as a release does. Map replies now carry the boot, and a worker takes it over its connection's. - A generation the extension does not tell for --memory-generation-grace-s (2 s) is unknown: every mapping goes, the boot reads <boot id>:unknown and maps are refused until it answers. The boot after the n-th such period is <boot id>:<generation>~<n>, so nothing mapped before one is trusted after it. - A map whose boot changed while it ran fails and its registration goes.
…'s pages Without them, a store's RDMA WRITEs into memory with a slow tier behind a cache make the device fill each written page from the slow tier first, page by page. On a CXL-SSD, a build that stored without write hints wrote at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other settings differed too). - write_hints (vLLM: maru_write_hints, which follows maru_read_hints unless set) makes a store, once it has its pages and before it posts the WRITEs, queue a write_hint on the hint thread: the pages, in write order, under a token for the store. Fire and forget; no end. With vLLM only the worker sends them (it stores; the scheduler hints loads). - The agent takes write_hint with the read hints' checks; it keeps no request open for it. A region extension says which hint actions it takes (optional hint_actions; read_hint and read_hint_end without it), the hello advertises those, Maru sends only those and the agent refuses others. A write hint gives no leave to change the pages. - The hint thread class is MemoryHints now that it sends both kinds.
The memory side brings hinted requests' bytes in in hint order, which is not the order their loads run in, so loads read chunks still being brought in while others' wait ready. A worker now tells the memory side when the load of a request starts. - read_promotes (vLLM: on in the worker with maru_read_hints and maru_async_load) makes MaruHandler.promote_read(request_id) queue a read_promote on the hint thread: no ranges, no end, fire and forget. The vLLM worker calls it once per remote_cxl load, before its first batch is read. - The worker did not send the request's hint (its scheduler did, on another connection), so it sends the promote to every agent of the pool whose extension takes it; the extension matches it to a hint by request id alone. - The agent takes read_promote like read_hint_end and keeps no request open for it; it advertises and accepts it only if the region extension lists it in hint_actions.
A queued write hint reaches the memory side about when the store's WRITEs do, so most WRITEs still landed on pages the device had to fill from its slow tier first. Stores run off the request path (vLLM's write-behind thread), so they can afford a short wait. - When an agent's region extension takes write_wait, a store sends its pages for that agent in a write_wait (the store's token, the ranges in write order, timeout_ms) and waits for the answer, then posts the WRITEs whatever it is. The waits of one store share write_wait_ms (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an agent that does not answer costs the reply timeout on top. Other agents still get a queued write hint. Waits are off unless write hints are on. - Waits live in waits.py (MemoryWaits): each thread has its own connection per agent, so one store's wait never holds up another's, a hint or a region map. An agent that cannot be reached is not asked again for 5 s; its stores send write hints meanwhile. - The agent checks a write_wait like a write hint, plus timeout_ms (1 to 10000), keeps no request open for it, and calls the extension outside its lock on the requesting connection's own thread; the extension's answer (a map, or None) is the reply. Only that connection waits. It is advertised and accepted only if the extension lists it in hint_actions.
… hint leaves vLLM
…ead sends the hint
…wait backs off Hints and waits are advice, but the store's write preparation and the load's read hint ran after the transfer count went up and before the try that brings it down. If either raised, the transfer never ended: the pages stayed sending, the store failed, and close() waited for the RDMA timeout. Both now run inside that try, and any exception from hint code is logged and ignored. One way they raised: the hint connections read the agent key lazily, so a key file changed after start made load_key raise. They now reuse the key the data path read at start. MemoryWaits also catches a failing connect and backs off instead of raising. A write wait that fails or times out now backs the agent off for UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do not each pay the reply timeout.
hello() reads boot without the lock, and boot was built from three fields the generation watch changes one by one. When the generation became known again, a concurrent hello could advertise the boot from before the unknown period. The boot is now one string, rebuilt under the lock whenever the generation state changes (known again, changed, unknown), so a reader sees the boot from before or after a change, never a mix.
…he extension closed on a failed start - MAX_HINT_RANGES moves to maru_remote.region_map beside MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the write_wait_ms config check all use these two, so both sides agree. - --max-hint-requests and --memory-generation-poll-s must be above zero (zero refused every read hint or polled in a busy loop). - When RegionMapService fails to start, the loaded region extension is closed before the agent exits.
…new handler shutdown() closed the scheduler's MaruHandler but left the arrival-hint pool running, so a request that arrived afterwards could create a new handler through _ensure_handler(). The scheduler connector now has its own shutdown(): it marks itself shut down, stops the arrival pool (waiting for a hint under way), then closes the handler. Arrival hints are submitted under the hint lock, so none lands on a stopped pool.
…er plugins A handler of the cxl backend (direct mapping) now takes read_hints and read_promotes. It gives the hint messages remote_cxl sends to a remote CXL agent's region extension (read_hint with the leading hit keys' bytes, read_hint_end, read_promote), in the same form, to the handler plugins whose hint_actions name them, through the new on_memory_hint hook, on one hint thread. The vLLM connector lets maru_read_hints and maru_read_hint_on_arrival reach the cxl backend (write hints stay with remote_cxl) and promotes each deferred cxl load as it starts.
MemoryHints takes its locate and send callables from the backend and holds the background thread, the queue, the drop policy and the end bookkeeping, none of which depends on RDMA or agents. It now lives in maru_handler/hints.py, so another backend (the cxl backend of #137) uses it without depending on the remote_cxl package. The shared wire limits stay in maru_remote.region_map, which maru_common already imports.
…tension, so the memory node can prepare KV before transfers (#130) * feat(remote_cxl_agent): region extensions with a checked read hint relay A remote CXL agent can load one out-of-tree region extension, registered under the maru.region_extensions entry-point group and named with --region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a package that provides one can turn it on for the installed service with a drop-in). It serves memory that is slow until a device has brought a page in (a CXL-SSD with a DRAM cache in front of its flash): - validate_access may refuse a region before it is mapped, mapped hears it once the NIC has it (raising keeps the region out of use), and retire is asked before a region goes back; while it says no, the region stays here unregistered and the sweep asks again. - hint takes read_hint / read_hint_end messages (op "extension") from any authenticated connection, once the agent has checked the epoch, the action and that every range lies in a region it maps and some connection holds, under a matching handle. The check runs under the agent's lock and the extension is called after it, so no other request waits for the extension. A connection may keep at most --max-hint-requests (4096) hint requests open. - The optional connection_closed(conn) is called for every connection that closes, so the extension can end what its hints started. - The hello advertises region_extension {"version": 1, "actions": [...]} and RegionMapClient keeps the hint actions it advertises; anything else means no hints. An agent that fails to load its extension exits. Without one, nothing changes. * feat(remote_cxl): read hints, so the memory side can prepare a load's bytes early A serving engine knows which stored keys a request will load well before its worker starts the load. Read hints carry that to the memory side, where a region extension can bring the bytes into a faster tier. Maru's KV data path stays as it is: a hint is advice that never gates a READ or WRITE, and one that is late or lost costs only the time it would have saved. - read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on. MaruHandler.hint_read / end_read_hint queue a hint and its end; one thread per backend probes maru-server for where the hits are (a probe with locations, which takes no lease) and sends read_hint and read_hint_end to every agent whose hello advertises them, in messages of at most 4096 ranges. At most 1024 hints wait; ends are never dropped. The thread keeps at most 4096 hinted requests open and ends the oldest past that. - A client that loads with read_hints on hints each load as its READs start, under a name of its own, and ends that hint after the load (for engines that name no request to their storage). - The vLLM scheduler connector hints a request when it first finds its asynchronous load, or on arrival with maru_read_hint_on_arrival, and ends the hint when vLLM reports the load done (or the request finishes first). Only the scheduler gets read_hints, so vLLM loads are not hinted twice. - The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since vLLM's connector API is first called at the next step boundary. It is installed only if the method takes (self, request) and passes on only a result whose first item is a request with an id; otherwise hints on arrival are off with a warning. * docs: read hints design and configuration * feat(remote_cxl_agent): fresh region maps and memory generations A region extension may front memory whose contents a device keeps in a write-back cache (a CXL-SSD). Two things about such memory reach it now. Fresh regions. A region its owner allocated in this session holds no KV until the owner writes to it, and the owner writes only after a successful map reply. Until the first one, its maps carry fresh=true, retries included; after it (an agent restart, a remap, a region given back) they do not. The agent takes fresh only with the region's own handle and only on the map that maps the region there first, and passes it to RegionExtension.mapped(fresh=) before the reply leaves, so a device can drop the region's stale contents before any WRITE reaches it. fresh is the owner's word, to be used only on the region's own bytes. An agent that does not know fresh ignores it. Memory generations. A write-back cache loses what it held on a reset without a reboot of its node, and Maru already treats a changed agent boot as a reboot: owners give their regions back and readers load misses. An extension may tell the memory's generation (optional memory_generation(), local state, outside the agent's lock), and the agent advertises "<node boot id>:<generation>" as its boot. - The agent reads the generation at start (a failure stops it), on a thread every --memory-generation-poll-s (100 ms) and on every hello, map and sweep, one caller at a time. When it changes, every mapping is deregistered from the NIC, so RDMA through an old registration fails, and dropped as a release does. Map replies now carry the boot, and a worker takes it over its connection's. - A generation the extension does not tell for --memory-generation-grace-s (2 s) is unknown: every mapping goes, the boot reads <boot id>:unknown and maps are refused until it answers. The boot after the n-th such period is <boot id>:<generation>~<n>, so nothing mapped before one is trusted after it. - A map whose boot changed while it ran fails and its registration goes. * docs: fresh region maps and memory resets * feat(remote_cxl): write hints, so the memory side can prepare a store's pages Without them, a store's RDMA WRITEs into memory with a slow tier behind a cache make the device fill each written page from the slow tier first, page by page. On a CXL-SSD, a build that stored without write hints wrote at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other settings differed too). - write_hints (vLLM: maru_write_hints, which follows maru_read_hints unless set) makes a store, once it has its pages and before it posts the WRITEs, queue a write_hint on the hint thread: the pages, in write order, under a token for the store. Fire and forget; no end. With vLLM only the worker sends them (it stores; the scheduler hints loads). - The agent takes write_hint with the read hints' checks; it keeps no request open for it. A region extension says which hint actions it takes (optional hint_actions; read_hint and read_hint_end without it), the hello advertises those, Maru sends only those and the agent refuses others. A write hint gives no leave to change the pages. - The hint thread class is MemoryHints now that it sends both kinds. * feat(remote_cxl): promote a hinted load when it starts The memory side brings hinted requests' bytes in in hint order, which is not the order their loads run in, so loads read chunks still being brought in while others' wait ready. A worker now tells the memory side when the load of a request starts. - read_promotes (vLLM: on in the worker with maru_read_hints and maru_async_load) makes MaruHandler.promote_read(request_id) queue a read_promote on the hint thread: no ranges, no end, fire and forget. The vLLM worker calls it once per remote_cxl load, before its first batch is read. - The worker did not send the request's hint (its scheduler did, on another connection), so it sends the promote to every agent of the pool whose extension takes it; the extension matches it to a hint by request id alone. - The agent takes read_promote like read_hint_end and keeps no request open for it; it advertises and accepts it only if the region extension lists it in hint_actions. * feat(remote_cxl): a store waits briefly for its pages to be prepared A queued write hint reaches the memory side about when the store's WRITEs do, so most WRITEs still landed on pages the device had to fill from its slow tier first. Stores run off the request path (vLLM's write-behind thread), so they can afford a short wait. - When an agent's region extension takes write_wait, a store sends its pages for that agent in a write_wait (the store's token, the ranges in write order, timeout_ms) and waits for the answer, then posts the WRITEs whatever it is. The waits of one store share write_wait_ms (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an agent that does not answer costs the reply timeout on top. Other agents still get a queued write hint. Waits are off unless write hints are on. - Waits live in waits.py (MemoryWaits): each thread has its own connection per agent, so one store's wait never holds up another's, a hint or a region map. An agent that cannot be reached is not asked again for 5 s; its stores send write hints meanwhile. - The agent checks a write_wait like a write hint, plus timeout_ms (1 to 10000), keeps no request open for it, and calls the extension outside its lock on the requesting connection's own thread; the extension's answer (a map, or None) is the reply. Only that connection waits. It is advertised and accepted only if the extension lists it in hint_actions. * docs: write hints, store waits and read promotes * docs: call read and write hints prefetch hints, and show where a read hint leaves vLLM * docs: shorter panel titles in the prefetch hint figure; say which thread sends the hint * fix(remote_cxl): hint code never fails a store or load, and a failed wait backs off Hints and waits are advice, but the store's write preparation and the load's read hint ran after the transfer count went up and before the try that brings it down. If either raised, the transfer never ended: the pages stayed sending, the store failed, and close() waited for the RDMA timeout. Both now run inside that try, and any exception from hint code is logged and ignored. One way they raised: the hint connections read the agent key lazily, so a key file changed after start made load_key raise. They now reuse the key the data path read at start. MemoryWaits also catches a failing connect and backs off instead of raising. A write wait that fails or times out now backs the agent off for UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do not each pay the reply timeout. * fix(remote_cxl_agent): a hello never advertises a half-changed boot hello() reads boot without the lock, and boot was built from three fields the generation watch changes one by one. When the generation became known again, a concurrent hello could advertise the boot from before the unknown period. The boot is now one string, rebuilt under the lock whenever the generation state changes (known again, changed, unknown), so a reader sees the boot from before or after a change, never a mix. * fix(remote_cxl_agent): shared hint limits, positive CLI values, and the extension closed on a failed start - MAX_HINT_RANGES moves to maru_remote.region_map beside MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the write_wait_ms config check all use these two, so both sides agree. - --max-hint-requests and --memory-generation-poll-s must be above zero (zero refused every read hint or polled in a busy loop). - When RegionMapService fails to start, the loaded region extension is closed before the agent exits. * fix(vllm): scheduler shutdown stops the arrival hints and creates no new handler shutdown() closed the scheduler's MaruHandler but left the arrival-hint pool running, so a request that arrived afterwards could create a new handler through _ensure_handler(). The scheduler connector now has its own shutdown(): it marks itself shut down, stops the arrival pool (waiting for a hint under way), then closes the handler. Arrival hints are submitted under the hint lock, so none lands on a stopped pool. * fix(remote_cxl_agent): a map never succeeds while the memory generation is unknown map() checks the generation before mapping, and _establish compares the boot it read at the start with the boot at the end. If the generation turned unknown between the check and the first boot read, both boots read ":unknown", the comparison passed, and the map succeeded. Its registration then outlived the unknown period: the sweep kept it and the recovery released nothing, so the owner could RDMA into memory whose state was unknown. _establish now also refuses a map whose generation is unknown at the end. A boot that is known at the end already differs from an unknown one read at the start, so that case was covered. * fix(vllm): a lookup hint goes to the pool its load reads With several remote_cxl pools, the scheduler's lookup hint was routed by the first key it named. When vLLM's own prefix cache held a chunk or more, that key was not the request's first chunk key, which routes the load, so the hint went to another pool and was dropped there. The hint is now routed by the request's first chunk key, as the load and the arrival hint are. * refactor(handler): MemoryHints moves out of the remote_cxl package MemoryHints takes its locate and send callables from the backend and holds the background thread, the queue, the drop policy and the end bookkeeping, none of which depends on RDMA or agents. It now lives in maru_handler/hints.py, so another backend (the cxl backend of #137) uses it without depending on the remote_cxl package. The shared wire limits stay in maru_remote.region_map, which maru_common already imports.
|
Reviewed the current head,
Validation: the existing local hint, plugin loader, and |
The scheduler hinted bare chunk keys, while TP workers store one object per KV slice (and, layerwise, per layer; with several KV cache groups, per group). The lookup of those keys missed, so a cxl hint plugin got no read hint outside TP=1 chunkwise with one group. The scheduler now names the objects the workers load, chunk by chunk, from the same load rule the worker uses (_group_load_items), and builds the keys on one hint thread that also carries the ends, so the scheduler's path stays short and an end never passes its hint. A metadata-only handler (a vLLM scheduler's) returned from close() before its plugins' on_close, and closing its hint queue sent no end for the requests still open. close() now ends every open hint, then calls on_close while the RPC connection is live.
|
@seohui-XCENA Replying to your review (#137 (comment)). Both are fixed in 6427d2f.
The scheduler now hints the objects the workers load instead of bare chunk keys: every TP slice of a chunk, each layer's object in layerwise format (not the
On close, a cxl handler now sends the queued hints, then Your three cases are now regression tests (TP and layerwise hint delivery through a real MaruHandler and plugin, and the close order), plus a multi-group key test. On the previous head, only TP=1 chunkwise passes. Could you take another look? |
# Conflicts: # docs/source/api_reference/plugins.md # docs/source/design_doc/read_hints.md
seohui-XCENA
left a comment
There was a problem hiding this comment.
The hint shutdown ordering needs to be addressed before merging. I reproduced callbacks running after on_close when the hint thread exceeds its shutdown timeout.
Separately, the multi-group CXL load path does not emit read_promote. This can be addressed here or documented as a limitation for follow-up.
The existing relevant tests passed: 325 passed, 27 skipped. Additional targeted tests reproduced both issues.
- MemoryHints.close() returns False when its thread is still busy past the timeout, and MaruHandler.close() went on to on_close regardless, so a hint or end could reach a plugin after on_close. Past the timeout the handler now abandons the queue (the messages still queued are dropped) and waits for the plugin call under way before on_close. - _group_load_job (several KV cache groups) sent no read_promote; it now promotes the hinted request before its first read, as the one-group load does.
seohui-XCENA
left a comment
There was a problem hiding this comment.
Verified the fixes at 9c63aae. Both review findings are addressed:
- Handler shutdown waits for the hint thread to exit before plugin teardown, with the timeout/drop behavior documented.
- Multi-group loads now emit
read_promotebefore their first read.
Local validation: 360 passed, 27 skipped across the relevant hint, plugin, and connector tests, including the added regression cases. This validation used CPU tensors and mocked components; it does not establish GPU/CXL device performance.
At approval time, Python 3.13/3.14, Resource Manager, lint, and Jenkins checks have passed. The Python 3.12 rerun is still in progress following the earlier Nexus dependency-download timeout.
🤔 Background & Motivation (Why)
Let a local CXL handler plugin prepare stored KV before a request loads it. Previously, read hints were accepted only by the remote CXL backend. A local plugin's retrieval hook ran when loading had already begun.
The scheduler identifies the KV a request will reuse. The worker loads that KV into GPU memory. This PR delivers the scheduler's early read hint and the worker's load-start notification to an independently installed plugin.
The scheduler's read hints, which
remote_cxlalready sent, also had a defect: they named whole-chunk keys, while workers store one object per TP slice, per layer in layerwise format, and per KV cache group. Unless TP=1, the format is chunkwise and there is one KV cache group, the hint's keys matched no stored object and no hint reached storage. The hints this PR gives to local plugins are built the fixed way.🏗️ Design Changes
%%{init: {"themeVariables": {"fontSize": "13px"}, "flowchart": {"nodeSpacing": 16, "rankSpacing": 24}}}%% flowchart LR S[Scheduler] -->|read_hint · end| P[Plugin] W[Worker] -->|read_promote| P K[(CXL KV)] ==>|load| WFigure 1. Thin arrows are the added plugin notifications; the thick arrow is the existing KV load. Previously, the retrieval hook notified the plugin only at load time; the scheduler's hint now provides an earlier preparation opportunity. Handlers and queues are omitted. Delivery is asynchronous, so promote may precede hint. The end notification follows load completion, an earlier request finish, or handler close unless close times out.
📝 Implementation Details
read_hintsandread_promoteson thecxlbackend and notify the plugin when a load of a hinted request starts, for models with one or several KV cache groups.hint_actionsandon_memory_hint(handler, message). Messages carry the request ID and, for a read hint, the leading stored byte ranges.read_hint_endnotifications, so an end never overtakes its hint.cxlhandler sends the queued hints, thenread_hint_endfor every open request, then calls pluginon_close, including on ametadata_onlyhandler (the one a vLLM scheduler uses). Previously that handler returned beforeon_closeand sent no end for open requests. If the queued hints take longer than 5 s, every message still queued is dropped, the ends of open requests included, and close waits without a time limit for the hint thread's call under way (a metadata lookup or a plugin callback), so no hint reaches a plugin afteron_close. Plugin callbacks must return promptly.remote_cxlbackend.✅ Tests
ruff check .,ruff format --check .,git diff --check.CUDA_VISIBLE_DEVICES= python -m pytest -q \ tests/unit/test_cxl_read_hints.py tests/unit/test_plugin_loader.py \ tests/unit/test_vllm*.py \ --deselect=tests/unit/test_vllm_remote_cxl_tp_groups.py::test_the_scheduler_looks_up_in_the_workers_namespaceThe deselected test also fails on unchanged main because the installed vLLM 0.22.1 lacks its required API. Validation used CPU tensors and fake components; it does not establish device performance.
🔗 Related Issues (optional)
🌿 Related PRs (optional)
🌿 Related Branches (optional)
📦 Release Note (for auto-generation / write in English)
NEW
CHANGED
FIXED
IMPORTANT NOTES
on_closemay follow open requests that got noread_hint_end; release per-request state inon_close.