Skip to content

feat(remote_cxl): prefetch hints (read and write) through a region extension, so the memory node can prepare KV before transfers - #130

Merged
youngrok-XCENA merged 22 commits into
mainfrom
feat/remote-cxl-read-hints
Nov 6, 2026
Merged

youngrok-XCENA merged 22 commits into
mainfrom
feat/remote-cxl-read-hints

Conversation

@youngrok-XCENA

@youngrok-XCENA youngrok-XCENA commented Oct 27, 2026 •

Copy link
Copy Markdown
Collaborator

🤔 Background & Motivation (Why)

Maru can place KV on remote memory that has a front cache, such as a CXL-SSD (a DRAM cache in front of flash). But Maru tells the memory node only which regions are mapped, so the device cannot bring KV into its DRAM before a READ arrives: a READ of KV still on flash waits for flash. Maru does know which KV a request will read between the request's arrival and its READ (a median of 14.3 ms in the serving replay below). This PR sends that to the memory node as prefetch hints: the KV a load will read (read hints) and the pages a store will write (write hints), so that a CXL-SSD can bring them from flash into its DRAM cache before the transfer. Maru moves no KV for a hint; the device moves its own pages, through a region extension: a hook on the remote CXL agent, implemented by a separately installed package. This PR adds the hooks only; Maru's KV data path is unchanged, and memory nodes backed by plain DRAM or CXL DRAM are unaffected. In the serving replay, read hints cut the mean time to first token (TTFT) of turns that reuse KV from 1.18× to 1.004× of the same path with all KV in device DRAM.

Scope: remote_cxl only. With the local cxl backend, where the GPU node maps the CXL-SSD itself and there is no agent, the hint settings (read_hints, write_hints, read_promotes, where a read promote tells the memory node that a hinted load starts now so its KV goes first, and their vLLM forms such as maru_read_hints) are rejected at startup, and MaruHandler.hint_read, promote_read and end_read_hint do nothing. The existing handler plugin hook on_batch_retrieve runs at load time, too late to prepare anything. A follow-up PR, stacked on this one, will allow these settings on the cxl backend and pass the calls to handler plugins (maru.handler_plugins).

Figure 1 covers reads; writes and resets are in the table below.

Two panels of the same vLLM engine with the Maru connector, each above a strip for the memory node (CXL-SSD) on a time axis from "request arrives" to "load starts". (a) Without prefetch hints: the request goes through the input thread, the input queue and the scheduler's waiting queue to the worker, whose RDMA READ arrives at "load starts"; nothing happens on the memory node while the request waits in vLLM, and then the READs wait while the device brings the KV from flash into its DRAM cache. (b) With prefetch hints: a Maru wrapper on the input thread hands the request to the scheduler's Maru connector, whose hint thread sends a read hint that reaches the memory node at "request arrives"; the device prefetches the KV while the request waits, and the READs at "load starts" find it in DRAM

Figure 1. Where a read hint leaves vLLM (schematic). With prefetch hints (b), Maru's wrapper runs on vLLM's input thread and calls the scheduler's Maru connector there; its hint thread sends the read hint before the request enters the input queue, so the device prefetches while the request waits (median 14.3 ms in the serving replay). Mean per-request increase in KV load time over a paired run with all KV in device DRAM: +41 ms without read hints, +1.2 ms with them. Path (b) is opt-in (maru_read_hint_on_arrival) and these numbers used it; the default path is compared in the measurement paragraph below. The memory node strip is simplified: a hint reaches the device through the remote CXL agent and the region extension.

What goes wrong without this PR, on a CXL-SSD used as remote_cxl memory:

Without this PR What happens Evidence
Reads A READ of KV still on flash waits while the device reads it. One 16 MiB chunk: 6.65 ms on flash vs 1.59 ms once in DRAM (measured on the device). In serving, with read hints off and everything else as in this PR: TTFT 1.18× [1.13, 1.24] of all KV in device DRAM, KV load +41 ms vs +1.2 ms with hints (3 pairs). The device's read-ahead hides part of the flash cost; with it off too, TTFT is 7.85×.
Writes A WRITE to a page the cache does not hold waits while the device first reads that page from flash. Write hints let that read happen before the WRITE. Stores at 1.3 GB/s without write hints vs 10.3 GB/s with pages prepared, and 8 reuse requests of the next turn missed (the run's total was not recorded); effect on TTFT not measured (observed during development, other settings differed).
Resets Maru learns that remote memory was lost only when the memory node's kernel boot ID changes. A device whose cache resets while the node stays up gives no signal, so KV not yet written to flash could be read back as hits. With this PR, an extension can report a memory generation that catches the resets the device reports. Inferred from the code; no device reset was performed.
Device code Maru has no place for device-specific logic. #124, the earlier attempt, put a check with the memory node in every READ's path.

Serving replay: one CXL-SSD, vLLM 0.22.1, Llama-3.1-8B, 48 conversations × 6 turns, 33.6 GiB of KV behind a 16 GiB device DRAM cache. Metric: mean TTFT of the turns that reuse earlier KV, relative to the same path with all KV in device DRAM. With this PR and maru_read_hint_on_arrival (path (b) in Figure 1): 1.004× [0.993, 1.015] (5 pairs). With the default path, where the read hint leaves when the scheduler first looks the request up: 1.005× [0.979, 1.031] (3 pairs), the same within the intervals; the device's prefetch rate (measured on an idle device) was close to the RDMA READ rate of the 100GbE link, so a prefetch that starts with the load still kept ahead of the READs. No run without the PR was made on this build. Brackets are 95% intervals over paired runs.

Replaces #124, which reached the same TTFT (1.004×) but made every RDMA READ ask the memory node first: a round trip per READ, with device policy in the data path.

🏗️ Design Changes

Before: the memory node hears only which regions are mapped.

flowchart LR
  subgraph gpuB["GPU node"]
    mB["Maru (vLLM scheduler + worker)"]
  end
  subgraph memB["memory node"]
    agB["remote CXL agent"]
    devB[("CXL-SSD")]
  end
  mB -->|"map regions"| agB
  mB ==>|"RDMA READ / WRITE"| devB
Loading

After: Maru sends prefetch hints (read and write) to the agent, which checks them and hands them to the region extension (the agent's hook, implemented by a package installed on the memory node) that drives the device. Thick line: KV bytes; solid: control messages; dotted: device commands.

flowchart LR
  subgraph gpuA["GPU node"]
    mA["Maru (vLLM scheduler + worker)"]
  end
  subgraph memA["memory node"]
    agA["remote CXL agent<br/>(checks every hinted range)"]
    extA["region extension<br/>(separate package)"]
    devA[("CXL-SSD")]
  end
  mA -->|"map regions, read and write hints"| agA
  agA -->|"checked hints, region maps"| extA
  extA -.->|"prefetch, write-back,<br/>memory generation"| devA
  mA ==>|"RDMA READ / WRITE (unchanged)"| devA
  classDef pkg fill:#fbe3cc,stroke:#c06000,color:#111
  class extA pkg
Loading
  • Hints are advice: no READ or WRITE waits for one, except a store's short wait for its pages.
  • Everything is opt-in, and an agent gets only the hints its extension says it takes. Without an extension, nothing changes.

📝 Implementation Details

Review in three parts; each part passes the unit tests on its own. The full design is in docs/source/design_doc/read_hints.md.

Commits Part
1–3 Region extension on the agent; read hints from the vLLM scheduler
4–5 Memory generations (resets) and fresh region maps
6–11 Write hints, store waits, read promotes; docs figure

Please look at:

  • vLLM internals. maru_read_hint_on_arrival wraps vLLM's EngineCore.preprocess_add_request, because the connector API is first called at the next step boundary. It is installed only if the method takes (self, request); otherwise hints fall back to the scheduler's first lookup, with a warning. Checked against vLLM 0.22.1 and 0.32.0 sources; run only on 0.22.1. On the measured 100GbE link it gave the same TTFT as the default path (above); a link faster than the device's prefetch, where the earlier start is expected to shorten the load, was not measured.
  • Lookups without a lease. probe with locations gives the scheduler the read handles of its hits without a lease; the hint only names them, and every READ still leases.
  • Agent behavior with an extension. The agent checks every hinted range against the regions it maps. While the extension keeps a region (retire answers False), a release of it errors and a new map of it is refused; agent shutdown waits up to 30 s for the extension.
  • Resets. A generation change drops every mapping; a generation the extension does not report for 2 s makes the agent refuse maps until it answers.
  • Defaults that change behavior. In vLLM, maru_write_hints follows maru_read_hints, so turning on read hints also turns on store waits: up to 50 ms (maru_write_wait_ms), plus the reply timeout if the agent does not answer. Stores run on the write-behind thread (on by default for remote_cxl).

✅ Tests

  • Unit tests: 2102 passed on the last commit; ruff and the unit tests pass on every commit
  • Integration tests
  • Manual tests: serving replay and device checks on a CXL-SSD with an out-of-tree extension (above)
  • No tests needed (reason: )

🔗 Related Issues (optional)

🌿 Related PRs (optional)

🌿 Related Branches (optional)

📦 Release Note (for auto-generation / write in English)

NEW

CHANGED

FIXED

IMPORTANT NOTES

@youngrok-XCENA
youngrok-XCENA force-pushed the feat/remote-cxl-read-hints branch from d7af561 to f3692fe Compare October 30, 2026 10:55
@youngrok-XCENA youngrok-XCENA changed the title feat(remote_cxl): read hints, so the memory node can prepare KV ranges before the load reads them feat(remote_cxl): read and write hints through a region extension, so the memory node can prepare KV before transfers Oct 30, 2026
@youngrok-XCENA youngrok-XCENA changed the title feat(remote_cxl): read and write hints through a region extension, so the memory node can prepare KV before transfers feat(remote_cxl): prefetch hints (read and write) through a region extension, so the memory node can prepare KV before transfers Nov 2, 2026
A remote CXL agent can load one out-of-tree region extension, registered
under the maru.region_extensions entry-point group and named with
--region-extension (or MARU_REMOTE_CXL_AGENT_REGION_EXTENSION, so a
package that provides one can turn it on for the installed service with
a drop-in). It serves memory that is slow until a device has brought a
page in (a CXL-SSD with a DRAM cache in front of its flash):

- validate_access may refuse a region before it is mapped, mapped hears
  it once the NIC has it (raising keeps the region out of use), and
  retire is asked before a region goes back; while it says no, the
  region stays here unregistered and the sweep asks again.
- hint takes read_hint / read_hint_end messages (op "extension") from
  any authenticated connection, once the agent has checked the epoch, the
  action and that every range lies in a region it maps and some
  connection holds, under a matching handle. The check runs under the
  agent's lock and the extension is called after it, so no other request
  waits for the extension. A connection may keep at most
  --max-hint-requests (4096) hint requests open.
- The optional connection_closed(conn) is called for every connection
  that closes, so the extension can end what its hints started.
- The hello advertises region_extension {"version": 1, "actions": [...]}
  and RegionMapClient keeps the hint actions it advertises; anything else
  means no hints.

An agent that fails to load its extension exits. Without one, nothing
changes.
… bytes early

A serving engine knows which stored keys a request will load well before
its worker starts the load. Read hints carry that to the memory side,
where a region extension can bring the bytes into a faster tier. Maru's
KV data path stays as it is: a hint is advice that never gates a READ or
WRITE, and one that is late or lost costs only the time it would have
saved.

- read_hints (maru_read_hints in vLLM, remote_cxl only) turns them on.
  MaruHandler.hint_read / end_read_hint queue a hint and its end; one
  thread per backend probes maru-server for where the hits are (a probe
  with locations, which takes no lease) and sends read_hint and
  read_hint_end to every agent whose hello advertises them, in messages
  of at most 4096 ranges. At most 1024 hints wait; ends are never
  dropped. The thread keeps at most 4096 hinted requests open and ends
  the oldest past that.
- A client that loads with read_hints on hints each load as its READs
  start, under a name of its own, and ends that hint after the load
  (for engines that name no request to their storage).
- The vLLM scheduler connector hints a request when it first finds its
  asynchronous load, or on arrival with maru_read_hint_on_arrival, and
  ends the hint when vLLM reports the load done (or the request
  finishes first). Only the scheduler gets read_hints, so vLLM loads are
  not hinted twice.
- The arrival hint wraps vLLM's EngineCore.preprocess_add_request, since
  vLLM's connector API is first called at the next step boundary. It is
  installed only if the method takes (self, request) and passes on only
  a result whose first item is a request with an id; otherwise hints on
  arrival are off with a warning.
A region extension may front memory whose contents a device keeps in a
write-back cache (a CXL-SSD). Two things about such memory reach it now.

Fresh regions. A region its owner allocated in this session holds no KV
until the owner writes to it, and the owner writes only after a
successful map reply. Until the first one, its maps carry fresh=true,
retries included; after it (an agent restart, a remap, a region given
back) they do not. The agent takes fresh only with the region's own
handle and only on the map that maps the region there first, and passes
it to RegionExtension.mapped(fresh=) before the reply leaves, so a
device can drop the region's stale contents before any WRITE reaches
it. fresh is the owner's word, to be used only on the region's own
bytes. An agent that does not know fresh ignores it.

Memory generations. A write-back cache loses what it held on a reset
without a reboot of its node, and Maru already treats a changed agent
boot as a reboot: owners give their regions back and readers load
misses. An extension may tell the memory's generation (optional
memory_generation(), local state, outside the agent's lock), and the
agent advertises "<node boot id>:<generation>" as its boot.

- The agent reads the generation at start (a failure stops it), on a
  thread every --memory-generation-poll-s (100 ms) and on every hello,
  map and sweep, one caller at a time. When it changes, every mapping is
  deregistered from the NIC, so RDMA through an old registration fails,
  and dropped as a release does. Map replies now carry the boot, and a
  worker takes it over its connection's.
- A generation the extension does not tell for
  --memory-generation-grace-s (2 s) is unknown: every mapping goes, the
  boot reads <boot id>:unknown and maps are refused until it answers.
  The boot after the n-th such period is <boot id>:<generation>~<n>, so
  nothing mapped before one is trusted after it.
- A map whose boot changed while it ran fails and its registration goes.
…'s pages

Without them, a store's RDMA WRITEs into memory with a slow tier behind a
cache make the device fill each written page from the slow tier first,
page by page. On a CXL-SSD, a build that stored without write hints wrote
at 1.3 GB/s and the next turn missed 8 reuse lookups; a later build with
the pages prepared wrote at 10.3 GB/s (an uncontrolled observation: other
settings differed too).

- write_hints (vLLM: maru_write_hints, which follows maru_read_hints
  unless set) makes a store, once it has its pages and before it posts
  the WRITEs, queue a write_hint on the hint thread: the pages, in write
  order, under a token for the store. Fire and forget; no end. With vLLM
  only the worker sends them (it stores; the scheduler hints loads).
- The agent takes write_hint with the read hints' checks; it keeps no
  request open for it. A region extension says which hint actions it
  takes (optional hint_actions; read_hint and read_hint_end without it),
  the hello advertises those, Maru sends only those and the agent
  refuses others. A write hint gives no leave to change the pages.
- The hint thread class is MemoryHints now that it sends both kinds.
The memory side brings hinted requests' bytes in in hint order, which is
not the order their loads run in, so loads read chunks still being
brought in while others' wait ready. A worker now tells the memory side
when the load of a request starts.

- read_promotes (vLLM: on in the worker with maru_read_hints and
  maru_async_load) makes MaruHandler.promote_read(request_id) queue a
  read_promote on the hint thread: no ranges, no end, fire and forget.
  The vLLM worker calls it once per remote_cxl load, before its first
  batch is read.
- The worker did not send the request's hint (its scheduler did, on
  another connection), so it sends the promote to every agent of the
  pool whose extension takes it; the extension matches it to a hint by
  request id alone.
- The agent takes read_promote like read_hint_end and keeps no request
  open for it; it advertises and accepts it only if the region extension
  lists it in hint_actions.
A queued write hint reaches the memory side about when the store's WRITEs
do, so most WRITEs still landed on pages the device had to fill from its
slow tier first. Stores run off the request path (vLLM's write-behind
thread), so they can afford a short wait.

- When an agent's region extension takes write_wait, a store sends its
  pages for that agent in a write_wait (the store's token, the ranges in
  write order, timeout_ms) and waits for the answer, then posts the
  WRITEs whatever it is. The waits of one store share write_wait_ms
  (MaruConfig, default 50, 0 never waits; vLLM maru_write_wait_ms); an
  agent that does not answer costs the reply timeout on top. Other agents
  still get a queued write hint. Waits are off unless write hints are on.
- Waits live in waits.py (MemoryWaits): each thread has its own
  connection per agent, so one store's wait never holds up another's, a
  hint or a region map. An agent that cannot be reached is not asked
  again for 5 s; its stores send write hints meanwhile.
- The agent checks a write_wait like a write hint, plus timeout_ms (1 to
  10000), keeps no request open for it, and calls the extension outside
  its lock on the requesting connection's own thread; the extension's
  answer (a map, or None) is the reply. Only that connection waits.
  It is advertised and accepted only if the extension lists it in
  hint_actions.
@youngrok-XCENA
youngrok-XCENA force-pushed the feat/remote-cxl-read-hints branch from 7eef3b6 to 94365de Compare November 2, 2026 03:10
@youngrok-XCENA
youngrok-XCENA marked this pull request as ready for review November 2, 2026 03:12

@seohui-XCENA seohui-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read the diff and ran the remote_cxl and vLLM unit tests on 94365de (1188 passed). In one case, hint code can break a store or load, which the hints-are-advice design is meant to rule out (inline on client.py L878). The rest are suggestions. I reproduced the first two inline issues with unit tests on the PR's fixtures, not on a device.

Nits:

  • _MAX_HINT_RANGES (service.py L96) and the 10_000 in config.py L171 repeat hints.MAX_HINT_RANGES and region_map.MAX_WRITE_WAIT_MS. Both sides must agree, so region_map.py could hold them.
  • The agent CLI accepts --memory-generation-poll-s ≤ 0 (a busy loop) and --max-hint-requests ≤ 0 (every read hint is refused). It also doesn't close the loaded extension when RegionMapService fails to start.
  • shutdown() doesn't stop _arrival_pool. A request that arrives afterwards can create a new scheduler handler through _ensure_handler().

Comment thread maru_remote_cxl_agent/service.py Resolved
…wait backs off

Hints and waits are advice, but the store's write preparation and the
load's read hint ran after the transfer count went up and before the try
that brings it down. If either raised, the transfer never ended: the
pages stayed sending, the store failed, and close() waited for the RDMA
timeout. Both now run inside that try, and any exception from hint code
is logged and ignored.

One way they raised: the hint connections read the agent key lazily, so
a key file changed after start made load_key raise. They now reuse the
key the data path read at start. MemoryWaits also catches a failing
connect and backs off instead of raising.

A write wait that fails or times out now backs the agent off for
UNREACHABLE_RETRY_S, as the module docstring says, so the next stores do
not each pay the reply timeout.
hello() reads boot without the lock, and boot was built from three
fields the generation watch changes one by one. When the generation
became known again, a concurrent hello could advertise the boot from
before the unknown period. The boot is now one string, rebuilt under
the lock whenever the generation state changes (known again, changed,
unknown), so a reader sees the boot from before or after a change,
never a mix.
…he extension closed on a failed start

- MAX_HINT_RANGES moves to maru_remote.region_map beside
  MAX_WRITE_WAIT_MS; the worker's hint sender, the agent and the
  write_wait_ms config check all use these two, so both sides agree.
- --max-hint-requests and --memory-generation-poll-s must be above zero
  (zero refused every read hint or polled in a busy loop).
- When RegionMapService fails to start, the loaded region extension is
  closed before the agent exits.
…new handler

shutdown() closed the scheduler's MaruHandler but left the arrival-hint
pool running, so a request that arrived afterwards could create a new
handler through _ensure_handler(). The scheduler connector now has its
own shutdown(): it marks itself shut down, stops the arrival pool
(waiting for a hint under way), then closes the handler. Arrival hints
are submitted under the hint lock, so none lands on a stopped pool.
@youngrok-XCENA

Copy link
Copy Markdown
Collaborator Author

Thanks for the careful read. All three nits are fixed:

  • Shared limits (539f5ac): MAX_HINT_RANGES now lives in maru_remote/region_map.py beside MAX_WRITE_WAIT_MS. The worker's hint sender, the agent's range check and the write_wait_ms config check all import these two.
  • Agent CLI (539f5ac): --max-hint-requests and --memory-generation-poll-s must be above zero (argparse rejects 0 and negative values). When RegionMapService fails to start, the loaded region extension is closed before the agent exits. Tests: test_the_command_line_refuses_a_hint_limit_or_poll_of_zero and test_main_closes_the_extension_when_the_service_cannot_start.
  • Scheduler shutdown (373abf6): the scheduler connector now has its own shutdown(). It marks itself shut down, stops _arrival_pool (waiting for a hint under way), then closes the handler. After that, request_arrived() does nothing and _ensure_handler() creates no handler. Arrival hints are submitted under the hint lock, so none lands on a stopped pool. Test: test_shutdown_stops_arrival_hints_and_creates_no_new_handler.

Re-requesting your review.

@seohui-XCENA seohui-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked on 373abf6 and ran the unit tests (2112 passed). The earlier items are all fixed. Two new ones, both reproduced with unit tests on the PR's fixtures, not on a device. The first is a one-line fix that I'd like in before merge. The second is optional.

Comment thread maru_remote_cxl_agent/service.py Resolved
Comment thread maru_vllm/connector.py Resolved
Comment thread maru_handler/hints.py Resolved
Comment thread maru_remote_cxl_agent/extension.py Resolved
youngrok-XCENA and others added 3 commits November 5, 2026 17:10
…on is unknown

map() checks the generation before mapping, and _establish compares the
boot it read at the start with the boot at the end. If the generation
turned unknown between the check and the first boot read, both boots read
":unknown", the comparison passed, and the map succeeded. Its registration
then outlived the unknown period: the sweep kept it and the recovery
released nothing, so the owner could RDMA into memory whose state was
unknown.

_establish now also refuses a map whose generation is unknown at the end.
A boot that is known at the end already differs from an unknown one read
at the start, so that case was covered.
With several remote_cxl pools, the scheduler's lookup hint was routed by
the first key it named. When vLLM's own prefix cache held a chunk or more,
that key was not the request's first chunk key, which routes the load, so
the hint went to another pool and was dropped there. The hint is now
routed by the request's first chunk key, as the load and the arrival hint
are.
MemoryHints takes its locate and send callables from the backend and holds
the background thread, the queue, the drop policy and the end bookkeeping,
none of which depends on RDMA or agents. It now lives in
maru_handler/hints.py, so another backend (the cxl backend of #137) uses it
without depending on the remote_cxl package. The shared wire limits stay in
maru_remote.region_map, which maru_common already imports.

@seohui-XCENA seohui-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked on 35e481a: both items are fixed, and the repros from my last review pass. Unit tests: 2163 passed; the 107 failures are CUDA-kernel tests that fail the same way on main (c55266b) here.

One thing from the merge with #141: the scheduler's read hint (and the arrival hint) names the bare chunk keys, but on remote_cxl a chunk is now stored as key + slice suffix at TP>1 and as key_c{n}_g{gid} with several KV cache groups. _hint_locate stops at the first miss, so in either setup no read hint is sent. In a two-pool unit test with two slices, no agent got a hint, while the same test at TP=1 does; the several-groups case follows from the key names. Load-time hints and write hints name the real keys and are unaffected. Nothing breaks; the prefetch just doesn't happen.

Could you either state the limit here (e.g. in read_hints.md and the maru_read_hints row: scheduler read hints apply at TP=1 with one KV cache group for now) and handle it in a follow-up PR, or have the hint name the objects the load reads in this PR? Either is fine with me.

@youngrok-XCENA
youngrok-XCENA enabled auto-merge (squash) November 6, 2026 04:39

@kihwan-XCENA kihwan-XCENA left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

…r now

Since #141, remote_cxl stores a chunk under its key plus a slice suffix
(TP > 1) or a group suffix (several KV cache groups). The scheduler's read
hint names the bare chunk keys, finds no stored chunk and sends nothing.
Loads, the hints a worker sends as a load starts, read promotes and write
hints are unaffected. A follow-up makes the hint name the objects the load
reads.
@youngrok-XCENA

Copy link
Copy Markdown
Collaborator Author

@seohui-XCENA Thanks for catching the interaction with #141. I took the first option: 60c677a states the limit in read_hints.md (Scope) and in the maru_read_hints row of the config reference. Scheduler read hints apply at TP = 1 with one KV cache group for now; loads, the hints a worker sends as a load starts, read promotes and write hints are unaffected.

A follow-up PR will make the scheduler's hint name the objects the load reads (every slice and group key of each chunk).

@youngrok-XCENA
youngrok-XCENA merged commit 6cd6cfe into main Nov 6, 2026
@youngrok-XCENA
youngrok-XCENA deleted the feat/remote-cxl-read-hints branch November 6, 2026 15:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants