RPC: [create_node] invalid data ptr — split scheduler ships compute-flagged views to the wrong RPC worker (reproduced with deepseek4 and glm-dsa, >=2 workers, master) #28047

Summary

FWIW, I found running large MoE models across two or more RPC workers crashes
deterministically during warmup when the model's graph contains
compute-flagged (GGML_TENSOR_FLAG_COMPUTE) views whose backing buffer lives
on a different RPC device than the node consuming them. The receiving worker
correctly rejects the foreign buffer handle in create_node
(result->buffer == nullptr && result->data != nullptr), closes the
connection, and the head aborts on recv failed / "Remote RPC server crashed".

I have reproduced this with two unrelated architectures, which I believe
localizes the bug to the split scheduler rather than either model's graph
code:

• DeepSeek-V4-Pro (deepseek4): the rejected tensor is a view of the
  per-layer CSA compressor state (dsv4_csa_state_kv_l16), allocated on
  model.dev_layer(il) by llama_dsv4_comp_state.
• GLM-5.3 (glm-dsa): the rejected tensor is a graph-time i32 view from
  the DSA indexer/top-k path (top_k-40, ne=[256,2,1,1]), failing at the
  same graph position (node 30) on every run.

Single-worker runs are always fine. The same configs on gpt-oss-20b (vanilla
MoE) and DeepSeek-V4-Flash (same family as Pro, no per-layer compressor
schedule) are fine with any worker count.

Reproduced on 74ce15741 and on current master (f1793c1c4), CUDA
(GB10/aarch64), over both TCP and RDMA transports — the transport is not a
factor.

Environment

• 6× NVIDIA DGX Spark (GB10, 128 GB unified each), aarch64, CUDA 13.0
• 1 head (llama-server) + 5 workers (ggml-rpc-server), 200 GbE
• Models: DeepSeek-V4-Pro Q4_K_M-XL (~940 GB), GLM-5.3 UD-Q5_K_XL (~562 GB,
  unsloth), -ngl 99 -fa on, layer split

Crash signature

Head:

ggml/src/ggml-rpc/ggml-rpc.cpp:566: Remote RPC server crashed or returned malformed response
(earlier builds: E recv failed (bytes_recv=0, size_to_recv=8), then ggml_abort in graph_compute)

Worker (error locally instrumented to print tensor identity):

# deepseek4:
[create_node] invalid data ptr: name=dsv4_csa_state_kv_l16 (view) op=VIEW type=f32
  ne=[1024,32,1,1] data=0xf4101ca00000 buffer_id=0xc7b4b5d9e320 view_src_id=0x... view_offs=0 flags=16
# glm-dsa:
[create_node] invalid data ptr: name=top_k-40 (view) op=VIEW type=i32
  ne=[256,2,1,1] ... view_offs=0 flags=16
[graph_compute] failed to create graph node 30

flags=16 = GGML_TENSOR_FLAG_COMPUTE in both cases: in-place state/compute
views that cannot be relocated to another backend without changing semantics.

Elimination table

All on the same hosts; "crash" = the worker-side rejection above.

Note GLM-5.3 needs no tensor overrides at all — a plain --rpc layer
split crashes — so this is not limited to -ot/--cpu-moe configurations.

Analysis

• The client-side serialize_tensor ships the view's rpc_tensor.buffer
  remote handle; when the scheduler assigns the consuming node to a different
  RPC worker than the one owning that buffer, the receiving worker cannot
  resolve the handle and rejects the node — correctly, as far as it goes.
• The real question is why ggml_backend_sched_split_graph admits a node
  whose pre-allocated FLAG_COMPUTE view input lives on another RPC device.
  ggml_backend_rpc_device_supports_op is return true (TODO in tree), so
  op-support can't steer assignment; supports_buft is endpoint-aware, but
  whatever path admits these nodes appears not to consult it for
  pre-allocated view inputs.
• A fix that pins consumers of pre-allocated compute-flagged views to the
  buffer-owning backend (or splits around them) would presumably cover both
  architectures.

Validated workaround (deepseek4 only)

Forcing the three llama_dsv4_comp_state constructions in
llama-kv-cache-dsv4.cpp to offload=false (state buffers on CPU instead of
model.dev_layer(il)) makes the previously-deterministic Pro config load and
generate correctly with 3 workers and cross-device expert spreads. The state
views are ~128 KB/layer, so the cost is negligible. This supports the
diagnosis (buffer locality × split scheduling), but it is model-specific
surgery — and no equivalent exists for glm-dsa, where the rejected view
is a transient graph tensor rather than persistent state. I'm happy to test
patches on the 6-node cluster; the GLM-5.3 repro is a plain --rpc launch
away and fails within the first warmup decode.

Side notes from the same investigation

1. -ot to RPC buffer types works but is underdocumented: --rpc must
   precede -ot, and buffer type names are RPC0[<endpoint>].
2. With --n-cpu-moe on a multi-device setup, the remaining GPU expert
   layers are a contiguous tail that all land on the last device — a note in
   the docs would save others the OOM loop.

Note: I used Fable to help debug this, create these tables, and such. That said all reproductions, the elimination
table, and the workaround were run and verified by me on the hardware above. I have Kimi-k2.6 running on 6 gb10 nodes at q8 and am trying to switch to GLM-5.3