Skip to content

DGX Spark: fix silent ARM codegen, add LlamaCppEx.RPC for two-node runs - #85

Merged
nyo16 merged 1 commit into
masterfrom
dgx-spark-two-node
Aug 14, 2026
Merged

DGX Spark: fix silent ARM codegen, add LlamaCppEx.RPC for two-node runs#85
nyo16 merged 1 commit into
masterfrom
dgx-spark-two-node

Conversation

@nyo16

@nyo16 nyo16 commented Aug 13, 2026

Copy link
Copy Markdown
Owner

Brings up NVIDIA DGX Spark (GB10, aarch64, sm_121a) as a first-class target,
and adds the ggml RPC backend so one model can span two machines.

4 commits, 38 files, +5863/−70. Each commit builds and passes on its own, so
git bisect works.

Why this is not just a "new hardware" PR

The most valuable change here affects every aarch64 user, not just Spark
owners: ggml's CPU backend was being compiled at base ARMv8-A on an ARMv9.2
part, silently. GCC 13.3 predates Cortex-X925/A725 and rejects
-mcpu=cortex-x925, so ggml's -mcpu=native probe degraded to the base
architecture behind a soft CMake warning and a zero exit status.

objdump -d on the emitted libggml-cpu.a:

build sdot smmla SVE
before (-mcpu=native) 0 0 none
after (-march=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2) 1134 370 present

Those are the Q4/Q8 quantized matmul kernels — i.e. most of the CPU backend's
throughput was missing and nothing said so.

What's in it

1. Build correctness (5f790b2)

  • LLAMA_CPU_ARM_ARCH names the CPU architecture instead of probing for it.
    Reaching it requires GGML_NATIVE=OFF, which drops ggml-cuda from one
    architecture to a 7-architecture fat binary, so LLAMA_CUDA_ARCH is now a
    hard $(error) when it's missing — a paired flag rather than a mysterious 6×
    build later.
  • A flag change did not invalidate the build. Toggling a variable changes
    CXXFLAGS/LDFLAGS/CMAKE_FLAGS while touching no source file, so make kept
    stale objects, links and CMakeCache.txt. Surfaced as
    Function not found 'Elixir.LlamaCppEx.NIF':model_load/11.
  • Every MIX_ENV shares one llama_cpp_ex_nif.so. Mix symlinks priv/ into
    each env's build tree, so dev/test/bench write one artifact while keeping
    separate objects. Building test with LLAMA_RPC=1 and bench without left
    whichever ran last in place, and the artifact's own timestamp looked current to
    the other env. Same hole let a downloaded precompiled artifact replace a source
    build. The link is now gated on a marker recording the config hash and a
    digest of the linked bytes
    .

2. LlamaCppEx.RPC (fd04df6) — opt-in via LLAMA_RPC=1. Without the flag
every entry point returns {:error, :rpc_unsupported} and a CPU/Metal build
compiles and loads unchanged.

3. Tooling, benchmarks, runbook (65ed68b) — scripts/spark/ (no sudo
anywhere), six benchmarks, and docs/dgx-spark.md as the one- or two-node
runbook.

4. Upstream defect checklist (0adec62) — three llama.cpp defects we work
around, recorded in the release guide's API-compatibility step with the command
that says whether each is still needed.

Results

model prefill decode
Qwen3-8B Q4_K_M 4331 t/s 40.5 t/s
Qwen3-30B-A3B Q4_K_M 3287 t/s 90.9 t/s

Three of four beat the published single-Spark references; the fourth is 7% under.

  • Prefer MoE on this chip. Qwen3.6-35B-A3B has more total parameters than
    the dense 27B and decodes 5.5× faster — only ~3B are active per token and
    decode is bandwidth-bound.
  • MTP pays on dense, not on sparse. 1.61× on Qwen3.6-27B dense
    (n_draft: 3), 1.03× on 35B-A3B. Speculative decoding spends compute to save
    bandwidth; an MoE has little bandwidth to save.
  • Two nodes buy capacity, not speed. A model that fits on one node measures
    the same either way (pipeline parallelism is disabled whenever an RPC device
    participates). A model that does not fit is the entire argument: Qwen3-235B-A22B
    Q4_K_M (142.1 GB) runs at 13.7 t/s across the pair, versus a global OOM
    on one node that also killed an unrelated system service.
  • RDMA is worth 11% of decode and 27% of prefill over TCP, and TCP is a
    survivable fallback.
  • -sm tensor across two hosts runs, is correct, and is 2.7× slower. No prior
    report of this combination exists. The CUDA all-reduce bails to nullptr on any
    non-CUDA member, and the generic path degrades the RPC backend's absent 2-D
    tensor hooks into a loop of 1-D transfers. Use :layer.
  • cpuidle tuning does nothing, one node or two, despite a 43× effect on ICMP.
    This retires the one recommendation that would have needed root.

Two negative results are kept on purpose.

Reviewer notes

The device-ordering trap is the easiest thing here to get wrong.
LlamaCppEx.devices/0 reports the ggml registry (RPC last); llama.cpp
rebuilds a different list for placement with RPC first. So gpu_index stops
indexing :tensor_split once a remote device is registered, and a backwards
split still produces correct tokens while benchmarking badly. That's why
:devices exists — used verbatim, no reordering — and why model_load went from
11 to 12 args across model.ex, nif.ex and llama_nif.cpp together.

Two upstream properties are documented rather than papered over:

  • A peer failure kills the BEAM. RPC_STATUS_ASSERT is GGML_ABORT, so there
    is no error to catch. Registration is the one checkable step
    ({:error, :unreachable}); after that the VM is the unit of restart. Real
    isolation means a sidecar OS process — out of scope.
  • RPC.Server.terminate/2 cannot stop the native server (upstream's accept loop
    is while (true) with no shutdown hook), so it's restart: :temporary and it
    traps exits so the warning actually reaches you on a supervised shutdown.

Verification

  • macOS (Metal, no RPC): 421 passed, mix credo --strict clean, mix dialyzer
    0 errors, mix format --check-formatted clean, mix docs link-clean.
  • DGX Spark (CUDA 13, LLAMA_RPC=1): 421 passed, 423 with --include rpc
    against a live two-node worker.

Known and deferred — a parallel review flagged 7 non-blocking warnings, none
on the shipped library path. The three worth picking up: orphaned cpuidle pollers
if a bench script dies mid-run; test/rpc_test.exs's @refusals set staying green
in exactly the stale-NIF case commit 1 prevents; and sync.sh having no models/
exclude.

The three upstream issues are drafted but not fileddocs/release-guide.md
has a slot for the URLs.

@nyo16
nyo16 force-pushed the dgx-spark-two-node branch from dd798c3 to 65ed68b Compare August 13, 2026 15:44
@nyo16 nyo16 changed the title Dgx spark two node DGX Spark: fix silent ARM codegen, add LlamaCppEx.RPC for two-node runs Aug 13, 2026
@nyo16
nyo16 force-pushed the dgx-spark-two-node branch from 0adec62 to 4690420 Compare August 13, 2026 19:56
Brings up NVIDIA DGX Spark (GB10, aarch64, sm_121a) as a first-class
target and adds the ggml RPC backend so one model can span two machines.
Every number below was measured on the hardware.

Build correctness. LLAMA_CPU_ARM_ARCH names the CPU architecture instead
of probing for it. GCC 13.3 predates Cortex-X925/A725 and rejects
-mcpu=cortex-x925, so ggml's -mcpu=native probe degraded to base ARMv8-A
behind a soft CMake warning and a zero exit status: the emitted
libggml-cpu.a had 0 sdot, 0 smmla and no SVE, against 1134 / 370 /
present once named. Those are the Q4/Q8 quantized matmul kernels, so
this affects every aarch64 user, not only Spark owners. Reaching it
requires GGML_NATIVE=OFF, which drops ggml-cuda to a 7-architecture fat
binary, so LLAMA_CUDA_ARCH is a paired flag rather than optional.

Two build-integrity bugs found while doing that, both silent. A flag
change did not invalidate the build: toggling a variable alters
CXXFLAGS, LDFLAGS or CMAKE_FLAGS while touching no source, so make kept
stale objects and shipped a NIF that did not match its own flags. And
every MIX_ENV shares one llama_cpp_ex_nif.so -- Mix symlinks priv/ into
each environment's build tree -- so building test with LLAMA_RPC=1 and
bench without it left whichever ran last in place, and a test suite
reported {:error, :rpc_unsupported} against a live worker the same tree
had just talked to. The same hole let a downloaded precompiled artifact
replace a source build. Both are now gated on a configuration stamp and
a marker recording the config hash plus a digest of the linked bytes.

LlamaCppEx.RPC, opt-in via LLAMA_RPC=1. add_server/1, add_servers/1,
devices/0, ping/1, supported?/0, and RPC.Server for the worker side.
Registration reports {:error, :unreachable} rather than vanishing,
because upstream collapses an unreachable endpoint and a protocol
mismatch into a null registration that ggml_backend_register ignores.
RPC.Server is restart: :temporary on purpose: the listening socket lives
on a detached thread in the same OS process, so it survives the
GenServer and a restart could not bind. Two upstream properties are
documented rather than papered over -- RPC_STATUS_ASSERT is GGML_ABORT,
so a peer failure takes the BEAM down, and llama.cpp rebuilds its own
device list with RPC devices first, so :tensor_split indexes an order
LlamaCppEx.devices/0 does not show.

llama.cpp bumped to a94d563ed (61 commits past b10362). Upstream removed
common_speculative_need_embd in f785fc9ea, so the MTP prefill no longer
requests logits at every prompt position -- the draft impls arrange
their own hidden-state extraction. Draft acceptance is identical to the
decimal at every depth after the bump (86.9 / 76.4 / 68.2 / 57.1 % for
n_draft 1-4), which is the claim worth making: the drafts themselves are
unchanged. llama_model_default_params also moved load_mode to the new
LLAMA_LOAD_MODE_AUTO; the NIF sets that field explicitly, so behaviour
did not change.

Tooling: scripts/spark/ for the remote loop (no sudo anywhere), six
benchmarks, and docs/dgx-spark.md as a runbook for one or two Sparks.
Two bugs fixed there: make outside mix wrote to /obj and would have
rm -rf'd it on clean, and rpc-worker.sh compiled inside its own
readiness window, reporting failure for a worker that came up fine.

Results. Qwen3.6-27B Q4_K_M 4331 t/s prefill, 40.5 t/s decode;
Qwen3.6-30B-A3B 3287 t/s, 90.9 t/s. MTP is worth 1.6x on the dense
model and nothing on sparse MoE, which spends compute to save bandwidth
an A3B does not need. Two nodes buy capacity, not speed: a 142 GB model
that OOMs one node runs at 13.7 t/s across the pair. -sm tensor works
across hosts and is 2.7x slower -- ggml_backend_cuda_comm_init returns
nullptr on any non-CUDA member, so the CUDA all-reduce falls to a
generic path and the RPC backend's NULL 2-D hooks degrade it to 1-D
transfers. Use :layer over RPC. cpuidle tuning does nothing despite a
43x effect on ICMP; that negative result is kept on purpose.

Verified: macOS (Metal) 423 passed, credo and dialyzer clean, docs
link-clean. DGX Spark (CUDA 13, LLAMA_RPC=1) 423 passed, 424 with
--include rpc_live against a live two-node worker, build flags 3/3.
@nyo16
nyo16 force-pushed the dgx-spark-two-node branch from 26bab9b to 8225990 Compare August 13, 2026 20:42
@nyo16
nyo16 merged commit 5f69d13 into master Aug 14, 2026
9 checks passed
@nyo16
nyo16 deleted the dgx-spark-two-node branch August 14, 2026 13:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant