diff --git a/.gitignore b/.gitignore index e3fcedc..027a278 100644 --- a/.gitignore +++ b/.gitignore @@ -26,6 +26,10 @@ llama_cpp_ex-*.tar /priv/*.so /priv/*.dylib /priv/*.dll +# Names the build configuration that produced llama_cpp_ex_nif.so. Mix symlinks +# priv/ into every MIX_ENV, so the artifact is shared and its own timestamp +# cannot tell you which flags built it. +/priv/.llama_cpp_ex_nif.built models/ # Dialyzer PLTs (see :dialyzer plt_local_path in mix.exs). diff --git a/CHANGELOG.md b/CHANGELOG.md index 4f31700..277b0fe 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,126 @@ # Changelog +## Unreleased + +DGX Spark (GB10) support: a silent ARM code-generation bug fixed, the ggml RPC +backend wired up so a model can span two machines, and a measured runbook for +both configurations in [docs/dgx-spark.md](docs/dgx-spark.md). + +Verified on macOS (Metal, no RPC): **423 passed, 143 excluded**. On a DGX Spark +(CUDA 13.0, `sm_121a`): **423 passed**, and **424 with `--include rpc_live`** +against a live two-node worker. + +### Fixed + +- **The ggml CPU backend was silently compiled at base ARMv8-A on GB10.** GCC + 13.3 predates Cortex-X925/A725 and rejects `-mcpu=cortex-x925`, so ggml's + `-mcpu=native` probe degraded to the base architecture behind a soft CMake + warning and a zero exit status. Measured on the emitted `libggml-cpu.a`: + **0 `sdot`, 0 `smmla`, no SVE**, versus 1134 / 370 / present once the + architecture is named. Those are the Q4/Q8 quantized matmul kernels. New + `LLAMA_CPU_ARM_ARCH` names it explicitly. +- **The build could ship a NIF that did not match its own flags.** Toggling a + build variable changes `CXXFLAGS`, `LDFLAGS` or `CMAKE_FLAGS` while touching no + source file, so make kept a stale object, a stale link, or a stale + `CMakeCache.txt`. Observed twice as + `Function not found 'Elixir.LlamaCppEx.NIF':model_load/11` and once as an RPC + build reporting `:rpc_unsupported` at runtime — all silent, all looking like + Elixir bugs. The cmake configure, the compile and the link now depend on a + configuration stamp whose name hashes all three flag sets. +- **Every `MIX_ENV` shares one `llama_cpp_ex_nif.so`, and they were overwriting + each other.** Mix symlinks `priv/` into each environment's build tree, so dev, + test and bench write one artifact while keeping separate objects and separate + llama.cpp trees. Building test with `LLAMA_RPC=1` and bench without it left + whichever ran last in place, and the artifact's own timestamp then looked + current to the other environment — a test suite reported + `{:error, :rpc_unsupported}` against a live RPC worker the same tree had just + talked to. The same hole let a downloaded precompiled artifact silently + replace a source build. The link is now gated on a marker beside the artifact + recording what it *is* — the configuration hash plus a digest of the linked + bytes — so any replacement, from any source, forces a relink. +- **`make` outside `mix` wrote to `/obj` and `/priv`.** `elixir_make` always sets + `MIX_APP_PATH`; a human running a documented command like + `LLAMA_RPC=1 make rpc-server` does not, and an empty value was silently + destructive rather than an error — `BUILD` became `/obj`, so cmake configured + into the filesystem root (`Unable to (re)create the private pkgRedirects + directory: /obj/rpc_server_build/CMakeFiles/pkgRedirects`) and `make clean` + would have tried to `rm -rf /obj`. It now falls back to + `_build/standalone/lib/llama_cpp_ex`, covering the defined-but-empty case too. +- **`scripts/spark/rpc-worker.sh start` compiled inside its own readiness + window.** The worker unit runs `mix run`, which compiles on demand, and the + wait loop cannot distinguish "still compiling" from "never going to listen" — + it only sees an active unit and a silent port. After a llama.cpp bump that + compile is a multi-minute rebuild, so the 120s budget expired, the script + reported a failure, and the worker came up fine on its own minutes later. The + build now happens before the unit is created, which also makes a compile error + arrive as a compile error. + +### Added + +- **`LlamaCppEx.RPC`** — register a remote machine's devices into the local + device registry so a model's layers can live on another host. + `add_server/1`, `add_servers/1`, `devices/0`, `ping/1`, `supported?/0`. + Registration reports `{:error, :unreachable}` rather than vanishing, which + matters because upstream collapses an unreachable endpoint and a protocol + mismatch into a null registration that `ggml_backend_register` silently + ignores. `supported?/0` reports whether the backend was compiled in at all, so + callers and tests never have to infer a build problem from `:rpc_unsupported`. +- **`LlamaCppEx.RPC.Server`** — the worker side, a supervised GenServer owning + the native server thread. `restart: :temporary`, because a restart could not + succeed: the listening socket lives on a detached thread in the same OS + process, so it survives the GenServer and the next bind would fail + `EADDRINUSE`. It traps exits so its "the native server is still running" + warning reaches you on a supervised shutdown, which is the case that matters. +- **`:rpc_servers`** on `Model.load/2` and `Server.start_link/1` — endpoints to + register before the load, in the order tensor placement needs. +- **`:devices`** on `Model.load/2` and `Server.start_link/1` — device names used + **verbatim** as the placement list. Not cosmetic: llama.cpp's automatic list + puts RPC devices first, which is *not* the order `LlamaCppEx.devices/0` + reports, so `:tensor_split` and `:main_gpu` otherwise index a list the caller + never saw. A backwards split still produces correct tokens and merely + benchmarks badly. +- **`split_mode: :tensor`** now encodes to llama.cpp's `LLAMA_SPLIT_MODE_TENSOR` + instead of raising `FunctionClauseError`, and an unknown split mode raises + `ArgumentError` naming the accepted set. +- **Build variables** `LLAMA_CPU_ARM_ARCH`, `LLAMA_CUDA_ARCH`, `LLAMA_RPC`, + `LLAMA_RPC_RDMA`, and a `make rpc-server` target for upstream's standalone + worker. `LLAMA_CPU_ARM_ARCH` without `LLAMA_CUDA_ARCH` on a CUDA build is a + hard `$(error)`: reaching the CPU flag needs `GGML_NATIVE=OFF`, which silently + turns one CUDA architecture into a seven-architecture fat binary. +- **`scripts/spark/`** — `bootstrap.sh`, `sync.sh`, `remote.sh`, + `verify-build-flags.sh`, `rpc-worker.sh`, `fetch_models.exs`, `rpc_check.exs`, + and the cpuidle matrices. No `sudo` anywhere. +- **Benchmarks** `bench/spark_baseline.exs`, `spark_two_node.exs`, + `spark_tuning.exs`, `spark_mtp.exs`, `spark_cpuidle.exs`, with results in + `bench/results/v0.8.43-dgx-spark-{baseline,two-node}.md`. + +### Changed + +- `Bench.Helpers.start_server/1` forwards any option the server declares instead + of a hand-maintained list of four, and raises on an option the server would + reject. A benchmark that silently drops the option it is measuring is worse + than no benchmark. +- `docs/multi-gpu.md` gains a remote-devices section, including the device + ordering trap and a worked `:devices` example. +- **llama.cpp bumped to `a94d563ed801`**, 61 commits past b10362. Upstream + removed `common_speculative_need_embd` in `f785fc9ea`: a draft implementation + that wants the target's hidden states now arranges its own extraction, so the + MTP prefill no longer requests logits at every prompt position — only at the + final token, which is the one we sample from. This is upstream's own migration; + `examples/speculative-simple` passes `false` for the whole prompt. + Verified by re-measuring rather than by reading: on the dense Qwen3.6-27B pair, + draft acceptance is **identical to the decimal** at every depth — 86.9 / 76.4 / + 68.2 / 57.1 % for `n_draft` 1–4, the same figures recorded at b10362 in + `bench/results/v0.8.43-dgx-spark-baseline.md`. Identical acceptance is the + claim worth making: it means the drafts themselves are unchanged, which a + throughput number alone cannot show. Decode moved 18.65 → 18.54 t/s at the + `n_draft: 3` peak, inside that run's recorded 18.4–18.7 range. + `llama_model_default_params()` also moved `load_mode` from + `LLAMA_LOAD_MODE_MMAP` to the new `LLAMA_LOAD_MODE_AUTO`, which drops mmap when + a device reports it cannot support it. The NIF always sets `load_mode` + explicitly from `:use_mmap`/`:use_mlock`/`:use_direct_io`, so behaviour is + unchanged and `:auto` is not exposed yet. + ## v0.8.43 llama.cpp bump to b10362, on top of b10280 from v0.8.42, plus the CUDA work that diff --git a/Makefile b/Makefile index f629757..f7a96d4 100644 --- a/Makefile +++ b/Makefile @@ -1,6 +1,20 @@ # Makefile for llama_cpp_ex NIF # Called by elixir_make during `mix compile` +# elixir_make always sets MIX_APP_PATH. A human running `make` directly does not, +# and an empty value is silently destructive rather than an error: PREFIX becomes +# /priv and BUILD becomes /obj, so `make clean` would try to rm -rf /obj and +# `make rpc-server` cmake-configures into it (observed: "Unable to (re)create the +# private pkgRedirects directory: /obj/rpc_server_build/CMakeFiles/pkgRedirects"). +# Falling back keeps every path inside the project. The directory name is not one +# of mix's environments on purpose -- it must not collide with a real build tree, +# and scripts/spark/rpc-worker.sh globs _build/*/lib/llama_cpp_ex so it is still +# found. Reading a variable (`make print-VAR`) never needed this. +# `?=` would miss the defined-but-empty case, which is the dangerous one. +ifeq ($(strip $(MIX_APP_PATH)),) +MIX_APP_PATH := _build/standalone/lib/llama_cpp_ex +endif + PREFIX = $(MIX_APP_PATH)/priv BUILD = $(MIX_APP_PATH)/obj NIF_SO = $(PREFIX)/llama_cpp_ex_nif.so @@ -22,7 +36,7 @@ endif # Pinned llama.cpp commit, used when vendor/llama.cpp has to be cloned. MUST # match the vendor/llama.cpp submodule; bump both together, see # docs/release-guide.md. Override to build the NIF against another revision. -LLAMA_COMMIT ?= 4801e3c567d5131dd41b387df5f2d4b1370d92be +LLAMA_COMMIT ?= a94d563ed801d1da1b8c2432946de07d0231bb3d # The commit actually on disk. A submodule can be bumped without LLAMA_COMMIT # following it, and the build has to key off what is really there. @@ -100,6 +114,47 @@ CUDA_LIBDIR := $(firstword $(wildcard $(CUDA_HOME)/lib64 $(CUDA_HOME)/lib)) # the same trade as -march=native, and the same answer. LLAMA_CUDA_NCCL ?= 0 +# The same discover-vs-declare hazard, one layer down and with a second flag +# chained to it. +# +# ggml probes the host CPU with -mcpu=native (ggml/src/ggml-cpu/CMakeLists.txt) +# whenever GGML_NATIVE is ON, which is the default. On GB10 with GCC 13.3 that +# probe fails *silently* -- the compiler predates Cortex-X925/A725 and rejects +# -mcpu=cortex-x925, cmake emits a soft warning and exits 0, and the CPU backend +# is quietly built at base ARMv8-A. Measured on the emitted libggml-cpu.a: +# 0 sdot, 0 smmla, no SVE, versus 1134 sdot + 370 smmla + SVE once the +# architecture is named. Those are the Q4/Q8 quantized matmul kernels, so this +# is not a rounding error. +# +# Naming it requires GGML_NATIVE=OFF, and that is the trap: with GGML_NATIVE +# off, ggml-cuda stops using `native` and falls back to a seven-architecture fat +# binary (ggml/src/ggml-cuda/CMakeLists.txt:27-28), turning a 1m50s build into a +# ~6x one for no benefit on a machine with exactly one known GPU. So the two +# move together or not at all, and setting one without the other is an error +# rather than a surprise six minutes later. +# +# DGX Spark (GB10): +# LLAMA_CPU_ARM_ARCH=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 +# LLAMA_CUDA_ARCH=121a-real +LLAMA_CPU_ARM_ARCH ?= +LLAMA_CUDA_ARCH ?= + +# The ggml RPC backend, which lets a model's layers live on another host. Off by +# default: it is a networked attack surface and a protocol version coupling, and +# neither belongs in a build nobody asked for. +# +# GGML_RPC_RDMA gets the same declare-don't-discover treatment as NCCL, for +# exactly the same reason. ggml/src/ggml-rpc/CMakeLists.txt:11-22 turns it ON +# whenever libibverbs happens to be installed on the build host, so the same +# source produces a different artifact on a DGX than on a laptop, silently. And +# because that file links ibverbs with target_link_libraries -- invisible to the +# hand-assembled link line below -- libggml-rpc.a comes out carrying undefined +# ibv_* symbols and the NIF fails to load. Same failure mode as +# `undefined symbol: ncclAllReduce`, same fix: declare it here and pair the link +# flag. +LLAMA_RPC ?= 0 +LLAMA_RPC_RDMA ?= 1 + # Backend selection (auto, metal, cuda, vulkan, cpu) LLAMA_BACKEND ?= auto @@ -149,6 +204,32 @@ ifneq (,$(filter -DGGML_CUDA=ON,$(CMAKE_FLAGS))) endif endif +# Outside the CUDA block on purpose: RPC is backend-independent, and a CPU-only +# worker node is a legitimate configuration. Both states are stated explicitly +# so the cmake configuration and the link line below provably agree. +ifneq ($(filter 1 true yes,$(LLAMA_RPC)),) + CMAKE_FLAGS += -DGGML_RPC=ON + # ggml/src/CMakeLists.txt:318-321 puts GGML_USE_RPC on the cmake `ggml` target + # as a PUBLIC definition. The NIF is compiled by this file with a hand-written + # g++ line that carries only -I flags, so it never inherits cmake target + # properties and would compile its #ifdef GGML_USE_RPC blocks out while + # linking against a libggml-rpc.a that is definitely there. State it. + CXXFLAGS += -DGGML_USE_RPC + # RDMA is Linux-only upstream (the CMakeLists forces it off elsewhere), so + # asking for it on macOS would be a configure-time lie. + ifeq ($(UNAME_S),Linux) + ifneq ($(filter 1 true yes,$(LLAMA_RPC_RDMA)),) + CMAKE_FLAGS += -DGGML_RPC_RDMA=ON + else + CMAKE_FLAGS += -DGGML_RPC_RDMA=OFF + endif + else + CMAKE_FLAGS += -DGGML_RPC_RDMA=OFF + endif +else + CMAKE_FLAGS += -DGGML_RPC=OFF +endif + # Portable builds, for artifacts that leave this machine. ggml defaults # GGML_NATIVE to ON unless cross-compiling (vendor/llama.cpp/ggml/CMakeLists.txt), # which adds -march=native (ggml/src/ggml-cpu/CMakeLists.txt) and ties the binary @@ -160,6 +241,57 @@ ifneq ($(filter 1 true yes,$(LLAMA_PORTABLE)),) LLAMA_PORTABLE_SUFFIX = -portable endif +# The arch pair, emitted after LLAMA_PORTABLE so the two GGML_NATIVE=OFF +# sources are visible together. LLAMA_PORTABLE already turns native off for a +# different reason -- artifacts that leave this machine, on release runners with +# no GPU and therefore no CUDA arch to pin -- so the filter below is what keeps +# the two from double-emitting the flag. +ifneq ($(strip $(LLAMA_CPU_ARM_ARCH)),) + ifeq ($(strip $(LLAMA_CUDA_ARCH)),) + ifneq (,$(filter -DGGML_CUDA=ON,$(CMAKE_FLAGS))) + $(error LLAMA_CPU_ARM_ARCH requires GGML_NATIVE=OFF, which drops ggml-cuda \ + from one architecture to a seven-architecture fat binary. Set \ + LLAMA_CUDA_ARCH too (DGX Spark GB10: LLAMA_CUDA_ARCH=121a-real), or \ + unset LLAMA_CPU_ARM_ARCH) + endif + endif + ifeq (,$(filter -DGGML_NATIVE=OFF,$(CMAKE_FLAGS))) + CMAKE_FLAGS += -DGGML_NATIVE=OFF + endif + CMAKE_FLAGS += -DGGML_CPU_ARM_ARCH=$(LLAMA_CPU_ARM_ARCH) +endif + +ifneq ($(strip $(LLAMA_CUDA_ARCH)),) + ifneq (,$(filter -DGGML_CUDA=ON,$(CMAKE_FLAGS))) + CMAKE_FLAGS += -DCMAKE_CUDA_ARCHITECTURES=$(LLAMA_CUDA_ARCH) + endif +endif + +# Both of the above select a different set of emitted instructions from the same +# sources, so they belong in the build-directory key alongside the backend -- +# otherwise toggling one reuses the previous CMakeCache.txt and silently +# no-ops, which is the exact failure the key exists to prevent. A short hash +# keeps the directory name readable; empty values hash to nothing so today's +# unflagged builds keep their existing directory and stay cache hits. +# Immediate assignment: a recursive one would re-run the shell on every +# expansion of LLAMA_BUILD. +ifneq ($(strip $(LLAMA_CPU_ARM_ARCH))$(strip $(LLAMA_CUDA_ARCH)),) + LLAMA_ARCH_SUFFIX := -$(shell printf '%s|%s' '$(LLAMA_CPU_ARM_ARCH)' '$(LLAMA_CUDA_ARCH)' \ + | { shasum -a 256 2>/dev/null || sha256sum; } | cut -c1-8) +endif + +# RPC adds a whole backend library plus the public GGML_USE_RPC define, and the +# RDMA toggle changes the code inside it, so both are part of the key too. +# Spelled out rather than hashed: these two show up in every two-node runbook +# and a readable directory name is worth more than eight characters. +ifneq ($(filter 1 true yes,$(LLAMA_RPC)),) + ifneq (,$(filter -DGGML_RPC_RDMA=ON,$(CMAKE_FLAGS))) + LLAMA_RPC_SUFFIX = -rpc + else + LLAMA_RPC_SUFFIX = -rpc-tcp + endif +endif + # Custom CMake args ifdef LLAMA_CMAKE_ARGS CMAKE_FLAGS += $(LLAMA_CMAKE_ARGS) @@ -168,10 +300,11 @@ endif # Build layout. Every key here is load-bearing, because each one selects a # different cmake configuration or a different set of sources: reusing one build # tree across them is what made submodule bumps and backend switches silently -# no-op. The directory carries the backend and portability, so a switch gets a -# clean CMakeCache.txt (and switching back is still a cache hit); the stamp -# carries the llama.cpp commit, so a bump forces a rebuild in place. -LLAMA_BUILD = $(BUILD)/llama_build-$(LLAMA_BACKEND)$(LLAMA_PORTABLE_SUFFIX) +# no-op. The directory carries the backend, portability, the architecture flags +# and RPC, so a switch gets a clean CMakeCache.txt (and switching back is still +# a cache hit); the stamp carries the llama.cpp commit, so a bump forces a +# rebuild in place. +LLAMA_BUILD = $(BUILD)/llama_build-$(LLAMA_BACKEND)$(LLAMA_PORTABLE_SUFFIX)$(LLAMA_ARCH_SUFFIX)$(LLAMA_RPC_SUFFIX) LLAMA_STAMP = $(LLAMA_BUILD)/.built-$(LLAMA_SHA_SHORT) # Platform-specific linker flags @@ -187,6 +320,13 @@ else ifneq ($(shell $(CXX) -fopenmp -E - < /dev/null 2>/dev/null && echo yes),) LDFLAGS += -lgomp endif + # Pairs with -DGGML_RPC_RDMA=ON above. ggml-rpc's CMakeLists links ibverbs + # with target_link_libraries, which this hand-assembled link line never sees, + # so without this libggml-rpc.a's undefined ibv_* symbols surface as a NIF + # that will not dlopen. + ifneq (,$(filter -DGGML_RPC_RDMA=ON,$(CMAKE_FLAGS))) + LDFLAGS += -libverbs + endif # ggml-cuda.a leaves the CUDA runtime, cuBLAS/cuBLASLt and the CUDA driver API # unresolved, but this line only ever added -lstdc++ -lm -lpthread. The .so # then linked and failed at load with `undefined symbol: cuMemCreate`, a @@ -221,10 +361,97 @@ NPROC := $(shell nproc 2>/dev/null || sysctl -n hw.ncpu 2>/dev/null || echo 4) NIF_SRC = c_src/llama_cpp_ex/llama_nif.cpp NIF_OBJ = $(BUILD)/llama_nif.o +# Everything downstream depends on the build *configuration*, not only on file +# timestamps, and three separate rules were blind to it: +# +# - Toggling LLAMA_RPC changes CXXFLAGS (-DGGML_USE_RPC) and the archive set +# while touching no source file, so make kept a previously linked .so. +# Observed twice: once as `Function not found model_load/11`, once as an RPC +# build reporting :rpc_unsupported at runtime. Silent, and it looks like an +# Elixir bug. +# - LDFLAGS changes (-lnccl, -libverbs) are invisible to every source rule. +# - A CMAKE_FLAGS change that does not alter the build directory -- NCCL, for +# instance -- left $(LLAMA_STAMP) newer than CMakeLists.txt, so cmake was +# never re-run and the archives kept the previous configuration. +# +# This stamp's *name* carries a hash of all three, so any change makes the +# prerequisite disappear and the cmake configure, the compile and the link all +# rerun. cmake is incremental, so the cost of a reconfigure is small and the +# alternative is a build that silently does not match its flags. +LLAMA_CONFIG_HASH := $(shell printf '%s|%s|%s|%s' \ + '$(CXXFLAGS)' '$(LDFLAGS)' '$(CMAKE_FLAGS)' '$(LLAMA_BUILD)' \ + | { shasum -a 256 2>/dev/null || sha256sum; } | cut -c1-12) +LLAMA_CONFIG_STAMP = $(BUILD)/.config-$(LLAMA_CONFIG_HASH) + +# ...and one more marker, next to the artifact rather than next to the objects, +# because the artifact is SHARED and gets overwritten from two directions. +# +# Mix symlinks $(MIX_APP_PATH)/priv to the project's priv/ in every MIX_ENV, so +# dev, test and bench all write one llama_cpp_ex_nif.so while each keeps its own +# objects and its own llama.cpp tree. Two consequences, both observed: +# +# 1. Build test with LLAMA_RPC=1 and bench without it, and whichever ran last +# is what every environment loads. The plain timestamp rule then sees a .so +# newer than its own object and skips the relink, so it stays wrong -- +# a test suite reporting {:error, :rpc_unsupported} against a live RPC +# worker the same tree had just talked to. +# 2. Mix restores a downloaded precompiled artifact into priv/ and copies it +# over the source build, with a fresh mtime, so no timestamp comparison can +# see it either. +# +# So the marker records what the artifact IS, not when it was written: the +# configuration hash plus a digest of the bytes that were linked. Anything that +# replaces the artifact -- another MIX_ENV, a downloaded release, a stray cp -- +# breaks the digest and forces a relink. +NIF_LINK_STAMP = $(PREFIX)/.llama_cpp_ex_nif.built +# The file is an argument, not stdin: a redirect here would attach to `cut`. +NIF_DIGEST = { shasum -a 256 "$(NIF_SO)" 2>/dev/null || sha256sum "$(NIF_SO)"; } | cut -c1-32 + # Targets -.PHONY: all clean +.PHONY: all clean rpc-server check-artifact + +# Serial, so check-artifact provably runs before the link decision below. +# Nothing here benefits from make's -j: the heavy build is cmake's own +# `--build -j$(NPROC)`. +.NOTPARALLEL: + +all: check-artifact $(NIF_LINK_STAMP) + +check-artifact: + @if [ -f "$(NIF_LINK_STAMP)" ]; then \ + want="$(LLAMA_CONFIG_HASH) $$($(NIF_DIGEST))"; \ + if [ "$$(cat "$(NIF_LINK_STAMP)")" != "$$want" ]; then \ + echo "==> $(NIF_SO) does not match this build; relinking"; \ + rm -f "$(NIF_LINK_STAMP)"; \ + fi; \ + fi + +# Upstream's standalone RPC worker. Not the production path -- LlamaCppEx.RPC.Server +# hosts the same server inside the NIF, under OTP supervision -- but when a +# two-node run misbehaves the first question is "is this us or upstream", and +# having both answers costs 1m32s of build time. +# +# Its own build directory, because it needs LLAMA_BUILD_TOOLS=ON and the NIF's +# tree deliberately does not build tools. The cmake target is `ggml-rpc-server`, +# not `rpc-server` (vendor/llama.cpp/tools/rpc/CMakeLists.txt:1). +RPC_SERVER_BUILD = $(BUILD)/rpc_server_build +RPC_SERVER_BIN = $(RPC_SERVER_BUILD)/bin/ggml-rpc-server + +rpc-server: $(RPC_SERVER_BIN) + @echo "built $(RPC_SERVER_BIN)" + +$(RPC_SERVER_BIN): $(LLAMA_DIR)/CMakeLists.txt + @test -n "$(filter 1 true yes,$(LLAMA_RPC))" || { \ + echo "error: rpc-server needs LLAMA_RPC=1"; exit 1; } + cmake -B $(RPC_SERVER_BUILD) -S $(LLAMA_DIR) $(CMAKE_FLAGS) -DLLAMA_BUILD_TOOLS=ON + cmake --build $(RPC_SERVER_BUILD) --config Release -j$(NPROC) --target ggml-rpc-server -all: $(NIF_SO) +# `make print-CMAKE_FLAGS` echoes one variable, fully expanded. `make -p` cannot +# do this: CMAKE_FLAGS is a recursive variable, so the database dump shows the +# unexpanded text. test/makefile_arch_flags_test.exs asserts on what this build +# will really hand to cmake, and that needs the expansion. +print-%: + @echo '$($*)' # Materialize vendor/llama.cpp when it is absent, which is the Hex-tarball case: # no vendor/ directory and no surrounding git repository. Pinned to LLAMA_COMMIT @@ -254,20 +481,30 @@ $(LLAMA_DIR)/CMakeLists.txt: @test -f $@ || { echo "error: clone finished but $@ is still missing"; exit 1; } # Build llama.cpp static libraries -$(LLAMA_STAMP): $(LLAMA_DIR)/CMakeLists.txt +$(LLAMA_STAMP): $(LLAMA_DIR)/CMakeLists.txt $(LLAMA_CONFIG_STAMP) @mkdir -p $(LLAMA_BUILD) cmake -B $(LLAMA_BUILD) -S $(LLAMA_DIR) $(CMAKE_FLAGS) cmake --build $(LLAMA_BUILD) --config Release -j$(NPROC) @rm -f $(LLAMA_BUILD)/.built-* @touch $@ +# Regenerated whenever the configuration hash changes, which is what forces the +# compile and the link below to rerun. Old stamps are swept so $(BUILD) does not +# accumulate one per configuration ever tried. +$(LLAMA_CONFIG_STAMP): + @mkdir -p $(dir $@) + @rm -f $(BUILD)/.config-* + @touch $@ + # Compile NIF -$(NIF_OBJ): $(NIF_SRC) c_src/llama_cpp_ex/llama_nif.h $(LLAMA_STAMP) +$(NIF_OBJ): $(NIF_SRC) c_src/llama_cpp_ex/llama_nif.h $(LLAMA_STAMP) $(LLAMA_CONFIG_STAMP) @mkdir -p $(dir $@) $(CXX) $(CXXFLAGS) -c $(NIF_SRC) -o $@ -# Link NIF - find all static libs from llama.cpp build -$(NIF_SO): $(NIF_OBJ) $(LLAMA_STAMP) +# Link NIF - find all static libs from llama.cpp build. +# The target is the marker, not the .so: see NIF_LINK_STAMP above for why the +# artifact's own timestamp cannot be trusted. +$(NIF_LINK_STAMP): $(NIF_OBJ) $(LLAMA_STAMP) $(LLAMA_CONFIG_STAMP) @mkdir -p $(PREFIX) @LIBS=$$(find $(LLAMA_BUILD) -name '*.a' \ ! -path '*/CMakeFiles/*' \ @@ -275,10 +512,11 @@ $(NIF_SO): $(NIF_OBJ) $(LLAMA_STAMP) ! -path '*/tests/*' \ | sort); \ if [ "$(UNAME_S)" = "Linux" ]; then \ - $(CXX) $(NIF_OBJ) -Wl,--start-group $$LIBS -Wl,--end-group $(LDFLAGS) -o $@; \ + $(CXX) $(NIF_OBJ) -Wl,--start-group $$LIBS -Wl,--end-group $(LDFLAGS) -o $(NIF_SO); \ else \ - $(CXX) $(NIF_OBJ) $$LIBS $(LDFLAGS) -o $@; \ + $(CXX) $(NIF_OBJ) $$LIBS $(LDFLAGS) -o $(NIF_SO); \ fi + @printf '%s %s' '$(LLAMA_CONFIG_HASH)' "$$($(NIF_DIGEST))" > $@ clean: - rm -rf $(BUILD) $(PREFIX)/llama_cpp_ex_nif.so + rm -rf $(BUILD) $(PREFIX)/llama_cpp_ex_nif.so $(NIF_LINK_STAMP) diff --git a/README.md b/README.md index 9c543ca..0948f35 100644 --- a/README.md +++ b/README.md @@ -112,7 +112,7 @@ Power users can pass arbitrary CMake flags: LLAMA_CMAKE_ARGS="-DGGML_CUDA_FORCE_CUBLAS=ON" mix compile ``` -Three more build variables: +More build variables: - `LLAMA_PORTABLE=1` drops `-march=native`. ggml turns it on by default, which tunes the binary to the exact CPU it was built on; the release workflow sets @@ -124,9 +124,28 @@ Three more build variables: the link line by hand rather than through cmake, that produced a NIF that failed to load with `undefined symbol: ncclAllReduce`. Turning it on also makes `libnccl.so.2` a load-time requirement. +- `LLAMA_CPU_ARM_ARCH=` names the architecture for ggml's CPU backend + instead of letting `-mcpu=native` probe for it. Needed wherever that probe + gives the wrong answer *silently* — on DGX Spark (GB10) with GCC 13.3 it + degrades to base ARMv8-A behind a soft warning, and the emitted + `libggml-cpu.a` loses every `sdot`, `smmla` and SVE instruction, i.e. the Q4/Q8 + quantized matmul kernels. On a CUDA build this **requires** `LLAMA_CUDA_ARCH` + and errors without it. +- `LLAMA_CUDA_ARCH=` sets `CMAKE_CUDA_ARCHITECTURES`, e.g. `121a-real` for + GB10. Reaching the CPU flag above needs `GGML_NATIVE=OFF`, which otherwise + turns one CUDA architecture into a seven-architecture fat binary — a ~6× build + with no runtime benefit. +- `LLAMA_RPC=1` builds the ggml RPC backend, which lets a model's layers live on + another machine. Off by default: it is a networked surface and a protocol + version coupling. `LLAMA_RPC_RDMA` (default `1` on Linux) declares whether the + transport may use RDMA, rather than letting ggml enable it based on whether + the build host happens to have `libibverbs`. See `LlamaCppEx.RPC`. - `LLAMA_COMMIT=` overrides the pinned llama.cpp commit used when `vendor/llama.cpp` has to be cloned. +On a DGX Spark, all of the above is `scripts/spark/remote.sh spark-1 mix compile` +— see [docs/dgx-spark.md](docs/dgx-spark.md) for the one- and two-node runbook. + ## Quick Start ```elixir diff --git a/bench/helpers.exs b/bench/helpers.exs index b4c4279..c6dc46f 100644 --- a/bench/helpers.exs +++ b/bench/helpers.exs @@ -25,28 +25,69 @@ defmodule Bench.Helpers do def start_server(opts \\ []) do model_path = System.get_env("LLAMA_MODEL_PATH") || raise "Set LLAMA_MODEL_PATH" - n_parallel = Keyword.get(opts, :n_parallel, 4) - server_opts = [ + defaults = [ model_path: model_path, - n_gpu_layers: Keyword.get(opts, :n_gpu_layers, -1), - n_parallel: n_parallel, - n_ctx: Keyword.get(opts, :n_ctx, 4096), + n_gpu_layers: -1, + n_parallel: 4, + n_ctx: 4096, temp: 0.0 ] - # Pass through new options - server_opts = - server_opts - |> maybe_add(opts, :cache_prompt) - |> maybe_add(opts, :batch_strategy) - |> maybe_add(opts, :kv_unified) - |> maybe_add(opts, :prompt_cache_ram_mb) + # Anything the server declares is forwardable. The list used to be four + # hand-maintained `maybe_add` calls, which meant every new tuning knob was + # silently dropped here until someone noticed — and a benchmark that ignores + # the option it is measuring is worse than no benchmark. + forwardable = LlamaCppEx.Server.start_option_keys() + {passthrough, unknown} = Keyword.split(opts, forwardable) - {:ok, server} = LlamaCppEx.Server.start_link(server_opts) + unknown = Keyword.drop(unknown, [:model_path]) + + if unknown != [] do + raise ArgumentError, + "Bench.Helpers.start_server/1 got options the server does not accept: " <> + "#{inspect(Keyword.keys(unknown))}" + end + + {:ok, server} = LlamaCppEx.Server.start_link(Keyword.merge(defaults, passthrough)) server end + @doc """ + Blocks until a server has finished loading, and returns the model. + + `Server.start_link/1` returns *before* the load: `init/1` stays cheap and + `handle_continue/2` does the work. `Server.get_model/1` is not a wait — it + raises `ArgumentError` as soon as one blocking timeout elapses, which a 63 GB + model on a cold page cache comfortably outlives. A benchmark that used it + instead lost the caller mid-load and took the VM down with + `CUDA error: driver shutting down`. + + Timing a load therefore means timing this call, not `start_link/1`. + """ + def await_model(server, timeout_ms \\ 900_000) do + deadline = System.monotonic_time(:millisecond) + timeout_ms + do_await_model(server, deadline) + end + + defp do_await_model(server, deadline) do + case LlamaCppEx.Server.fetch_model(server) do + {:ok, model} -> + model + + {:error, :not_ready} -> + if System.monotonic_time(:millisecond) < deadline do + Process.sleep(200) + do_await_model(server, deadline) + else + raise "model still loading after the deadline" + end + + {:error, reason} -> + raise "server failed to load its model: #{inspect(reason)}" + end + end + @doc """ The short-prompt suite: roughly 6, 110 and 220 tokens. @@ -138,11 +179,4 @@ defmodule Bench.Helpers do |> Enum.take(div(n_tokens, 4) + 16) |> IO.iodata_to_binary() end - - defp maybe_add(server_opts, source_opts, key) do - case Keyword.fetch(source_opts, key) do - {:ok, val} -> Keyword.put(server_opts, key, val) - :error -> server_opts - end - end end diff --git a/bench/results/v0.8.43-dgx-spark-baseline.md b/bench/results/v0.8.43-dgx-spark-baseline.md new file mode 100644 index 0000000..2679d22 --- /dev/null +++ b/bench/results/v0.8.43-dgx-spark-baseline.md @@ -0,0 +1,310 @@ +# Benchmark Results — DGX Spark (GB10) single-node baseline `[perf]` + +The first numbers for this library on NVIDIA DGX Spark, taken **after** the ARM +CPU architecture fix in the Makefile — before it, ggml's CPU backend was silently +compiled at base ARMv8-A with no `sdot`, no `smmla` and no SVE. + +- **Repo**: dgx-spark-2node, working tree over `a7814d35` +- **llama.cpp**: `4801e3c567d5` (b10362) +- **Hardware**: NVIDIA DGX Spark, GB10 Grace Blackwell (`sm_121a`), 121 GiB + unified memory, 20 cores (Cortex-X925 ×10 + A725 ×10) +- **Backend**: CUDA 13.0, driver 580.173.02 + (`LLAMA_BACKEND=cuda LLAMA_CPU_ARM_ARCH=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 LLAMA_CUDA_ARCH=121a-real`) +- **Runtime**: Elixir 1.20.2, Erlang/OTP 29, NIF 2.18 +- **Date**: 2026-08-12 + +**Prefill and decode are reported separately, and that is the whole point.** On +this chip they are two different machines: prefill is compute-bound and +excellent, decode is bandwidth-bound and is the number people are disappointed +by. A single tokens-per-second figure averages them into something that +describes neither. + +The split is derived rather than instrumented, using only the public API: two +generations from the same prompt with prompt caching off differ by exactly K +decode steps, so `decode = K / (t(1+K) - t(1))` and +`prefill = n_prompt / (t(1) - one_decode_step)`. That is llama-bench's pp/tg +split without needing llama-bench. + +Reproduce: + +```bash +scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ + mix run bench/spark_baseline.exs +``` + +## Qwen3-8B Q4_K_M — dense + +Median of 3, 64 decode steps. + +| prompt | tokens | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| short | 5 | 32.8 | 615.8 | 40.54 | +| medium | 101 | 48.1 | 4331.4 | 40.38 | +| long | 201 | 72.8 | 4193.0 | 40.15 | +| 1k | 1024 | 318.1 | 3499.7 | 39.24 | +| 8k | 8192 | 2807.5 | 2949.3 | 33.45 | +| 32k | 32768 | 16516.1 | 1989.5 | 21.84 | + +## Qwen3-30B-A3B Q4_K_M — MoE + +| prompt | tokens | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| short | 5 | 35.1 | 207.9 | 90.85 | +| medium | 101 | 79.0 | 1486.8 | 90.06 | +| long | 201 | 94.7 | 2410.5 | 88.70 | +| 1k | 1024 | 323.2 | 3287.4 | 85.65 | +| 8k | 8192 | 2623.3 | 3141.3 | 65.02 | +| 32k | 32768 | 15511.8 | 2116.4 | 34.80 | + +## Against the published single-Spark numbers + +| model | metric | external | measured | delta | +|---|---|---|---|---| +| Qwen3-8B | pp512 | 3167 t/s | 3500–4331 t/s | **+10% to +37%** | +| Qwen3-8B | tg | 43.7 t/s | 40.5 t/s | −7.3% | +| Qwen3-30B-A3B | pp512 | 2541 t/s | 3287 t/s | **+29%** | +| Qwen3-30B-A3B | tg | 89.3 t/s | 90.9 t/s | +1.8% | + +Three of four beat the reference and the fourth is 7.3% under, comfortably +inside the 20% band that would demand an explanation. The 8B decode gap is +plausibly the server's per-request overhead against a bare `llama-bench` loop +plus a concurrent 28 MB/s model download on the same box; it was not chased +further because it is not a finding. + +The prefill numbers are the interesting half. They are what the ARM CPU fix and +the single-architecture CUDA build buy, and they say this hardware is much better +at prompt processing than the community summaries suggest. + +## Qwen3.6 — the current generation + +Same methodology. Both from `unsloth/`. + +### Qwen3.6-27B Q4_K_M — dense + +| prompt | tokens | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| short | 5 | 104.0 | 247.4 | 11.93 | +| medium | 101 | 158.6 | 1352.1 | 11.92 | +| long | 201 | 281.3 | 1019.6 | 11.89 | +| 1k | 1024 | 1244.3 | 882.7 | 11.88 | +| 8k | 8192 | 10112.9 | 817.1 | 11.54 | +| 32k | 32768 | 44008.9 | 746.1 | 10.92 | + +### Qwen3.6-35B-A3B UD-Q4_K_M — MoE + +| prompt | tokens | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| short | 5 | 44.8 | 166.4 | 67.71 | +| medium | 101 | 90.0 | 1347.9 | 66.45 | +| long | 201 | 119.5 | 1930.1 | 65.26 | +| 1k | 1024 | 440.7 | 2406.6 | 65.56 | +| 8k | 8192 | 3398.9 | 2421.8 | 61.40 | +| 32k | 32768 | 14646.2 | 2240.1 | 54.40 | + +The dense/sparse gap is the story of this hardware in one comparison. The MoE +has **more** total parameters than the dense model and decodes **5.5× faster**, +because only ~3B are active per token and decode is bandwidth-bound. Prefill +goes the other way at short prompts (both are compute-bound there) and then the +MoE pulls ahead as the batch grows. + +Decode also barely degrades with context on the MoE — 67.7 → 54.4 t/s from 5 to +32768 tokens — while the dense 27B is flat at ~11.9 simply because it is +bandwidth-saturated at every context length. + +**On a Spark, prefer MoE.** A 35B-A3B gives better quality per token than an 8B +and decodes at two-thirds the speed of it. + +## Speculative decoding (MTP) + +The Qwen3.6 MTP repos are the same weights plus a Multi-Token Prediction head, +read only when the model is loaded with `load_mtp: true`. That makes a clean A/B. +256-token greedy generations through the chat template, **median of 5** with the +range alongside. + +Two methodology points, both of which changed a conclusion: + +- **Five samples, not one.** MTP is the noisier arm. A single run per setting + first suggested `n_draft: 2` beat `n_draft: 3` on the dense model; with n=5 + they are within noise of each other and 3 is marginally ahead. +- **The plain model is freed before the MTP arm.** Holding ~20 GB of unrelated + weights resident cost the MTP arm ~10% on this unified-memory part — enough to + turn the MoE result from "1.03× neutral" into "0.94× loss". + +| model | config | decode t/s | range | vs baseline | acceptance | +|---|---|---|---|---|---| +| 27B dense | no MTP head | 11.59 | 11.5–11.6 | — | — | +| 27B dense | `n_draft: 1` | 16.88 | 16.9–16.9 | 1.46× | 86.9% | +| 27B dense | `n_draft: 2` | 18.38 | 18.2–18.4 | 1.59× | 76.4% | +| 27B dense | **`n_draft: 3`** | **18.65** | 18.4–18.7 | **1.61×** | 68.2% | +| 27B dense | `n_draft: 4` | 17.42 | 17.3–17.5 | 1.50× | 57.1% | +| 35B-A3B MoE | no MTP head | 65.36 | 65.1–65.8 | — | — | +| 35B-A3B MoE | `n_draft: 1` | 67.56 | 67.0–67.7 | 1.03× | 81.0% | +| 35B-A3B MoE | `n_draft: 2` | 62.29 | 61.4–62.3 | 0.95× | 67.2% | +| 35B-A3B MoE | `n_draft: 3` | 63.28 | 63.0–63.5 | 0.97× | 64.9% | +| 35B-A3B MoE | `n_draft: 4` | 47.66 | 44.8–48.6 | 0.73× | 40.9% | + +No dense range overlaps its baseline, so **1.6× on the dense 27B is unambiguous**. +The MoE is neutral at best and loses past `n_draft: 1`. + +**MTP pays on dense models and roughly breaks even on sparse MoE.** Speculative +decoding spends compute (a batched verification pass) to save memory bandwidth +(sequential decode steps), which is exactly the trade this chip wants — but only +where decode is actually bandwidth-bound. On the 35B-A3B only ~3B parameters move +per token, so there is little to save and the draft-and-verify overhead eats it. + +Two caveats worth carrying: + +- **The best `n_draft` is model-shaped.** 3 on the dense model (2 within noise), + 1 on the MoE. `LlamaCppEx.MTP`'s moduledoc quotes ~2× at ~75% acceptance with + `n_draft: 3`; the acceptance rate reproduces (76.4% at depth 2 on dense) but + the speedup is 1.6×, not 2×. +- **Acceptance decays fast with depth** — 87% → 76% → 68% → 57% on the dense + model — so past the sweet spot you pay twice: wasted draft compute and a longer + verification batch. + +### This does not reproduce the README's MoE number + +The README reports **+16%** at `n_draft: 2` for Qwen3.6-35B-A3B on GB10, from an +interleaved n=11 run on **UD-Q4_K_XL**. The measurement above is UD-Q4_K_**M**. +The draft acceptance rates agree closely — 67.2% here versus 68.5% there at +`n_draft: 2` — so the drafting behaves the same and the throughput economics do +not. Quantization is the likely difference. Neither number is retracted here; +measure your own quantization before relying on either. + +### Qwen3.6 instruct checkpoints need the chat template + +`Qwen3.6-35B-A3B` emits end-of-generation **immediately** on a bare completion +prompt — zero tokens, from both the plain and the MTP path — while the identical +prompt inside the chat template generates normally. The 27B tolerates raw +completion. `bench/spark_mtp.exs` now templates via +`LlamaCppEx.Chat.apply_template/3` and refuses to report a zero-token baseline +rather than dividing by it. + +## cpuidle and core placement — a negative result + +The largest measured effect on this machine is cpuidle exit latency: LPI-3 exit +is 433 µs, ICMP RTT on the 200 Gb/s link reads 1.2 ms against a real 1.39 µs, and +holding the cores awake takes ICMP to 0.028 ms — a 43× effect. The obvious next +question is whether it also costs inter-token latency. + +It does not. + +Qwen3-8B, 101-token prompt, median of 7, 64 decode steps: + +| condition | TTFT median ms | TTFT worst ms | decode t/s | ms/token | +|---|---|---|---|---| +| a-default | 43.15 | 52.66 | 42.28 | 23.653 | +| b-poller (`nice -19` spinner per cpu) | 55.87 | 72.01 | 41.84 | 23.901 | +| c-BEAM busy-wait (`+sbwt very_long` …) | 44.58 | 51.56 | 42.23 | 23.678 | +| d-X925 cores (`taskset -c 5-9,15-19`) | 44.53 | 52.06 | 42.34 | 23.620 | +| e-X925 + BEAM busy-wait | 43.45 | 49.56 | 42.65 | 23.444 | + +Every condition is within 2% on decode, and the poller **hurts** TTFT by 29% — +twenty busy loops steal real cycles even at `nice -19`. The explanation is +straightforward in hindsight: a decode loop keeps the CPU continuously busy, so +it never enters a deep C-state and there is no exit latency to avoid. + +**So there is no reason to ask for `idle=poll` or cpuidle limits on the kernel +cmdline.** That was going to be the "needs your password" recommendation; the +measurement retired it. See the two-node results for the same question asked +again where a network wake is involved on every token — same answer. + +`taskset -c 5-9,15-19` is still worth using out of hygiene (it is where the big +cores are, and `-c 0-9` is entirely little cores), but it is not a performance +lever for inference on this box. + +## Runtime tuning matrix + +Qwen3-8B Q4_K_M, 1024-token prompt, 64 decode steps, one variable at a time. + +### Flash attention + +| setting | prefill t/s | decode t/s | decode vs baseline | +|---|---|---|---| +| `flash_attn: :auto` | 3487 | 41.61 | — | +| `flash_attn: :enabled` | 3493 | 40.97 | −1.5% | +| `flash_attn: :disabled` | 2644 | 39.76 | −4.4% | + +`:auto` already turns it on. Disabling it costs **24% of prefill**. Leave it +alone. + +### KV cache type + +| setting | prefill t/s | decode t/s | decode vs baseline | +|---|---|---|---| +| f16 (default) | 3487 | 40.24 | — | +| q8_0 | 3288 | 39.33 | −2.3% | +| q4_0 | 3335 | 37.63 | −6.5% | + +Quantizing the KV cache is a **loss** here, in both directions. On a discrete GPU +it buys VRAM headroom worth paying for; on 121 GiB of unified memory there is no +headroom to buy, so all that is left is the dequantization cost. + +### Load mode + +| setting | prefill t/s | decode t/s | decode vs baseline | +|---|---|---|---| +| mmap (default) | 3436 | 40.33 | — | +| mlock + mmap | 3461 | 38.76 | −3.9% | +| direct I/O | 3442 | 38.93 | −3.5% | +| no mmap | 3479 | 39.34 | −2.5% | + +Default mmap wins. `mlock` is pointless here for the same reason KV quantization +is: "pinned in RAM" and "resident on the GPU" are the same physical DRAM. + +### Offload + +| setting | prefill t/s | decode t/s | decode vs baseline | +|---|---|---|---| +| `n_gpu_layers: 99` | 3479 | 39.98 | — | +| `n_gpu_layers: 0` | 2209 | 17.11 | **−57.2%** | + +Unified memory does **not** make offload irrelevant. The bytes live in the same +DRAM either way, but which engine runs the matmuls still decides everything: +−57% decode, −37% prefill on the CPU. Always offload everything; there is never +a reason to partially offload on this hardware, because there is no VRAM ceiling +to ration against. + +### Batch size + +| setting | prefill t/s | decode t/s | decode vs baseline | +|---|---|---|---| +| `n_batch` default (2048) | 3504 | 39.70 | — | +| `n_batch: 512` | 3461 | 40.96 | +3.2% | +| `n_batch: 4096` | 3521 | 41.06 | +3.4% | +| `n_ubatch: 256` | 3562 | 40.99 | +3.3% | +| `n_ubatch: 1024` | 3506 | 41.19 | +3.8% | + +All within run-to-run noise (the cpuidle matrix above puts that at ±2%). No +effect worth acting on; the default is fine. + +### Concurrency + +| setting | prefill t/s | decode t/s per request | vs baseline | +|---|---|---|---| +| `n_parallel: 1` | 3490 | 39.85 | — | +| `n_parallel: 4` | 3459 | 40.00 | +0.4% | +| `n_parallel: 8` | 3392 | 39.67 | −0.5% | + +The best result in the matrix. Eight concurrent slots cost **nothing** +per-request, so aggregate throughput scales essentially linearly to 8×. If you +are serving, serve. + +## Use these settings on a Spark + +```elixir +LlamaCppEx.Server.start_link( + model_path: path, + n_gpu_layers: 99, # always; never partially offload + n_parallel: 8, # ~8x throughput for ~0 per-request cost + n_ctx: 4096 * 8 + # flash_attn: leave at :auto — it is already on, and off costs 24% prefill + # type_k/type_v: leave at f16 — quantizing the KV cache is a loss here + # use_mmap: leave at true — mlock and direct I/O both cost a few percent + # n_batch: leave at default — no measurable effect either way +) +``` + +Build with `LLAMA_CPU_ARM_ARCH` and `LLAMA_CUDA_ARCH` set; see +[docs/dgx-spark.md](../../docs/dgx-spark.md). Pin to the X925 cores with +`taskset -c 5-9,15-19` for hygiene, not for speed. diff --git a/bench/results/v0.8.43-dgx-spark-two-node.md b/bench/results/v0.8.43-dgx-spark-two-node.md new file mode 100644 index 0000000..c41bedf --- /dev/null +++ b/bench/results/v0.8.43-dgx-spark-two-node.md @@ -0,0 +1,207 @@ +# Benchmark Results — two DGX Sparks over the ggml RPC backend `[perf]` + +What a second DGX Spark buys, measured rather than assumed, including the cases +where the answer is "nothing" and the case where the answer is "the difference +between running and not running". + +- **Repo**: dgx-spark-2node, working tree over `a7814d35` +- **llama.cpp**: `4801e3c567d5` (b10362) +- **Hardware**: two DGX Sparks, GB10 (`sm_121a`), 121 GiB unified memory each, + direct-attached ConnectX-7 (RoCE v2, MTU 9000, PCIe Gen5 x4 → 13.98 GB/s) +- **Build**: `LLAMA_BACKEND=cuda LLAMA_RPC=1 LLAMA_RPC_RDMA=1`, + `LLAMA_CPU_ARM_ARCH=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 LLAMA_CUDA_ARCH=121a-real` +- **Topology**: `spark-1` client, `spark-2` worker on `10.100.64.2:50052`, + RDMA confirmed active on every run +- **Date**: 2026-08-12 + +Prefill and decode are separated by the same derivation as the single-node +results: two generations from one prompt with prompt caching off differ by +exactly K decode steps. + +Reproduce: + +```bash +scripts/spark/rpc-worker.sh start spark-2 +scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 --big-cores --forward-agent \ + spark-1 mix run bench/spark_two_node.exs b1 # b1 | b2 | b3 | b4 +scripts/spark/cpuidle-two-node.sh # b5 +``` + +## B2 — the headline: a model that does not fit on one node + +Qwen3-235B-A22B Q4_K_M, **142.1 GB** across three shards, against 130.0 GB of +unified memory per node. 512-token prompt, 32 decode steps. + +| run | load s | TTFT ms | prefill t/s | decode t/s | worker RSS | +|---|---|---|---|---|---| +| single-node, `use_mmap: true` | — | — | — | — | **global OOM** | +| two-node 50/50, RDMA | 538.4 | 1281.9 | 423.5 | **13.69** | 344 → 580 MiB | + +The single-node leg produces no number because it does not degrade — it OOMs: + +``` +oom-kill: constraint=CONSTRAINT_NONE, global_oom +Out of memory: Killed process 1599 (avahi-daemon) +NVRM: Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal @ mem_desc.c:1359 +``` + +The box survived; the OOM killer took an unrelated system service with it +(`avahi-daemon`, so the node stopped resolving over mDNS from the control node +while remaining perfectly healthy and reachable by IP). Unified memory is the +reason there is no graceful path: there is no separate VRAM to spill into, so +"offload everything" and "keep it in page cache" contend for the same 130 GB. + +It is off by default in the bench for that reason; `SPARK_INCLUDE_SINGLE=1` +reproduces it deliberately. + +**13.7 t/s on a 235B model is the number that justifies the second Spark**, and +it is the only one that does. The nine-minute cold load is ~71 GB of weights +crossing the fabric. + +## B1 — RPC overhead, controlled + +gpt-oss-120b MXFP4, 63.4 GB, fits one node. Anything the second node costs here +is pure RPC overhead. 1024-token prompt, 64 decode steps. + +| run | load s | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| single-node | 64.8 | 563.6 | 1889.0 | 46.50 | +| two-node 50/50 | 151.3 | 590.4 | 1797.6 | 48.14 | +| two-node, warm worker cache | 139.5 | 601.6 | 1766.6 | 45.48 | + +Decode across repeat runs landed in a 45–49 t/s band for **both** configurations, +so the honest reading is **no material difference**, not the +3.5% the first pair +suggests. + +That is worth dwelling on, because it is not what the mechanism predicts. +Pipeline parallelism is disabled whenever an RPC device participates (the RPC +backend reports `async = false, events = false`, and llama.cpp checks that before +enabling pipelining), so the two nodes execute *sequentially* — decode should get +worse. It does not, because splitting also halves each node's per-token weight +traffic, and on a bandwidth-bound part that relief roughly cancels the sequential +penalty. + +The real cost is **load time: 2.3× worse cold.** The worker's content-addressed +tensor cache took a warm load only from 151 s to 139 s, well short of the +near-parity hoped for, because the client still reads and hashes every tensor +locally to decide whether the worker already has it — only the transfer is +skipped. + +## B3 — RDMA versus TCP, in tokens + +There is no runtime switch for the transport. Selection is silent +auto-negotiation with no env var and no endpoint scheme; the only levers are +`GGML_RDMA_DEV` naming a device that does not exist, or a `LLAMA_RPC_RDMA=0` +build. Both ends must agree. + +| transport | load s | TTFT ms | prefill t/s | decode t/s | worker RSS | +|---|---|---|---|---|---| +| RDMA | 113.1 | 595.7 | 1784.0 | 46.05 | 350 → 597 MiB | +| TCP (forced) | 62.9 | 805.8 | 1310.5 | 40.90 | 348 → 588 MiB | +| delta | — | **+35%** | **−26.6%** | **−11.2%** | + +RDMA is worth real tokens. It also means the fallback is survivable: if RDMA +wedges (upstream issue #24813 — multi-node load deadlocking at the +`init_tensor`→`set_tensor` transition, closed as *stale, unfixed*, one week +before our pinned commit), forcing TCP costs about a tenth of decode rather than +the run. + +The load-time column is not a clean comparison — the worker's tensor cache was +already warm for the TCP leg — so read the three inference columns only. + +## B4 — batching amortisation + +Per-token RTT is fixed per *graph*, not per token, so a bigger batch should +amortise it. gpt-oss-120b, two-node, RDMA. + +| `n_parallel` | load s | TTFT ms | prefill t/s | decode t/s per request | worker RSS | +|---|---|---|---|---|---| +| 1 | 54.2 | 592.6 | 1788.9 | 49.48 | 349 → 595 MiB | +| 4 | 90.9 | 596.6 | 1783.3 | 44.71 | 595 → 595 MiB | +| 8 | 97.8 | 617.3 | 1722.0 | 44.25 | 595 → 595 MiB | + +Eight concurrent requests cost 11% of per-request decode for 8× the aggregate. +It amortises. + +The RSS column is the useful accident here: 349 → 595 MiB on the first client, +then **595 → 595** for the second and third. The worker retains ~245 MiB per +model share and never returns it, but within one worker lifetime it does *not* +accumulate per run. Restart between experiments anyway. + +## B5 — cpuidle, two-node + +Phase 3 found no cpuidle effect single-node. Two nodes is a fairer test: every +token now involves a network wake on the far side, and LPI-3 exit latency here is +433 µs — the effect that makes ICMP read 1.2 ms on a link whose real RTT is +1.39 µs. Pollers on **both** nodes. + +| condition | load s | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| default | 66.7 | 575.1 | 1845.1 | 49.81 | +| `nice -19` poller per cpu, both nodes | 53.1 | 597.6 | 1773.5 | 49.51 | + +−0.6% decode, +4% TTFT. Same answer as single-node: **no.** Both machines stay +busy enough during a generation that neither reaches a deep C-state, so there is +no exit latency to pay. The "ask the user for `idle=poll`" recommendation is +retired in both configurations. + +## The decode fast path + +Decode across an RPC device is only affordable because a repeated graph collapses +to a 4-byte `GRAPH_RECOMPUTE`; a cache miss re-serialises every tensor descriptor +on every token. The cache keys on the graph uid being unchanged, so a varying +batch shape would revert to the slow path. + +Worker journal over a two-node generation: **92 `graph_recompute`, zero +`graph_compute`**. `LlamaCppEx.Server`'s per-tick batch composition does not break +it. + +```bash +scripts/spark/rpc-worker.sh logs spark-2 4000 | grep -c graph_recompute +``` + +## Tensor parallelism ("tp=2") across two hosts + +Novel result — no prior report of this combination exists. Qwen3-8B, 512-token +prompt, 32 decode steps. + +Single node first, to price the Meta device itself: + +| configuration | prefill t/s | decode t/s | +|---|---|---| +| `split_mode: :none`, flash on | 3700.1 | 38.88 | +| `split_mode: :tensor`, one local GPU | 3865.5 | 38.46 (−1.1%) | + +Free. Now across two nodes: + +| configuration | load s | prefill t/s | decode t/s | vs layer split | +|---|---|---|---|---| +| `:layer`, 2 nodes | 4.7 | 3262.4 | 36.58 | — | +| `:tensor`, 2 nodes | 6.9 | 140.9 | 13.30 | **−63.6%** | +| `:tensor`, `GGML_CUDA_ALLREDUCE=none` | 6.0 | 137.4 | 13.59 | −63.2% | +| `:tensor`, `GGML_CUDA_ALLREDUCE=internal` | 6.2 | 277.6 | 10.33 | −71.8% | + +**It runs, and it is correct** — output byte-identical to the layer-split +reference — and it is 2.7× slower on decode, 23× slower on prefill. + +Two source facts explain it, and the second contradicts the obvious reading: + +1. `ggml_backend_cuda_comm_init` returns `nullptr` the moment **any** member + backend is not CUDA. An RPC device is not CUDA, so the CUDA all-reduce never + engages — which is why all three `GGML_CUDA_ALLREDUCE` settings land in the + same place. The knob cannot help. +2. The generic meta-backend butterfly that runs instead moves data with + `ggml_backend_tensor_{set,get}_2d`, and the RPC backend leaves both hooks + `NULL`. That is **not** fatal: `ggml-backend.cpp` falls back to a loop of + `n_copies` separate 1-D transfers. Every all-reduce becomes a burst of + individual network round trips, once per layer, per token. + +### Verdict + +Use `split_mode: :layer` over RPC. `:row` throws at load on CUDA; `:tensor` is +in-process tensor parallelism whose all-reduce is `ncclCommInitAll` (one distinct +physical GPU per rank, single process) and cannot span hosts as designed. Layer +split buys **capacity, not speed**. + +Re-check against a future llama.cpp bump using upstream `d6f303004`, +`adb541a6a`, `91fef9536`. diff --git a/bench/spark_baseline.exs b/bench/spark_baseline.exs new file mode 100644 index 0000000..79bcc91 --- /dev/null +++ b/bench/spark_baseline.exs @@ -0,0 +1,122 @@ +Code.require_file("helpers.exs", __DIR__) + +# Single-node baseline for a DGX Spark (GB10). +# +# scripts/spark/remote.sh --env MIX_ENV=bench spark-1 \ +# mix run bench/spark_baseline.exs +# +# Prefill and decode are reported **separately**, because on this chip they are +# two different machines. Prefill is compute-bound and excellent; decode is +# bandwidth-bound and the number people are disappointed by. A single +# tokens-per-second figure averages the two into something that describes +# neither, and the published Spark comparisons quote them apart. +# +# The separation uses only the public API: two generations from the same prompt +# with prompt caching off differ by exactly K decode steps, so +# +# decode = K / (t(1+K) - t(1)) +# prefill = n_prompt / (t(1) - one_decode_step) +# +# That is llama-bench's pp/tg split, derived rather than instrumented. + +n_ctx = 40_960 +decode_steps = 64 + +server = Bench.Helpers.start_server(n_parallel: 1, n_ctx: n_ctx, cache_prompt: false) +model = Bench.Helpers.await_model(server) + +defmodule SparkBaseline do + def time_ms(fun) do + t0 = System.monotonic_time(:microsecond) + fun.() + (System.monotonic_time(:microsecond) - t0) / 1000 + end + + # Median of `n` runs. Decode on a bandwidth-bound part is noisy enough that a + # single sample is not worth printing. + def median(values) do + sorted = Enum.sort(values) + len = length(sorted) + + case rem(len, 2) do + 1 -> Enum.at(sorted, div(len, 2)) + 0 -> (Enum.at(sorted, div(len, 2) - 1) + Enum.at(sorted, div(len, 2))) / 2 + end + end + + def split(server, prompt, n_prompt, decode_steps, samples) do + gen = fn max_tokens -> + {:ok, _} = LlamaCppEx.Server.generate(server, prompt, max_tokens: max_tokens) + end + + # Warm the kernels and any autotuning before the first timed run. + gen.(4) + + t_one = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1) end)) + t_many = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1 + decode_steps) end)) + + per_decode = (t_many - t_one) / decode_steps + prefill_ms = t_one - per_decode + + %{ + n_prompt: n_prompt, + prefill_ms: prefill_ms, + ttft_ms: t_one, + prefill_tps: n_prompt * 1000 / prefill_ms, + decode_tps: 1000 / per_decode + } + end +end + +# The short suite plus the >1k regime, same inputs as bench/server_generate.exs +# so the numbers stay comparable with everything under bench/results/. +inputs = + Bench.Helpers.prompts() + |> Map.merge(Bench.Helpers.long_prompts(model)) + |> Enum.map(fn {name, prompt} -> + {:ok, tokens} = LlamaCppEx.Tokenizer.encode(model, prompt) + {name, prompt, length(tokens)} + end) + |> Enum.sort_by(fn {_, _, n} -> n end) + +IO.puts("\nprefill / decode split (median of 3, #{decode_steps} decode steps)\n") +IO.puts("| prompt | tokens | TTFT ms | prefill t/s | decode t/s |") +IO.puts("|---|---|---|---|---|") + +for {name, prompt, n_prompt} <- inputs do + r = SparkBaseline.split(server, prompt, n_prompt, decode_steps, 3) + + IO.puts( + "| #{name} | #{r.n_prompt} | #{Float.round(r.ttft_ms, 1)} | " <> + "#{Float.round(r.prefill_tps, 1)} | #{Float.round(r.decode_tps, 2)} |" + ) +end + +IO.puts(""" + +External single-Spark references for comparison (llama.cpp, Q4_K_M): + Qwen3-8B pp512 3167 t/s tg 43.7 t/s + Qwen3-30B-A3B pp512 2541 t/s tg 89.3 t/s +A gap over 20% is a finding, not noise — explain it before tuning anything. +""") + +# The established Benchee suite, unchanged in shape from +# bench/server_generate.exs so the wall-clock numbers stay comparable. +benchee_inputs = Map.new(inputs, fn {name, prompt, _n} -> {name, {name, prompt}} end) + +Benchee.run( + %{ + "server generate 32 tokens" => fn {_name, prompt} -> + {:ok, _} = LlamaCppEx.Server.generate(server, prompt, max_tokens: 32) + end, + "server generate 128 tokens" => fn {_name, prompt} -> + {:ok, _} = LlamaCppEx.Server.generate(server, prompt, max_tokens: 128) + end + }, + inputs: benchee_inputs, + warmup: 1, + time: 10, + formatters: [{Benchee.Formatters.Console, extended_statistics: true}] +) + +GenServer.stop(server) diff --git a/bench/spark_cpuidle.exs b/bench/spark_cpuidle.exs new file mode 100644 index 0000000..8943648 --- /dev/null +++ b/bench/spark_cpuidle.exs @@ -0,0 +1,59 @@ +Code.require_file("helpers.exs", __DIR__) + +# Decode latency under one process-placement condition. Driven by +# scripts/spark/cpuidle-matrix.sh, which runs it once per condition; run it +# directly only to sanity-check a single setting. +# +# Deliberately narrow: TTFT and steady-state decode on a short prompt, which is +# where per-token wakeup latency shows up. A long prompt buries it under compute. +# +# The hypothesis under test is that cpuidle exit latency costs inter-token time. +# On these boxes LPI-3 exit is 433 us and holding cores out of deep idle takes +# ICMP RTT from 1.2 ms to 0.028 ms — a 43x effect on the network path. Whether it +# also costs decode is a different question, and this measures it rather than +# assuming either way. + +label = System.get_env("SPARK_COND") || "unlabelled" +samples = String.to_integer(System.get_env("SPARK_SAMPLES") || "7") +decode_steps = 64 + +server = Bench.Helpers.start_server(n_parallel: 1, n_ctx: 4096, cache_prompt: false) +model = Bench.Helpers.await_model(server) +prompt = Bench.Helpers.prompt_of_tokens(model, 101) + +time_ms = fn fun -> + t0 = System.monotonic_time(:microsecond) + fun.() + (System.monotonic_time(:microsecond) - t0) / 1000 +end + +median = fn values -> + sorted = Enum.sort(values) + len = length(sorted) + + case rem(len, 2) do + 1 -> Enum.at(sorted, div(len, 2)) + 0 -> (Enum.at(sorted, div(len, 2) - 1) + Enum.at(sorted, div(len, 2))) / 2 + end +end + +gen = fn n -> {:ok, _} = LlamaCppEx.Server.generate(server, prompt, max_tokens: n) end +gen.(4) + +ones = for _ <- 1..samples, do: time_ms.(fn -> gen.(1) end) +manys = for _ <- 1..samples, do: time_ms.(fn -> gen.(1 + decode_steps) end) + +t_one = median.(ones) +t_many = median.(manys) +per_decode = (t_many - t_one) / decode_steps + +# p99-ish tail on TTFT: the cpuidle story is a latency-tail story, so a median +# alone would hide exactly the effect being looked for. +worst_ttft = Enum.max(ones) + +IO.puts( + "RESULT\t#{label}\t#{Float.round(t_one, 2)}\t#{Float.round(worst_ttft, 2)}\t" <> + "#{Float.round(1000 / per_decode, 2)}\t#{Float.round(per_decode, 3)}" +) + +GenServer.stop(server) diff --git a/bench/spark_mtp.exs b/bench/spark_mtp.exs new file mode 100644 index 0000000..de15b4a --- /dev/null +++ b/bench/spark_mtp.exs @@ -0,0 +1,226 @@ +Code.require_file("helpers.exs", __DIR__) + +# What does Multi-Token Prediction buy on a DGX Spark? +# +# scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ +# mix run bench/spark_mtp.exs \ +# ~/models/unsloth/Qwen3.6-27B-GGUF/main/Qwen3.6-27B-Q4_K_M.gguf \ +# ~/models/unsloth/Qwen3.6-27B-MTP-GGUF/main/Qwen3.6-27B-Q4_K_M.gguf +# +# The two arguments are the same weights, one built without the MTP head and one +# with, which makes this a clean A/B rather than a comparison across quantizations. +# +# MTP should matter *more* here than on a bandwidth-rich part, and the reason is +# the shape of this chip: decode is memory-bandwidth-bound and prefill is +# compute-rich (measured at 3500-4300 t/s prefill against ~40 t/s decode on 8B). +# Speculative decoding converts decode steps into a batched verification pass — +# it spends the abundant resource to save the scarce one. Whether the draft +# acceptance rate is high enough to cash that in is the measurement. +# +# LlamaCppEx.MTP's own docs claim ~2x at ~75% acceptance with n_draft: 3 on +# Qwen 3.6. This is that claim, on this hardware. + +alias LlamaCppEx.{Model, MTP} + +{plain_path, mtp_path} = + case System.argv() do + [a, b | _] -> {a, b} + _ -> raise "usage: mix run bench/spark_mtp.exs " + end + +for p <- [plain_path, mtp_path] do + File.exists?(p) || raise "missing #{p}" +end + +raw_prompt = """ +Write a detailed technical explanation of how a memory-bandwidth-bound decode \ +loop differs from a compute-bound prefill pass in a transformer, and why that \ +distinction changes which optimisations are worth applying. +""" + +max_tokens = 256 + +# The chat template is not optional here. Qwen3.6-35B-A3B emits end-of-generation +# immediately when handed a bare completion prompt — measured: zero tokens, from +# both the plain and the MTP path — while the same prompt inside the template +# generates normally. The 27B tolerates raw completion, which is exactly the kind +# of difference that turns into a mystery if the harness does not template. +templated = fn model -> + case LlamaCppEx.Chat.apply_template(model, [%{role: "user", content: raw_prompt}]) do + {:ok, text} -> text + {:error, _} -> raw_prompt + end +end + +time_ms = fn fun -> + t0 = System.monotonic_time(:microsecond) + result = fun.() + {(System.monotonic_time(:microsecond) - t0) / 1000, result} +end + +median = fn values -> + s = Enum.sort(values) + n = length(s) + + if rem(n, 2) == 1, + do: Enum.at(s, div(n, 2)), + else: (Enum.at(s, div(n, 2) - 1) + Enum.at(s, div(n, 2))) / 2 +end + +# MTP is by far the noisier arm — acceptance varies run to run, so throughput +# does too. A single sample per setting is not enough to separate a real 1.4x +# from a lucky draw, which is exactly the trap the README's GB10 note avoids by +# interleaving n=11. Median of `reps`, and the range is reported alongside so a +# wide spread is visible rather than hidden behind the median. +reps = String.to_integer(System.get_env("SPARK_MTP_REPS") || "5") + +# --- Baseline: no MTP head, ordinary decode ---------------------------------- + +IO.puts("\n=== baseline (no MTP head)\n#{Path.basename(plain_path)}") + +# Scoped in a function so the plain model's NIF resource has no live reference +# once the baseline is done. Both models resident at once would put ~20 GB of +# unrelated weights in the way of the MTP arm on a unified-memory part, which is +# exactly the sort of confound that turns a 0.9x into a 1.1x. The explicit +# collection is what actually releases it: `fine` resources are freed by the GC, +# not at end of scope. +measure_baseline = fn -> + {:ok, plain} = Model.load(plain_path, n_gpu_layers: 99) + prompt = templated.(plain) + + # Warm the kernels. + {_, _} = time_ms.(fn -> LlamaCppEx.generate(plain, prompt, max_tokens: 8, temp: 0.0) end) + + samples = + for _ <- 1..reps do + {ms, {:ok, text}} = + time_ms.(fn -> LlamaCppEx.generate(plain, prompt, max_tokens: max_tokens, temp: 0.0) end) + + tokens = LlamaCppEx.Tokenizer.encode(plain, text) |> elem(1) |> length() + + if tokens == 0 do + raise """ + the baseline generated zero tokens. The model ended the sequence immediately, + which on a Qwen3.6 instruct checkpoint means the chat template did not apply — + check LlamaCppEx.Chat.apply_template/3 against #{Path.basename(plain_path)}. + """ + end + + tokens * 1000 / ms + end + + samples +end + +base_samples = measure_baseline.() +base_tps = median.(base_samples) + +:erlang.garbage_collect() +Process.sleep(500) + +IO.puts( + " #{Float.round(base_tps, 2)} t/s median of #{reps} " <> + "(#{Float.round(Enum.min(base_samples), 1)}-#{Float.round(Enum.max(base_samples), 1)})" +) + +# --- MTP: the same weights plus the head ------------------------------------- + +IO.puts("\n=== MTP (load_mtp: true)\n#{Path.basename(mtp_path)}") + +{:ok, mtp_model} = Model.load(mtp_path, n_gpu_layers: 99, load_mtp: true) +mtp_prompt = templated.(mtp_model) + +rows = + for n_draft <- [1, 2, 3, 4] do + # A fresh session per setting: MTP sessions hold two long-lived contexts and + # reusing one across configurations would carry KV state between them. + case MTP.init(mtp_model, n_draft: n_draft, n_ctx: 4096) do + {:ok, session} -> + {_, _} = time_ms.(fn -> MTP.generate(session, mtp_prompt, max_tokens: 8, temp: 0.0) end) + + samples = + for _ <- 1..reps do + {ms, result} = + time_ms.(fn -> + MTP.generate(session, mtp_prompt, max_tokens: max_tokens, temp: 0.0) + end) + + case result do + {:ok, text} -> + tokens = LlamaCppEx.Tokenizer.encode(mtp_model, text) |> elem(1) |> length() + {tokens * 1000 / ms, MTP.stats(session)} + + other -> + {:error, other} + end + end + + case Enum.reject(samples, &match?({:error, _}, &1)) do + [] -> + IO.puts(" n_draft=#{n_draft}: FAILED") + %{n_draft: n_draft, error: "generation failed"} + + ok -> + tps_values = Enum.map(ok, &elem(&1, 0)) + tps = median.(tps_values) + stats = ok |> List.last() |> elem(1) + + acceptance = + case stats do + %{acceptance_rate: r} when is_number(r) -> Float.round(r * 100, 1) + _ -> nil + end + + IO.puts( + " n_draft=#{n_draft}: #{Float.round(tps, 2)} t/s median of #{length(ok)} " <> + "(#{Float.round(Enum.min(tps_values), 1)}-#{Float.round(Enum.max(tps_values), 1)})" <> + if(acceptance, do: ", #{acceptance}% accepted", else: "") + ) + + %{ + n_draft: n_draft, + tps: tps, + lo: Enum.min(tps_values), + hi: Enum.max(tps_values), + acceptance: acceptance, + stats: stats + } + end + + {:error, reason} -> + IO.puts(" n_draft=#{n_draft}: init refused: #{inspect(reason)}") + %{n_draft: n_draft, error: inspect(reason)} + end + end + +IO.puts( + "\n| configuration | decode t/s (median of #{reps}) | range | vs baseline | draft acceptance |" +) + +IO.puts("|---|---|---|---|---|") + +IO.puts( + "| no MTP head | #{Float.round(base_tps, 2)} | " <> + "#{Float.round(Enum.min(base_samples), 1)}–#{Float.round(Enum.max(base_samples), 1)} | — | — |" +) + +for r <- rows do + if Map.has_key?(r, :error) do + IO.puts("| MTP n_draft=#{r.n_draft} | FAILED: #{r.error} | | | |") + else + speedup = Float.round(r.tps / base_tps, 2) + + IO.puts( + "| MTP n_draft=#{r.n_draft} | #{Float.round(r.tps, 2)} | " <> + "#{Float.round(r.lo, 1)}–#{Float.round(r.hi, 1)} | **#{speedup}x** | " <> + "#{if r.acceptance, do: "#{r.acceptance}%", else: "n/a"} |" + ) + end +end + +IO.puts("\nfull stats from the best run:") + +rows +|> Enum.reject(&Map.has_key?(&1, :error)) +|> Enum.max_by(& &1.tps, fn -> nil end) +|> IO.inspect() diff --git a/bench/spark_tensor_split.exs b/bench/spark_tensor_split.exs new file mode 100644 index 0000000..169f840 --- /dev/null +++ b/bench/spark_tensor_split.exs @@ -0,0 +1,221 @@ +Code.require_file("helpers.exs", __DIR__) + +# The tp=2 spike: does `split_mode: :tensor` do anything useful here? +# +# # 6.1 — single node, Meta device over one local GPU +# scripts/spark/remote.sh --env MIX_ENV=bench spark-1 \ +# mix run bench/spark_tensor_split.exs local +# +# # 6.2 — two nodes, Meta device over [CUDA0, RPC0] +# scripts/spark/rpc-worker.sh start spark-2 +# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \ +# mix run bench/spark_tensor_split.exs remote +# +# "tp=2" is vLLM vocabulary. In llama.cpp at b10362 it means +# LLAMA_SPLIT_MODE_TENSOR, added in #19378, which builds a Meta device wrapping +# N real devices. Everything interesting about it is in how the all-reduce is +# implemented: +# +# - ggml_backend_cuda_comm_init returns nullptr the moment ANY member backend +# is not CUDA (ggml-cuda.cu:1209-1213). With an RPC device in the set, the +# CUDA all-reduce — NCCL or the internal pipeline — never engages at all. +# - What runs instead is the meta backend's generic butterfly, which moves +# data with ggml_backend_tensor_{set,get}_2d. The RPC backend leaves both +# 2-D hooks NULL, and ggml-backend.cpp:360,382 then fall back to a LOOP of +# n_copies separate 1-D transfers. That is a performance cliff, not a +# failure — which is why this is worth running rather than reasoning about. +# +# GGML_CUDA_ALLREDUCE selects the CUDA path where one applies: nccl | internal | +# none. It is read in ggml_backend_cuda_comm_init and is the only lever. +# +# Time-box for the remote leg: 90 minutes. Capture the exact failure if it +# fails — the failure IS the deliverable. + +alias LlamaCppEx.Server + +mode = + case System.argv() do + [m | _] when m in ["local", "remote"] -> m + _ -> "local" + end + +endpoint = System.get_env("SPARK_RPC_ENDPOINT") || "10.100.64.2:50052" +model_path = System.get_env("LLAMA_MODEL_PATH") || raise "LLAMA_MODEL_PATH is required" +decode_steps = 32 +prompt_tokens = 512 + +defmodule Spike do + def time_ms(fun) do + t0 = System.monotonic_time(:microsecond) + fun.() + (System.monotonic_time(:microsecond) - t0) / 1000 + end + + def median(v) do + s = Enum.sort(v) + n = length(s) + + if rem(n, 2) == 1, + do: Enum.at(s, div(n, 2)), + else: (Enum.at(s, div(n, 2) - 1) + Enum.at(s, div(n, 2))) / 2 + end + + # Everything is wrapped, because the point of the exercise is to survive and + # report the failure rather than to succeed. A GGML_ABORT would take the VM + # with it and no rescue can help — that outcome is recorded from the outside, + # by the exit status and the tail of the log. + def attempt(label, opts, prompt_tokens, decode_steps) do + IO.puts("\n=== #{label}") + + try do + # start_link returns before the model is loaded (handle_continue/2 does + # the work), so the first GenServer.call is what waits. Timing has to + # include one, and here that also means an unsupported configuration + # surfaces as an exit from the call rather than a silent success. + load_ms = + time_ms(fn -> + {:ok, server} = Server.start_link(opts) + Process.put(:srv, {:ok, server}) + Process.put(:model, Bench.Helpers.await_model(server)) + end) + + case Process.get(:srv) do + {:ok, server} -> + model = Process.get(:model) + prompt = Bench.Helpers.prompt_of_tokens(model, prompt_tokens) + {:ok, tokens} = LlamaCppEx.Tokenizer.encode(model, prompt) + n_prompt = length(tokens) + + gen = fn n -> Server.generate(server, prompt, max_tokens: n) end + {:ok, _} = gen.(4) + + t_one = median(for _ <- 1..3, do: time_ms(fn -> {:ok, _} = gen.(1) end)) + t_many = median(for _ <- 1..3, do: time_ms(fn -> {:ok, _} = gen.(1 + decode_steps) end)) + per_decode = (t_many - t_one) / decode_steps + + {:ok, text} = gen.(16) + GenServer.stop(server) + + row = %{ + label: label, + load_s: load_ms / 1000, + prefill_tps: n_prompt * 1000 / (t_one - per_decode), + decode_tps: 1000 / per_decode, + sample: String.slice(text, 0, 40) + } + + IO.puts(" load #{Float.round(row.load_s, 1)} s") + IO.puts(" prefill #{Float.round(row.prefill_tps, 1)} t/s") + IO.puts(" decode #{Float.round(row.decode_tps, 2)} t/s") + IO.puts(" output #{inspect(row.sample)}") + row + + {:error, reason} -> + IO.puts(" REFUSED at load: #{inspect(reason)}") + %{label: label, error: inspect(reason)} + end + rescue + e -> + IO.puts(" RAISED: #{Exception.message(e)}") + %{label: label, error: Exception.message(e)} + catch + kind, value -> + IO.puts(" #{kind}: #{inspect(value)}") + %{label: label, error: "#{kind} #{inspect(value)}"} + end + end + + def table(rows, baseline_label) do + baseline = Enum.find(rows, &(&1.label == baseline_label && !Map.has_key?(&1, :error))) + + IO.puts("\n| configuration | load s | prefill t/s | decode t/s | vs baseline |") + IO.puts("|---|---|---|---|---|") + + for r <- rows do + if Map.has_key?(r, :error) do + IO.puts("| #{r.label} | — | — | — | **#{r.error}** |") + else + delta = + if baseline do + pct = (r.decode_tps / baseline.decode_tps - 1) * 100 + "#{if pct >= 0, do: "+", else: ""}#{Float.round(pct, 1)}%" + else + "-" + end + + IO.puts( + "| #{r.label} | #{Float.round(r.load_s, 1)} | #{Float.round(r.prefill_tps, 1)} | " <> + "#{Float.round(r.decode_tps, 2)} | #{delta} |" + ) + end + end + end +end + +base = [model_path: model_path, n_parallel: 1, n_ctx: 4096, temp: 0.0, cache_prompt: false] + +rows = + case mode do + "local" -> + # 6.1: does the Meta device cost anything at all with one local GPU, and + # is our architecture even accepted? llm_arch_supports_sm_tensor is a + # blocklist (llama-arch.cpp:1009-1042), so most architectures pass — + # qwen3 and gpt-oss both do. flash attention is force-enabled by the mode + # (llama-context.cpp), so :none is compared with it on to keep the + # comparison about the Meta device rather than about flash attention. + [ + Spike.attempt( + "split_mode :none (flash on)", + base ++ [split_mode: :none, flash_attn: :enabled], + prompt_tokens, + decode_steps + ), + Spike.attempt( + "split_mode :tensor, 1 local GPU", + base ++ [split_mode: :tensor], + prompt_tokens, + decode_steps + ) + ] + + "remote" -> + case LlamaCppEx.RPC.add_server(endpoint) do + {:ok, n} -> + IO.puts("registered #{endpoint}: #{n} device(s)") + + {:error, reason} -> + IO.puts(:stderr, "cannot reach #{endpoint}: #{inspect(reason)}") + System.halt(1) + end + + devices = LlamaCppEx.devices() + local = Enum.find(devices, &(&1.type in [:gpu, :igpu] and &1.backend != "RPC")) + remote = Enum.find(devices, &(&1.backend == "RPC")) + pair = [local.name, remote.name] + + IO.puts("Meta device over #{inspect(pair)}") + + IO.puts( + "GGML_CUDA_ALLREDUCE=#{System.get_env("GGML_CUDA_ALLREDUCE") || "(unset — Linux default is nccl)"}" + ) + + [ + Spike.attempt( + "layer split, 2 nodes (reference)", + base ++ [devices: pair, split_mode: :layer, tensor_split: [0.5, 0.5]], + prompt_tokens, + decode_steps + ), + Spike.attempt( + "tensor split, 2 nodes", + base ++ [devices: pair, split_mode: :tensor], + prompt_tokens, + decode_steps + ) + ] + end + +Spike.table( + rows, + if(mode == "local", do: "split_mode :none (flash on)", else: "layer split, 2 nodes (reference)") +) diff --git a/bench/spark_tuning.exs b/bench/spark_tuning.exs new file mode 100644 index 0000000..a586d22 --- /dev/null +++ b/bench/spark_tuning.exs @@ -0,0 +1,204 @@ +Code.require_file("helpers.exs", __DIR__) + +# One-variable-at-a-time runtime tuning matrix for a DGX Spark. +# +# scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ +# mix run bench/spark_tuning.exs +# +# Every row changes exactly one thing against the same baseline, so a number can +# be attributed. The deliverable is a short "use these settings" table, not a +# data dump — anything that moves less than a few percent gets reported as "no +# effect" and dropped from the recommendation. +# +# Unified memory makes several of these genuinely different from a discrete-GPU +# box: mlock and "offload to GPU" are the same physical DRAM here, so the usual +# advice about keeping weights off the host does not apply. + +alias LlamaCppEx.Server + +decode_steps = 64 +samples = 3 +prompt_tokens = 1024 + +defmodule Tuning do + def time_ms(fun) do + t0 = System.monotonic_time(:microsecond) + fun.() + (System.monotonic_time(:microsecond) - t0) / 1000 + end + + def median(values) do + sorted = Enum.sort(values) + len = length(sorted) + + case rem(len, 2) do + 1 -> Enum.at(sorted, div(len, 2)) + 0 -> (Enum.at(sorted, div(len, 2) - 1) + Enum.at(sorted, div(len, 2))) / 2 + end + end + + def run(label, opts, prompt_tokens, decode_steps, samples) do + IO.write(" #{String.pad_trailing(label, 26)}") + + try do + # Server.start_link/1 returns before the model is loaded — init/1 does + # only what is cheap and the load happens in handle_continue/2. So the + # first GenServer.call is what actually waits for it, and the timed region + # has to include one or the load time reads as zero. + load_ms = + time_ms(fn -> + {:ok, server} = Server.start_link(opts) + Process.put(:srv, server) + Process.put(:model, Bench.Helpers.await_model(server)) + end) + + server = Process.get(:srv) + model = Process.get(:model) + prompt = Bench.Helpers.prompt_of_tokens(model, prompt_tokens) + {:ok, tokens} = LlamaCppEx.Tokenizer.encode(model, prompt) + n_prompt = length(tokens) + + gen = fn n -> {:ok, _} = Server.generate(server, prompt, max_tokens: n) end + gen.(4) + + t_one = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1) end)) + t_many = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1 + decode_steps) end)) + per_decode = (t_many - t_one) / decode_steps + + GenServer.stop(server) + + row = %{ + label: label, + load_s: load_ms / 1000, + prefill_tps: n_prompt * 1000 / (t_one - per_decode), + decode_tps: 1000 / per_decode + } + + IO.puts( + "load #{Float.round(row.load_s, 1)}s " <> + "prefill #{Float.round(row.prefill_tps, 0)} t/s " <> + "decode #{Float.round(row.decode_tps, 2)} t/s" + ) + + row + rescue + e -> + IO.puts("FAILED: #{Exception.message(e)}") + %{label: label, error: Exception.message(e)} + end + end + + def table(title, rows, baseline_label) do + baseline = Enum.find(rows, &(&1.label == baseline_label)) + + IO.puts("\n### #{title}\n") + IO.puts("| setting | load s | prefill t/s | decode t/s | decode vs baseline |") + IO.puts("|---|---|---|---|---|") + + for r <- rows do + if Map.has_key?(r, :error) do + IO.puts("| #{r.label} | FAILED: #{r.error} | | | |") + else + delta = + if baseline && baseline[:decode_tps] do + pct = (r.decode_tps / baseline.decode_tps - 1) * 100 + "#{if pct >= 0, do: "+", else: ""}#{Float.round(pct, 1)}%" + else + "-" + end + + IO.puts( + "| #{r.label} | #{Float.round(r.load_s, 1)} | #{Float.round(r.prefill_tps, 0)} | " <> + "#{Float.round(r.decode_tps, 2)} | #{delta} |" + ) + end + end + end +end + +model_path = System.get_env("LLAMA_MODEL_PATH") || raise "LLAMA_MODEL_PATH is required" +base = [model_path: model_path, n_parallel: 1, n_ctx: 4096, temp: 0.0, cache_prompt: false] + +go = fn label, extra -> + Tuning.run(label, Keyword.merge(base, extra), prompt_tokens, decode_steps, samples) +end + +IO.puts( + "\n#{Path.basename(model_path)}, #{prompt_tokens}-token prompt, #{decode_steps} decode steps\n" +) + +# --- Batch sizes ------------------------------------------------------------- +# The server defaults n_batch to min(n_ctx, 2048). Prefill is the only thing +# that should care. +IO.puts("batch:") + +batch = + [ + go.("n_batch default", []), + go.("n_batch 512", n_batch: 512), + go.("n_batch 4096", n_batch: 4096), + go.("n_ubatch 256", n_ubatch: 256), + go.("n_ubatch 1024", n_ubatch: 1024) + ] + +# --- Flash attention --------------------------------------------------------- +IO.puts("\nflash attention:") + +flash = [ + go.("flash_attn auto", flash_attn: :auto), + go.("flash_attn enabled", flash_attn: :enabled), + go.("flash_attn disabled", flash_attn: :disabled) +] + +# --- KV cache type ----------------------------------------------------------- +# On unified memory the KV cache competes with the weights for the same DRAM +# bandwidth that decode is already limited by, so quantizing it is not only a +# capacity trade here. +IO.puts("\nKV cache type:") + +kv = [ + go.("KV f16 (default)", []), + go.("KV q8_0", type_k: :q8_0, type_v: :q8_0), + go.("KV q4_0", type_k: :q4_0, type_v: :q4_0) +] + +# --- Load mode --------------------------------------------------------------- +# mmap, mlock and direct I/O collapse into llama.cpp's single load_mode. On a +# unified-memory part "offloading" to the GPU and holding pages in RAM are the +# same physical memory, which is why mlock is worth measuring rather than +# dismissing. +IO.puts("\nload mode:") + +load_mode = [ + go.("mmap (default)", use_mmap: true), + go.("mlock + mmap", use_mmap: true, use_mlock: true), + go.("direct I/O", use_direct_io: true), + go.("no mmap", use_mmap: false) +] + +# --- Offload ----------------------------------------------------------------- +# There should be no reason to ever partially offload on this hardware: the GPU +# and the CPU address the same 121 GiB. Confirm, then say so in the docs. +IO.puts("\noffload:") + +offload = [ + go.("n_gpu_layers 99", n_gpu_layers: 99), + go.("n_gpu_layers 0 (CPU)", n_gpu_layers: 0) +] + +# --- Concurrency ------------------------------------------------------------- +# Throughput versus latency. Decode t/s here is per-request, so a drop with more +# slots is expected; the question is how gentle it is. +IO.puts("\nconcurrency:") + +parallel = + for n <- [1, 4, 8] do + go.("n_parallel #{n}", n_parallel: n, n_ctx: 4096 * n) + end + +Tuning.table("Batch sizes", batch, "n_batch default") +Tuning.table("Flash attention", flash, "flash_attn auto") +Tuning.table("KV cache type", kv, "KV f16 (default)") +Tuning.table("Load mode", load_mode, "mmap (default)") +Tuning.table("Offload", offload, "n_gpu_layers 99") +Tuning.table("Concurrency (decode t/s is per-request)", parallel, "n_parallel 1") diff --git a/bench/spark_two_node.exs b/bench/spark_two_node.exs new file mode 100644 index 0000000..516ef65 --- /dev/null +++ b/bench/spark_two_node.exs @@ -0,0 +1,399 @@ +Code.require_file("helpers.exs", __DIR__) + +# Two-node measurement suite for a pair of DGX Sparks. +# +# scripts/spark/rpc-worker.sh start spark-2 +# scripts/spark/remote.sh --env MIX_ENV=bench --env LLAMA_RPC=1 spark-1 \ +# mix run bench/spark_two_node.exs +# +# Benches, each answering one question: +# +# b1 RPC overhead, controlled. A model that FITS one node, run single-node +# and then layer-split across two. Isolates the cost of crossing the +# network from the benefit of more memory. Expect decode roughly +# unchanged and cold load materially worse. +# b2 The actual point. A model that does NOT fit one node: two-node RPC +# versus single-node paging weights off NVMe. This is the number that +# justifies the second Spark, and it is allowed to come out negative. +# b3 RDMA versus TCP, in tokens. There is no runtime switch, so TCP is +# forced with GGML_RDMA_DEV set to a device that does not exist. +# b4 Batching amortisation. Per-token RTT is fixed per *graph*, not per +# token, so n_parallel should amortise it. Measure, do not assume. +# +# Every run reports worker RSS before and after: upstream's worker is reported +# to grow and never release, and RSS is a column here rather than an afterthought. + +alias LlamaCppEx.Server + +endpoint = System.get_env("SPARK_RPC_ENDPOINT") || "10.100.64.2:50052" +# The worker's address on the fabric — not the control node's ssh alias, which +# does not resolve from here. +worker_node = System.get_env("SPARK_WORKER_HOST") || "10.100.64.2" + +which = + case System.argv() do + [w | _] -> w + [] -> "b1" + end + +defmodule TwoNode do + # Overridable because B2 is a different kind of run: a 142 GB model paged off + # NVMe on one node decodes slowly enough that the default 3x65 tokens would + # take the better part of an hour to say something a tenth of that already + # says. + @decode_steps String.to_integer(System.get_env("SPARK_DECODE_STEPS") || "64") + @samples String.to_integer(System.get_env("SPARK_SAMPLES") || "3") + + def time_ms(fun) do + t0 = System.monotonic_time(:microsecond) + fun.() + (System.monotonic_time(:microsecond) - t0) / 1000 + end + + def median([_ | _] = values) do + sorted = Enum.sort(values) + len = length(sorted) + + case rem(len, 2) do + 1 -> Enum.at(sorted, div(len, 2)) + 0 -> (Enum.at(sorted, div(len, 2) - 1) + Enum.at(sorted, div(len, 2))) / 2 + end + end + + # Same derivation as bench/spark_baseline.exs: two generations from one prompt + # with prompt caching off differ by exactly K decode steps. + def split(server, prompt, n_prompt, samples \\ @samples) do + gen = fn n -> {:ok, _} = Server.generate(server, prompt, max_tokens: n) end + gen.(4) + + t_one = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1) end)) + t_many = median(for _ <- 1..samples, do: time_ms(fn -> gen.(1 + @decode_steps) end)) + + per_decode = (t_many - t_one) / @decode_steps + + %{ + n_prompt: n_prompt, + ttft_ms: t_one, + prefill_tps: n_prompt * 1000 / (t_one - per_decode), + decode_tps: 1000 / per_decode + } + end + + # RSS of the remote worker, in MiB. The leak is on the other machine, so this + # has to cross the fabric: the client cannot see it, and the control node's + # ssh aliases do not resolve from here. Needs `remote.sh --forward-agent`, + # since the nodes hold no keys for each other. + # + # Returns nil rather than 0 when it cannot read — a missing measurement and a + # 0 MiB worker are very different claims, and one of them is a lie. + def worker_rss(host) do + args = [ + "-o", + "BatchMode=yes", + "-o", + "StrictHostKeyChecking=no", + "-o", + "UserKnownHostsFile=/dev/null", + "-o", + "ConnectTimeout=5", + host, + "pid=$(systemctl --user show llama-rpc-worker --property=MainPID --value); " <> + "awk '/^VmRSS:/{print int($2/1024)}' /proc/$pid/status 2>/dev/null" + ] + + # stderr stays separate: with UserKnownHostsFile=/dev/null ssh prints + # "Warning: Permanently added ..." on every connection, and merging it in + # made every reading unparseable — which then showed up as an honest-looking + # "n/a" instead of a number. + case System.cmd("ssh", args) do + {out, 0} -> + out + |> String.split("\n", trim: true) + |> List.last() + |> to_string() + |> Integer.parse() + |> case do + {mib, _} -> mib + :error -> nil + end + + _ -> + nil + end + rescue + _ -> nil + end + + def rss_str(nil), do: "n/a" + def rss_str(mib), do: "#{mib}" + + def measure(label, model_path, server_opts, prompt_tokens, worker_node) do + IO.puts("\n--- #{label}") + rss_before = worker_rss(worker_node) + + # Server.start_link/1 returns before the model is loaded: init/1 stays cheap + # and handle_continue/2 does the work, so the first GenServer.call is what + # waits for it. Load time is a headline number here — a two-node cold load + # pushes every remote tensor across the network — so the timed region has to + # include one call or it reads as zero. + load_ms = + time_ms(fn -> + {:ok, server} = Server.start_link(server_opts ++ [model_path: model_path]) + Process.put(:server, {:ok, server}) + Process.put(:model, Bench.Helpers.await_model(server)) + end) + + case Process.get(:server) do + {:ok, server} -> + model = Process.get(:model) + prompt = Bench.Helpers.prompt_of_tokens(model, prompt_tokens) + {:ok, tokens} = LlamaCppEx.Tokenizer.encode(model, prompt) + + result = split(server, prompt, length(tokens)) + rss_after = worker_rss(worker_node) + + GenServer.stop(server) + + row = + Map.merge(result, %{ + label: label, + load_ms: load_ms, + rss_before: rss_before, + rss_after: rss_after + }) + + report(row) + row + + {:error, reason} -> + IO.puts(" FAILED: #{inspect(reason)}") + %{label: label, error: reason} + end + end + + def report(r) do + IO.puts(" load #{Float.round(r.load_ms / 1000, 1)} s") + IO.puts(" prompt #{r.n_prompt} tokens") + IO.puts(" TTFT #{Float.round(r.ttft_ms, 1)} ms") + IO.puts(" prefill #{Float.round(r.prefill_tps, 1)} t/s") + IO.puts(" decode #{Float.round(r.decode_tps, 2)} t/s") + IO.puts(" worker RSS #{rss_str(r.rss_before)} -> #{rss_str(r.rss_after)} MiB") + end + + def table(rows) do + IO.puts("\n| run | load s | prompt | TTFT ms | prefill t/s | decode t/s | worker RSS MiB |") + IO.puts("|---|---|---|---|---|---|---|") + + for r <- rows do + if Map.has_key?(r, :error) do + IO.puts("| #{r.label} | FAILED: #{inspect(r.error)} | | | | | |") + else + IO.puts( + "| #{r.label} | #{Float.round(r.load_ms / 1000, 1)} | #{r.n_prompt} | " <> + "#{Float.round(r.ttft_ms, 1)} | #{Float.round(r.prefill_tps, 1)} | " <> + "#{Float.round(r.decode_tps, 2)} | #{rss_str(r.rss_before)} -> #{rss_str(r.rss_after)} |" + ) + end + end + end +end + +models = %{ + "120b" => + Path.join( + System.get_env("HOME"), + "models/ggml-org/gpt-oss-120b-GGUF/main/gpt-oss-120b-MXFP4.gguf" + ), + "235b" => + Path.join( + System.get_env("HOME"), + "models/unsloth/Qwen3-235B-A22B-GGUF/main/Q4_K_M/Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf" + ) +} + +# Register up front so the remote device exists before any load. Failing here is +# the whole reason add_server reports instead of aborting: after a load, a peer +# problem takes the VM with it. +register = fn -> + case LlamaCppEx.RPC.add_server(endpoint) do + {:ok, n} when n >= 1 -> + IO.puts("registered #{endpoint}: #{n} remote device(s)") + :ok + + {:ok, 0} -> + :ok + + {:error, reason} -> + IO.puts(:stderr, "cannot reach #{endpoint}: #{inspect(reason)}") + IO.puts(:stderr, "start it with: scripts/spark/rpc-worker.sh start #{worker_node}") + System.halt(1) + end +end + +# :devices is named explicitly in every two-node run. The automatic placement +# list puts RPC devices first, which is not the order LlamaCppEx.devices/0 +# reports, and a backwards split still produces correct tokens while +# benchmarking badly. Naming them makes the split mean what it says. +device_names = fn -> + devices = LlamaCppEx.devices() + local = Enum.find(devices, &(&1.type in [:gpu, :igpu] and &1.backend != "RPC")) + remote = Enum.find(devices, &(&1.backend == "RPC")) + {local.name, remote.name} +end + +n_ctx = 4096 +prompt_tokens = String.to_integer(System.get_env("SPARK_PROMPT_TOKENS") || "1024") + +case which do + "b1" -> + # Controlled: the model fits one node, so anything the second node costs is + # pure RPC overhead rather than a memory benefit. + path = models["120b"] + File.exists?(path) || raise "missing #{path}" + + single = + TwoNode.measure( + "120b single-node", + path, + [n_parallel: 1, n_ctx: n_ctx, cache_prompt: false], + prompt_tokens, + worker_node + ) + + register.() + {local, remote} = device_names.() + + two = + TwoNode.measure( + "120b two-node 50/50", + path, + [ + n_parallel: 1, + n_ctx: n_ctx, + cache_prompt: false, + devices: [local, remote], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ], + prompt_tokens, + worker_node + ) + + TwoNode.table([single, two]) + + "b2" -> + # The point. 142.1 GB of weights against 121 GiB (130.0 GB) of unified + # memory. + # + # The single-node leg is OFF by default, because it does not produce a + # number: it produces a global OOM. Measured on spark-1 — + # + # oom-kill: constraint=CONSTRAINT_NONE, global_oom + # Out of memory: Killed process 1599 (avahi-daemon) + # NVRM: Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal + # + # The box survived, but the OOM killer took out an unrelated system service + # (mDNS stopped resolving afterwards, from a different machine's point of + # view) and the run died. Unified memory is why: there is no separate VRAM + # to spill into, so "offload it all" and "keep it in RAM" compete for the + # same 130 GB and mmap cannot save you. Set SPARK_INCLUDE_SINGLE=1 to + # reproduce it deliberately. + path = models["235b"] + File.exists?(path) || raise "missing #{path} — run scripts/spark/fetch_models.exs 235b" + + single = + if System.get_env("SPARK_INCLUDE_SINGLE") == "1" do + [ + TwoNode.measure( + "235b single-node (mmap overflow)", + path, + [n_parallel: 1, n_ctx: n_ctx, cache_prompt: false, use_mmap: true], + prompt_tokens, + worker_node + ) + ] + else + IO.puts("\n--- 235b single-node: SKIPPED (OOMs the node; SPARK_INCLUDE_SINGLE=1 to try)") + [] + end + + register.() + {local, remote} = device_names.() + + two = + TwoNode.measure( + "235b two-node 50/50", + path, + [ + n_parallel: 1, + n_ctx: n_ctx, + cache_prompt: false, + devices: [local, remote], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ], + prompt_tokens, + worker_node + ) + + TwoNode.table(single ++ [two]) + + "b3" -> + # Transport A/B. Run this twice: once normally, once with the worker AND this + # process started under GGML_RDMA_DEV=nonexistent. There is no runtime + # switch — that variable and a -DGGML_RPC_RDMA=OFF build are the only levers. + path = models["120b"] + register.() + {local, remote} = device_names.() + + transport = if System.get_env("GGML_RDMA_DEV") in [nil, ""], do: "RDMA", else: "TCP (forced)" + + row = + TwoNode.measure( + "120b two-node over #{transport}", + path, + [ + n_parallel: 1, + n_ctx: n_ctx, + cache_prompt: false, + devices: [local, remote], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ], + prompt_tokens, + worker_node + ) + + TwoNode.table([row]) + + "b4" -> + # Per-token RTT is fixed per graph, not per token, so a bigger batch should + # amortise it. + path = models["120b"] + register.() + {local, remote} = device_names.() + + rows = + for n_parallel <- [1, 4, 8] do + TwoNode.measure( + "120b two-node n_parallel=#{n_parallel}", + path, + [ + n_parallel: n_parallel, + n_ctx: n_ctx * n_parallel, + cache_prompt: false, + devices: [local, remote], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ], + prompt_tokens, + worker_node + ) + end + + TwoNode.table(rows) + + other -> + IO.puts(:stderr, "unknown bench #{inspect(other)}; expected b1, b2, b3 or b4") + System.halt(2) +end diff --git a/c_src/llama_cpp_ex/llama_nif.cpp b/c_src/llama_cpp_ex/llama_nif.cpp index e80e17d..7a6bb9f 100644 --- a/c_src/llama_cpp_ex/llama_nif.cpp +++ b/c_src/llama_cpp_ex/llama_nif.cpp @@ -11,6 +11,19 @@ #include #include #include +#include + +// The ggml RPC backend is opt-in at build time (LLAMA_RPC=1). GGML_USE_RPC is +// set by the Makefile, not inherited from cmake: ggml puts it on the `ggml` +// target as a PUBLIC definition, and this translation unit is compiled by hand +// with only -I flags. +#ifdef GGML_USE_RPC +#include +#include +#include +#include +#include +#endif using namespace llama_cpp_ex; @@ -31,6 +44,10 @@ inline auto invalid_index = fine::Atom("invalid_index"); inline auto invalid_grammar = fine::Atom("invalid_grammar"); inline auto invalid_state = fine::Atom("invalid_state"); inline auto unsupported = fine::Atom("unsupported"); +inline auto rpc_unsupported = fine::Atom("rpc_unsupported"); +inline auto unreachable = fine::Atom("unreachable"); +inline auto no_devices = fine::Atom("no_devices"); +inline auto bind_timeout = fine::Atom("bind_timeout"); } // namespace atoms // --- Input validation at the NIF boundary --- @@ -252,6 +269,244 @@ fine::Ok<> backend_free(ErlNifEnv* env) { } FINE_NIF(backend_free, 0); +// --- RPC --- +// +// The ggml RPC backend puts a model's layers on another host. It is compiled in +// only when LLAMA_RPC=1; without it every function here reports +// {:error, :rpc_unsupported} so a CPU or Metal build keeps loading unchanged. +// +// Two upstream properties shape this API and neither is ours to fix: +// +// 1. RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30). Any peer crash, +// network failure or malformed response terminates the OS process — the +// whole BEAM. There is no error return, no retry, no reconnect. +// 2. Registration, by contrast, is safe: an unreachable endpoint makes +// ggml_backend_rpc_add_server return nullptr. So failures are *detectable +// before load* and *fatal during it*, and that asymmetry is why +// rpc_add_server reports rather than logs. + +#ifdef GGML_USE_RPC + +namespace { + +// "host:port", where host may be an IPv4 literal or a name. The RPC transport +// parses the same string itself; this copy exists only for the pre-flight bind. +bool rpc_split_endpoint(const std::string& endpoint, std::string& host, std::string& port) { + auto colon = endpoint.rfind(':'); + if (colon == std::string::npos || colon == 0 || colon + 1 == endpoint.size()) { + return false; + } + host = endpoint.substr(0, colon); + port = endpoint.substr(colon + 1); + return true; +} + +// Bind the endpoint, then immediately give it back. ggml_backend_rpc_start_server +// returns void and never returns at all on success, so it can report neither a +// bad host nor a port already in use — it prints to stderr and the thread just +// sits there. Doing the bind here first turns the common failures into an +// errno we can hand back to Elixir. It is a TOCTOU window, which is why the +// caller still waits for the real listener afterwards. +std::string rpc_preflight_bind(const std::string& host, const std::string& port) { + addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + + addrinfo* res = nullptr; + int rc = getaddrinfo(host.c_str(), port.c_str(), &hints, &res); + if (rc != 0) { + return std::string("cannot resolve ") + host + ": " + gai_strerror(rc); + } + + std::string error = "no usable address for " + host; + for (addrinfo* ai = res; ai; ai = ai->ai_next) { + int fd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (fd < 0) continue; + int one = 1; + setsockopt(fd, SOL_SOCKET, SO_REUSEADDR, &one, sizeof(one)); + if (bind(fd, ai->ai_addr, ai->ai_addrlen) == 0) { + close(fd); + error.clear(); + break; + } + error = std::string("cannot bind ") + host + ":" + port + ": " + std::strerror(errno); + close(fd); + } + freeaddrinfo(res); + return error; +} + +// Wait for something to accept a connection on the endpoint. The proof that the +// detached thread actually got as far as listen(), rather than printing to +// stderr and returning. The probe is closed before any HELLO is sent; the +// server's read then fails and it loops back to accept, which is harmless. +bool rpc_wait_until_listening(const std::string& host, const std::string& port, int timeout_ms) { + addrinfo hints{}; + hints.ai_family = AF_UNSPEC; + hints.ai_socktype = SOCK_STREAM; + + for (int waited = 0; waited < timeout_ms; waited += 20) { + addrinfo* res = nullptr; + if (getaddrinfo(host.c_str(), port.c_str(), &hints, &res) == 0) { + bool up = false; + for (addrinfo* ai = res; ai && !up; ai = ai->ai_next) { + int fd = socket(ai->ai_family, ai->ai_socktype, ai->ai_protocol); + if (fd < 0) continue; + up = connect(fd, ai->ai_addr, ai->ai_addrlen) == 0; + close(fd); + } + freeaddrinfo(res); + if (up) return true; + } + std::this_thread::sleep_for(std::chrono::milliseconds(20)); + } + return false; +} + +} // namespace +#endif // GGML_USE_RPC + +// Whether this build has the RPC backend compiled in. +// +// Exists so a test can assert the *exact* refusal for the build it is running on +// rather than accepting either atom. Without it, the natural way to write one +// test for both configurations is to allow {:rpc_unsupported, :unreachable} — +// which stays green when an RPC build reports :rpc_unsupported, i.e. precisely +// the stale/shared-artifact failure the Makefile's link marker exists to +// eliminate. A capability probe turns that into an exact assertion on both +// builds. +bool rpc_supported(ErlNifEnv* env) { +#ifdef GGML_USE_RPC + return true; +#else + return false; +#endif +} +FINE_NIF(rpc_supported, 0); + +// Registers a remote endpoint's devices in the global ggml device registry. +// +// Deliberately a backend-level NIF next to backend_init/device_list rather than +// a twelfth model_load argument: registration mutates process-global state, not +// one model, and it has to happen before a load so that tensor placement can see +// the remote devices. +// +// Returns the number of devices the endpoint contributed. Upstream memoizes per +// endpoint, so a repeat call is a no-op and returns 0 added. +std::variant, fine::Error> +rpc_add_server(ErlNifEnv* env, std::string endpoint) { +#ifndef GGML_USE_RPC + (void)endpoint; + return fine::Error(atoms::rpc_unsupported); +#else + size_t before = ggml_backend_dev_count(); + + ggml_backend_reg_t reg = ggml_backend_rpc_add_server(endpoint.c_str()); + if (!reg) { + // Both an unreachable endpoint and a HELLO major/minor mismatch collapse + // to nullptr here, and ggml_backend_register silently no-ops on nullptr + // — so without this check a dead or mismatched node simply vanishes and + // the model loads onto the wrong devices. + return fine::Error(atoms::unreachable); + } + + // ggml_backend_rpc_add_server only *builds* the registration. Without this + // second call the devices never enter ggml_backend_dev_count() and + // device_list will not see them. + ggml_backend_register(reg); + + return fine::Ok(static_cast(ggml_backend_dev_count() - before)); +#endif +} +// Dirty IO: a blocking TCP connect plus the HELLO round trip. +FINE_NIF(rpc_add_server, ERL_NIF_DIRTY_JOB_IO_BOUND); + +// Starts the worker-side RPC server. `device_names` empty means every non-CPU +// device, falling back to the CPU device, matching tools/rpc/rpc-server.cpp. +// +// Returns the device names actually being served. The server itself runs on a +// detached std::thread and never comes back: ggml_backend_rpc_start_server's +// accept loop is `while (true)` and the cleanup after it is dead code. +// +// Detaching that thread is exactly what decouples it from the scheduler that +// called us, so THIS NIF returns normally and the never-returning loop costs no +// scheduler at all. What the NIF does spend is real though: a blocking +// getaddrinfo in the pre-flight bind, another per poll iteration, and up to +// 5000 ms of connect-polling before it can honestly claim to be listening. That +// is thousands of times the ~1 ms a normal scheduler expects, and +// RPC.Server.start_link/1 is the sort of call made during application start, so +// it belongs on a dirty IO scheduler like rpc_add_server. +std::variant>, fine::Error, + fine::Error> +rpc_start_server(ErlNifEnv* env, std::string endpoint, std::string cache_dir, + int64_t n_threads, std::vector device_names) { +#ifndef GGML_USE_RPC + (void)endpoint; (void)cache_dir; (void)n_threads; (void)device_names; + return fine::Error(atoms::rpc_unsupported); +#else + std::string host, port; + if (!rpc_split_endpoint(endpoint, host, port)) { + return fine::Error(std::string("endpoint must be \"host:port\", got: " + endpoint)); + } + + std::vector devices; + if (!device_names.empty()) { + for (const auto& name : device_names) { + ggml_backend_dev_t dev = ggml_backend_dev_by_name(name.c_str()); + if (!dev) { + return fine::Error(std::string("unknown device: " + name)); + } + devices.push_back(dev); + } + } else { + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + ggml_backend_dev_t dev = ggml_backend_dev_get(i); + if (ggml_backend_dev_type(dev) != GGML_BACKEND_DEVICE_TYPE_CPU) { + devices.push_back(dev); + } + } + if (devices.empty()) { + if (ggml_backend_dev_t cpu = ggml_backend_dev_by_type(GGML_BACKEND_DEVICE_TYPE_CPU)) { + devices.push_back(cpu); + } + } + } + + if (devices.empty()) { + return fine::Error(atoms::no_devices); + } + + if (std::string error = rpc_preflight_bind(host, port); !error.empty()) { + return fine::Error(error); + } + + std::vector served; + served.reserve(devices.size()); + for (auto dev : devices) served.emplace_back(ggml_backend_dev_name(dev)); + + // Copied into the thread: the NIF's locals are gone the moment it returns, + // and the server outlives every one of them. + std::thread([endpoint, cache_dir, n_threads, devices]() mutable { + ggml_backend_rpc_start_server(endpoint.c_str(), + cache_dir.empty() ? nullptr : cache_dir.c_str(), + static_cast(n_threads), + devices.size(), devices.data()); + }).detach(); + + // Spawning a thread is not the same as serving, and reporting :ok from the + // spawn alone would make every misconfiguration look like a success until + // the first client hangs. + if (!rpc_wait_until_listening(host, port, 5000)) { + return fine::Error(atoms::bind_timeout); + } + + return fine::Ok(served); +#endif +} +// Dirty IO: a blocking bind, two blocking getaddrinfo calls, and a poll that can +// run for 5 s. Bounded, but nowhere near a normal scheduler's budget. +FINE_NIF(rpc_start_server, ERL_NIF_DIRTY_JOB_IO_BOUND); + // --- Devices --- // Enumerates ggml backend devices for VRAM-aware placement and budgeting. @@ -326,7 +581,7 @@ std::variant>, fine::Error> model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap, int64_t main_gpu, int64_t split_mode, std::vector tensor_split, bool use_mlock, bool use_direct_io, bool vocab_only, bool check_tensors, - bool load_mtp) { + bool load_mtp, std::vector device_names) { auto params = llama_model_default_params(); params.n_gpu_layers = static_cast(n_gpu_layers); params.main_gpu = static_cast(main_gpu); @@ -360,6 +615,34 @@ model_load(ErlNifEnv* env, std::string path, int64_t n_gpu_layers, bool use_mmap params.tensor_split = ts_float.data(); } + // llama_model_params.devices is used verbatim: no reordering, no dedup, no + // CPU filtering (src/llama.cpp:152-176). That is the point. The default + // path instead rebuilds the list with RPC devices at the FRONT + // (src/llama.cpp:263-273), which does not match the ggml registry order + // device_list reports — so with a remote device registered, tensor_split + // silently indexes a different list than the caller was looking at. + // Naming the devices is the only way to make placement deterministic. + // + // NULL-terminated, so it must outlive the load call. + std::vector devs; + if (!device_names.empty()) { + devs.reserve(device_names.size() + 1); + for (const auto& name : device_names) { + ggml_backend_dev_t dev = ggml_backend_dev_by_name(name.c_str()); + if (!dev) { + std::string available; + for (size_t i = 0; i < ggml_backend_dev_count(); i++) { + if (i) available += ", "; + available += ggml_backend_dev_name(ggml_backend_dev_get(i)); + } + return fine::Error("unknown device: " + name + " (available: " + available + ")"); + } + devs.push_back(dev); + } + devs.push_back(nullptr); + params.devices = devs.data(); + } + llama_model* model = llama_model_load_from_file(path.c_str(), params); if (!model) { return fine::Error(std::string("failed to load model from: " + path)); @@ -1860,16 +2143,18 @@ fine::Ok<> generate_mtp_tokens( sp.ctx_tgt->forget_batch(); sp.ctx_dft->forget_batch(); - // For MTP, hidden states are extracted via set_embeddings_pre_norm on - // ctx_tgt (set up in the MTP impl's constructor). We only need to know - // that drafts depend on per-position outputs, so logits=true must be - // requested for every prefill token. - const bool need_embd = common_speculative_need_embd(sp.spec); - - // Prefill the target context with the prompt. For MTP we request logits - // at every position so the streaming hook in common_speculative_process - // can mirror t_h_pre_norm into ctx_dft (see speculative.cpp). The full - // batch is then fed back to the speculative state. + // Prefill the target context with the prompt, then hand each decoded batch + // to the speculative state. + // + // Upstream removed common_speculative_need_embd in f785fc9ea: a draft + // implementation that needs the target's hidden states now arranges its own + // extraction -- the MTP impls call llama_set_embeddings_nextn(ctx_tgt, ...) + // in their constructors and read it back through + // llama_get_embeddings_nextn, which is a separate path from the per-token + // logits flag. So the caller no longer has to request logits on every + // prefill position; upstream's examples/speculative-simple now passes false + // for the whole prompt. We ask for logits on the final prompt token only, + // because we sample the first generated token from them below. int n_batch = llama_n_batch(ctx_tgt); llama_pos n_past = 0; for (size_t i = 0; i < prompt.size(); i += n_batch) { @@ -1879,9 +2164,7 @@ fine::Ok<> generate_mtp_tokens( llama_batch batch = llama_batch_init(n, 0, 1); BatchFreeGuard batch_guard(batch); for (int j = 0; j < n; j++) { - const bool want_logits = need_embd - ? true - : (is_last_chunk && j == n - 1); + const bool want_logits = is_last_chunk && j == n - 1; common_batch_add(batch, prompt[i + j], static_cast(i + j), { seq_id }, want_logits); } @@ -1895,8 +2178,7 @@ fine::Ok<> generate_mtp_tokens( if (!proc_ok) { fprintf(stderr, "MTP prefill: common_speculative_process returned false " - "at chunk i=%zu n=%d need_embd=%d logits_on_each=%d\n", - i, n, (int) need_embd, (int) need_embd); + "at chunk i=%zu n=%d\n", i, n); } } n_past = static_cast(prompt.size()); diff --git a/docs/cross-platform-builds.md b/docs/cross-platform-builds.md index b2327a9..200e3ec 100644 --- a/docs/cross-platform-builds.md +++ b/docs/cross-platform-builds.md @@ -15,7 +15,7 @@ with. | Linux (x86_64) + NVIDIA, CUDA 12 | Yes — **CUDA** (`-cu12`) | CUDA | Supported | | Linux (x86_64) + NVIDIA, CUDA 13 | Yes — **CUDA** (`-cu13`) | CUDA | Supported | | Linux (x86_64) + AMD | No | CPU | Supported via `LLAMA_BACKEND=vulkan` | -| Linux (aarch64) + NVIDIA | No | CUDA if a toolkit is found | Tested on DGX Spark (GB10), source build | +| Linux (aarch64) + NVIDIA | No | CUDA if a toolkit is found | Tested on DGX Spark (GB10), source build — see [DGX Spark](dgx-spark.md) | | Linux (aarch64), musl | No | as above | Supported, source build | | Windows (WSL2) | No | Same as Linux | Supported, source build | @@ -130,6 +130,48 @@ LLAMA_CMAKE_ARGS="-DCMAKE_CUDA_ARCHITECTURES=89" mix compile LLAMA_BACKEND=cuda LLAMA_CMAKE_ARGS="-DGGML_CUDA_F16=ON" mix compile ``` +### CPU and CUDA architecture flags + +Two paired variables, for hosts where ggml's `-mcpu=native` probe gives the +wrong answer: + +```bash +LLAMA_BACKEND=cuda \ +LLAMA_CPU_ARM_ARCH=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 \ +LLAMA_CUDA_ARCH=121a-real \ + mix compile +``` + +`LLAMA_CPU_ARM_ARCH` names the architecture for ggml's CPU backend instead of +letting it probe. On a DGX Spark (GB10, Cortex-X925 + A725) with GCC 13.3 the +probe fails **silently**: the compiler predates those cores, rejects +`-mcpu=cortex-x925`, and falls back to base ARMv8-A behind a soft CMake warning +and a zero exit status. The cost is not subtle — `objdump` on the emitted +`libggml-cpu.a` finds **0 `sdot`, 0 `smmla`, no SVE** either way, versus +**1134 `sdot`, 370 `smmla`** and SVE once the architecture is named. Those are +the Q4/Q8 quantized matmul kernels. + +`LLAMA_CUDA_ARCH` sets `CMAKE_CUDA_ARCHITECTURES`, and on a CUDA build it is +**required** whenever `LLAMA_CPU_ARM_ARCH` is set — the Makefile errors out +otherwise. Reaching the CPU flag requires `GGML_NATIVE=OFF`, and with native off +ggml-cuda stops compiling for the GPU it can see and produces a seven-architecture +fat binary instead. That is a roughly 6× build-time regression for no runtime +benefit, and nothing warns about it, so the two variables move together. + +Both are part of the build-directory key, so toggling either lands in a fresh +cmake tree rather than silently reusing a stale `CMakeCache.txt`. Switching back +is still a cache hit. + +To check that the flags survived cmake all the way into the machine code: + +```bash +scripts/spark/verify-build-flags.sh +``` + +`LLAMA_PORTABLE=1` also sets `GGML_NATIVE=OFF`, for the different reason above. +The two never double-emit it, and portable builds still need no CUDA +architecture — the release runners have no GPU to pin one for. + ## Platform-Specific Instructions ### macOS (Apple Silicon) @@ -218,6 +260,10 @@ fails with that message rather than quietly linking against nothing. build without it compiles for the GPU actually present. - NCCL is off unless `LLAMA_CUDA_NCCL=1`. See the README's build variables for why the default is not ggml's. +- On **aarch64** (DGX Spark and friends) also set `LLAMA_CPU_ARM_ARCH` and + `LLAMA_CUDA_ARCH` — see "CPU and CUDA architecture flags" above, and + [DGX Spark](dgx-spark.md) for the measured values. Without them the CPU + backend is built at base ARMv8-A with no quantized matmul kernels, silently. ### Linux (Vulkan) diff --git a/docs/dgx-spark.md b/docs/dgx-spark.md new file mode 100644 index 0000000..664fe17 --- /dev/null +++ b/docs/dgx-spark.md @@ -0,0 +1,724 @@ +# DGX Spark + +A runbook for running `llama_cpp_ex` on NVIDIA DGX Spark (GB10), single node and +across two nodes. Everything here is measured on the hardware rather than +inferred from spec sheets; where a number differs from the marketing figure, the +measurement wins and the reason is given. + +## The machine + +| | | +|---|---| +| SoC | GB10 Grace Blackwell, compute capability **12.1** (`sm_121a`) | +| Memory | **121 GiB unified** — simultaneously host RAM and GPU memory | +| CPU | 20 cores: Cortex-X925 ×10, Cortex-A725 ×10 | +| OS | Ubuntu 24.04 LTS, kernel 6.17 (DGX OS) | +| CUDA | 13.0 at `/usr/local/cuda`, driver 580.173.02 | +| Toolchain | GCC 13.3, cmake 3.28 | + +Three things about this machine are counter-intuitive enough to cost a day each: + +**`nvidia-smi` cannot report GPU memory.** It prints `memory.total = [N/A]`, +because the addressing mode is ATS and there is no separate GPU pool to report. +Use `free -h`. `LlamaCppEx.devices/0` reports it correctly (130.7 GB total on +`CUDA0`) because it asks ggml, not `nvidia-smi`. The device type comes back as +`:igpu` for the same reason. + +**The CPU clusters are interleaved.** The performance cores are **5-9, 15-19**; +the efficiency cores are **0-4, 10-14**. `taskset -c 0-9` therefore pins a job +entirely to little cores. Measured on a 1-byte RDMA ping-pong: p50 21.5 µs on +cpu19 versus 29.1 µs on cpu0. + +**`ping` lies by three orders of magnitude.** ICMP RTT over the 200 Gb/s direct +link reads ~1.2 ms. That is cpuidle exit latency (LPI-3 is 433 µs), not the wire +— the 1 GbE LAN measures 1.34 ms, which is the tell. Hold the cores out of deep +idle and ICMP drops to 0.028 ms. The real numbers are **1.39 µs RTT over RDMA** +and **~19 µs p50 over TCP**. + +### The fabric + +Two point-to-point ConnectX-7 links, no switch: + +| link | spark-1 | spark-2 | MTU | measured | +|---|---|---|---|---| +| 0 | 10.100.64.1 | 10.100.64.2 | 9000 | 13.98 GB/s | +| 1 | 10.100.65.1 | 10.100.65.2 | 9000 | 13.98 GB/s | + +RoCE v2, GID 3, `active_mtu` 4096. 13.98 GB/s is 89% of the theoretical +15.75 GB/s, and the ceiling is **PCIe Gen5 x4** per port (`max_link_width = 4`, +so this is the design and not a training failure) — not the 200 Gb/s wire rate. +Both links together do reach 24.5 GB/s, but a single `ggml-rpc` peer connection +cannot use both: the transport opens one queue pair per socket and picks the HCA +by matching a GID against the socket's local address. Budget 13.98 GB/s. + +## The remote development loop + +The control node is a Mac; the Sparks are `spark-1` and `spark-2` over ssh with +`ControlMaster` configured. Nothing below needs `sudo` on either Spark — the +boxes require a password for it, so no package installs, no `sysctl`, no kernel +cmdline changes and no system units are available. Where root *would* help it is +called out explicitly rather than done silently. + +```bash +scripts/spark/bootstrap.sh # once per node: toolchain, directories, fact sheet +scripts/spark/sync.sh # push the working tree to both nodes +scripts/spark/remote.sh spark-1 mix test +``` + +### `bootstrap.sh` + +Idempotent per-node provisioning. Creates `~/models`, `~/.cache/llama.cpp/rpc` +and `~/src/llama_cpp_ex`, prints a fact sheet, and provisions the Elixir +toolchain. + +The toolchain is the only interesting part. `spark-2` has no asdf and we cannot +`apt install` the OTP build dependencies, so `asdf install erlang` is not a plan. +The boxes are identical — same distro, kernel, architecture — and a kerl-built +OTP links only against libraries present on both, so bootstrap copies +`spark-1:~/.asdf` across the fabric instead. Measured at **8 seconds**. + +The nodes have no ssh keys for each other and this script does not add any: +authentication rides a forwarded agent (`ssh -o ControlPath=none -A`, because +`-A` on a session that reuses an existing multiplexed master does nothing). If +no agent is forwardable it relays the 431 MB through the control node instead. + +Ubuntu's own Elixir is **not** a fallback: 24.04 ships 1.14 / OTP 25 and +`mix.exs` requires `~> 1.18`. If the copy fails, bootstrap prints the +`asdf install` path and the exact `apt` line for the user to run with their +password. + +Note that nothing puts asdf's **shims** on `PATH`. asdf 0.19's shims are +`exec asdf exec ` and the `asdf` binary lives in `/usr/bin` on `spark-1` +and nowhere on `spark-2`. The install directories go on `PATH` directly instead, +which works identically on both nodes and skips a process per invocation. + +### `sync.sh` + +`rsync -az --delete` of the *working* tree, uncommitted changes included, which +is why this is rsync and not `git pull`. About 36 s cold, a few seconds warm. + +The exclusions are in `scripts/spark/lib.sh` with a reason each. One is +non-obvious: **`vendor/llama.cpp/.git`** is excluded, 36 MB of history the build +does not need. `Makefile` falls back to `LLAMA_COMMIT` when that directory is +absent, so `LLAMA_SHA` still resolves — and `sync.sh` verifies exactly that on +every run rather than trusting it. + +After every sync both nodes are proven **byte-identical to the control node** +over the synced file set, by content digest. This is load-bearing rather than +hygiene: the ggml RPC `HELLO` handshake compares only `major`/`minor` and +**ignores `patch`**, so two builds from drifted trees connect happily and then +misinterpret each other's op codes. + +### `remote.sh` + +Runs a command on a node under the environment contract in +`scripts/spark/lib.sh`: `PATH` for the toolchain and CUDA, `CUDA_HOME`, the +build flags below, `LLAMA_CACHE_DIR`, and `LLAMA_SMOKE_GEN_MODEL` / +`LLAMA_MODEL_PATH` when those files exist. + +The default is a **non-login** shell, deliberately. DGX OS puts `nvcc` on the +login `PATH` only, via `/etc/profile.d/nv_paths.sh`, which `ssh host cmd`, a +systemd unit and every CI shell never source. Rather than paper over that with +`bash -lc`, the contract sets `CUDA_HOME` and the Makefile's toolkit discovery +takes it from there. `--login` exists for the rare case that genuinely needs the +profile; if a *build* needs it, the bug is in the contract. + +Useful flags: `--big-cores` wraps the command in `taskset -c 5-9,15-19`, +`--env K=V` adds one variable, `--print` shows the remote script without running +it. + +## Build flags + +```bash +LLAMA_BACKEND=cuda \ +LLAMA_CPU_ARM_ARCH=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 \ +LLAMA_CUDA_ARCH=121a-real \ + mix compile +``` + +`remote.sh` exports all three, so on a Spark this is just `mix compile`. + +### Why `LLAMA_CPU_ARM_ARCH` is not optional + +ggml probes the host with `-mcpu=native` when `GGML_NATIVE` is on, which is the +default. On GB10 with GCC 13.3 that probe **fails silently**: the compiler +predates Cortex-X925/A725 and rejects `-mcpu=cortex-x925`, cmake emits a soft +warning, and the build exits 0 with the CPU backend compiled at base ARMv8-A. + +Measured on the emitted `libggml-cpu.a`: + +| | `sdot` | `smmla` | SVE | +|---|---|---|---| +| `-mcpu=native` (default) | 0 | 0 | none | +| `-march=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2` | 1134 | 370 | 10678 operands | + +Those are the Q4/Q8 quantized matmul kernels. + +### Why `LLAMA_CUDA_ARCH` must accompany it + +Naming the CPU architecture requires `GGML_NATIVE=OFF`, and with native off +ggml-cuda stops compiling for the GPU it can see and emits a **seven-architecture +fat binary** instead — a ~6× build-time regression, silently. So the Makefile +makes `LLAMA_CPU_ARM_ARCH` without `LLAMA_CUDA_ARCH` a hard `$(error)` on a CUDA +build. Both variables are part of the build-directory key, so toggling either +gets a clean `CMakeCache.txt` and switching back is still a cache hit. + +`LLAMA_PORTABLE=1` also sets `GGML_NATIVE=OFF`, for the unrelated reason that +published artifacts must not carry `-march=native`. The two never double-emit +it, and portable builds still need no CUDA architecture — the release runners +have no GPU. + +### Verifying a build + +```bash +scripts/spark/remote.sh spark-1 scripts/spark/verify-build-flags.sh +``` + +``` +==> verifying _build/dev/lib/llama_cpp_ex/obj/llama_build-cuda-6b5f34be + PASS ggml-cuda: one architecture, compute_121a + PASS ggml-cpu: -march=armv9.2-a+dotprod+i8mm+fp16+bf16+sve2 + PASS libggml-cpu.a: 1134 sdot, 370 smmla, 10678 SVE operands +==> all checks passed +``` + +Three assertions, because each catches a different silent failure: the fat +binary, the flag not reaching the compiler, and the flag reaching the compiler +but producing nothing. The CUDA architecture is asserted from +`ggml-cuda.dir/flags.make` and **not** from `CMakeCache.txt` — +`CMAKE_CUDA_ARCHITECTURES` is an ordinary variable and never appears in the +cache. + +`test/makefile_arch_flags_test.exs` covers the Makefile side hermetically on both +macOS and Linux, including the `$(error)` and the build-directory key. + +A full clean CUDA build takes **2m14s** at `-j20` (1m47s before the flags; the +difference is the wider CPU code generation, not the fat binary). There is no +`ccache` on these boxes and installing one needs a password. + +## Models + +```bash +scripts/spark/remote.sh spark-1 mix run scripts/spark/fetch_models.exs --list +scripts/spark/remote.sh spark-1 mix run scripts/spark/fetch_models.exs 8b 30b +``` + +Downloads go through `LlamaCppEx.Hub.download/3` — the library's own path, +SHA-256 verified fail-closed — into `~/models///`. + +| label | size | role | +|---|---|---| +| `8b` | 5.0 GB | Qwen3-8B Q4_K_M — dense sanity check | +| `30b` | 18.6 GB | Qwen3-30B-A3B Q4_K_M — the MoE case this chip is good at | +| `120b` | 63.4 GB | gpt-oss-120b MXFP4 — big but **fits** one node | +| `235b` | 142.1 GB | Qwen3-235B-A22B Q4_K_M, 3 shards — does **not** fit one node | + +121 GiB is 130.0 GB, so `120b` is the controlled A/B for measuring pure RPC +overhead (same model, one node versus two) and `235b` at 142.1 GB is the case +that justifies a second Spark at all. + +### Qwen3.6 and the MTP variants + +| label | size | role | +|---|---|---| +| `q36-27b` | 16.8 GB | Qwen3.6-27B Q4_K_M — current-generation dense | +| `q36-27b-mtp` | 17.1 GB | the same weights **plus the MTP head** | +| `q36-35b` | 22.1 GB | Qwen3.6-35B-A3B UD-Q4_K_M — current-generation MoE | +| `q36-35b-mtp` | 22.7 GB | the same weights **plus the MTP head** | + +The MTP repos are not different quantizations; they are the same model with +Multi-Token Prediction layers included. llama.cpp reads those layers only when +the model is loaded with `load_mtp: true`, which makes the plain and MTP files a +clean A/B for what speculative decoding buys on this hardware. See +`LlamaCppEx.MTP`. + +## Running on ONE Spark + +Full numbers and methodology in +`bench/results/v0.8.43-dgx-spark-baseline.md`, and the two-node numbers in +`bench/results/v0.8.43-dgx-spark-two-node.md`. +The short version: + +| model | prefill (pp) | decode (tg) | +|---|---|---| +| Qwen3-8B Q4_K_M | 3500–4331 t/s | 40.5 t/s | +| Qwen3-30B-A3B Q4_K_M | 3287 t/s | 90.9 t/s | + +Three of the four figures beat the published single-Spark references; the fourth +is 7% under. **Prefill is this machine's strong suit** and decode is bandwidth- +bound, so quote them separately or you describe neither. + +### Settings that matter + +```elixir +LlamaCppEx.Server.start_link( + model_path: path, + n_gpu_layers: 99, # always. n_gpu_layers: 0 costs 57% of decode + n_parallel: 8, # ~8x aggregate throughput for ~0 per-request cost + n_ctx: 4096 * 8 +) +``` + +And the settings that do **not** matter, each of which looks like it should: + +| knob | verdict | +|---|---| +| `flash_attn` | leave `:auto` — it is already on, and `:disabled` costs 24% of prefill | +| `type_k` / `type_v` | leave f16. Quantizing the KV cache **loses** 2–7% and buys nothing: there is no separate VRAM to free | +| `use_mlock`, `use_direct_io` | leave off. Both cost 3–4%; "pinned in RAM" and "resident on the GPU" are the same DRAM here | +| `n_batch`, `n_ubatch` | no measurable effect; the default is fine | +| cpuidle / `idle=poll` | **no effect**, single-node or two-node. See below | +| `taskset -c 5-9,15-19` | hygiene, not speed. But never use `-c 0-9` — those are the little cores | + +### The cpuidle story, and why you can ignore it + +cpuidle exit latency is the largest measured effect on this machine — LPI-3 exit +is 433 µs, and it is why `ping` reads 1.2 ms on a link whose real RTT is 1.39 µs. +It is natural to assume it also costs inter-token latency. + +It does not. Every condition tested — a `nice -19` poller on every core, BEAM +busy-wait tuning, X925 pinning, and the same again across two nodes with a +network wake on every token — landed within 2% of doing nothing, and the poller +made TTFT *worse*. A decode loop keeps the CPU busy, so it never enters a deep +C-state and there is no exit latency to avoid. + +**So do not go asking for `idle=poll` on the kernel cmdline.** It was the one +thing this work expected to need root for, and the measurement retired it. + +## Qwen3.6 and speculative decoding (MTP) + +Two shapes of the current generation, each measured with and without the +Multi-Token Prediction head. The MTP repos are the *same weights* plus the head, +so this is a clean A/B rather than a comparison across quantizations. + +256-token greedy generations through the chat template, **median of 5** with the +range alongside. Five samples matter here: MTP is the noisier arm, and a single +run per setting is not enough to tell a real 1.6× from a lucky draw. The plain +model is freed before the MTP arm runs — leaving ~20 GB of unrelated weights +resident cost the MTP arm about 10% on this unified-memory part, which is +exactly the kind of confound that flips a conclusion. + +### Qwen3.6-27B Q4_K_M — dense + +| config | decode t/s | range | vs baseline | acceptance | +|---|---|---|---|---| +| no MTP head | 11.59 | 11.5–11.6 | — | — | +| MTP `n_draft: 1` | 16.88 | 16.9–16.9 | 1.46× | 86.9% | +| MTP `n_draft: 2` | 18.38 | 18.2–18.4 | 1.59× | 76.4% | +| **MTP `n_draft: 3`** | **18.65** | 18.4–18.7 | **1.61×** | 68.2% | +| MTP `n_draft: 4` | 17.42 | 17.3–17.5 | 1.50× | 57.1% | + +No range overlaps the baseline: **MTP is worth 1.6× on the dense model**, and +`n_draft` 2 and 3 are within noise of each other. + +### Qwen3.6-35B-A3B UD-Q4_K_M — MoE + +| config | decode t/s | range | vs baseline | acceptance | +|---|---|---|---|---| +| no MTP head | 65.36 | 65.1–65.8 | — | — | +| MTP `n_draft: 1` | 67.56 | 67.0–67.7 | 1.03× | 81.0% | +| MTP `n_draft: 2` | 62.29 | 61.4–62.3 | 0.95× | 67.2% | +| MTP `n_draft: 3` | 63.28 | 63.0–63.5 | 0.97× | 64.9% | +| MTP `n_draft: 4` | 47.66 | 44.8–48.6 | 0.73× | 40.9% | + +Essentially neutral at best, and a loss past `n_draft: 1`. + +> The README reports **+16%** at `n_draft: 2` for Qwen3.6-35B-A3B on GB10, from +> an interleaved n=11 run on **UD-Q4_K_XL**. This measurement is UD-Q4_K_**M**, +> and the draft acceptance rates agree closely (67.2% here versus 68.5% there at +> `n_draft: 2`) while the throughput economics do not. Take the quantization as +> the likely difference and measure your own before relying on either number. + +### The rule this gives you + +**MTP pays on dense models and roughly breaks even on sparse MoE.** The mechanism +is the one that makes this chip interesting: speculative decoding spends compute +(a batched verification pass) to save memory bandwidth (sequential decode steps). +On the dense 27B every token reads all 27B of weights, so that trade is strongly +favourable — 1.6×. On the 35B-A3B only ~3B parameters move per token, decode is +already cheap, and the draft-and-verify overhead eats the gain. + +Two more things the numbers say: + +- **The best `n_draft` is model-shaped, so measure it.** On the dense model 3 is + best and 2 is within noise; on the MoE anything above 1 loses. There is no + single default that transfers, which is also the README's conclusion for + Metal. +- **Acceptance decays fast with draft depth** — 87% → 76% → 68% → 57% on the + dense model — so past the sweet spot you pay twice: wasted draft compute and a + longer verification batch. + +```elixir +{:ok, model} = LlamaCppEx.Model.load(mtp_path, n_gpu_layers: 99, load_mtp: true) +{:ok, session} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 4096) +{:ok, text} = LlamaCppEx.MTP.generate(session, prompt, max_tokens: 256) +``` + +Reproduce: + +```bash +scripts/spark/remote.sh --env MIX_ENV=bench --big-cores spark-1 \ + mix run bench/spark_mtp.exs +``` + +> #### Qwen3.6 instruct checkpoints need the chat template {: .warning} +> +> `Qwen3.6-35B-A3B` emits end-of-generation **immediately** when handed a bare +> completion prompt — zero tokens, from both the plain and the MTP path — while +> the identical prompt inside the chat template generates normally. The 27B +> tolerates raw completion, which is exactly the kind of difference that becomes +> a mystery if your harness does not template. Use `LlamaCppEx.chat/3`, or +> `LlamaCppEx.Chat.apply_template/3` when you need the prompt as a string (as +> `LlamaCppEx.MTP` does). + +--- + +# Running on TWO Sparks + +## What two nodes actually buy you + +Read this before building anything on it, because the honest answer is narrower +than the marketing: + +- **Capacity: yes.** A model that does not fit in 130 GB runs. Nothing else on + this pair will run it at all. +- **Speed: no, but also not the loss you would expect.** Pipeline parallelism is + *disabled* whenever an RPC device participates — the RPC backend reports + `async = false, events = false` and llama.cpp checks that before enabling + pipelining — so the two nodes execute **sequentially**. For a model that fits + on one node, the second node measured within a few percent either way + (see B1 below), because the sequential penalty and the halved per-node + bandwidth pressure roughly cancel. +- **Tensor parallelism ("tp=2"): technically yes, practically no.** It runs and + it is correct, and it is 2.7× slower on decode. See the verdict section. + +## The mechanism, in one paragraph + +One node runs a *worker* (`LlamaCppEx.RPC.Server`) exposing its GPU on a TCP +endpoint. The other node is the *client*: it registers that endpoint with +`LlamaCppEx.RPC.add_server/1`, at which point the remote GPU appears in +`LlamaCppEx.devices/0` as `RPC0` and can hold part of a model like any other +device. `split_mode: :layer` then gives each device a contiguous range of layers +and its own KV cache. On Linux the transport auto-negotiates RDMA over the +ConnectX-7 link; on this pair it always does. + +## Runbook + +```bash +# 1. Worker on spark-2, bound to the fabric address, tensor cache on. +scripts/spark/rpc-worker.sh start spark-2 + +# 2. Prove the whole chain before believing any number. +scripts/spark/remote.sh --env LLAMA_RPC=1 --env MIX_ENV=test spark-1 \ + mix run scripts/spark/rpc_check.exs 10.100.64.2:50052 + +# 3. Use it. +scripts/spark/remote.sh --env LLAMA_RPC=1 --env MIX_ENV=bench spark-1 \ + mix run bench/spark_two_node.exs b1 + +# 4. Stop it between runs. The worker leaks; see below. +scripts/spark/rpc-worker.sh stop spark-2 +``` + +Other subcommands: `status`, `rss`, `logs [n]`, `restart`. Useful flags: +`--debug` (`GGML_RPC_DEBUG=1` on the worker), `--tcp` (force TCP for an A/B), +`--upstream` (run upstream's `ggml-rpc-server` instead of ours, as a reference), +`--no-cache`, `--threads N`, `--port N`. + +The worker needs an RPC build, which `rpc-worker.sh` arranges by exporting +`LLAMA_RPC=1` into the unit. The client needs one too — pass +`--env LLAMA_RPC=1` to `remote.sh`. + +In code: + +```elixir +{:ok, _} = LlamaCppEx.RPC.add_server("10.100.64.2:50052") + +{:ok, server} = + LlamaCppEx.Server.start_link( + model_path: path, + n_gpu_layers: 99, + devices: ["CUDA0", "RPC0"], # name them; see the ordering trap below + split_mode: :layer, + tensor_split: [0.5, 0.5] + ) +``` + +`:rpc_servers` does the registration for you, in the right order: + +```elixir +LlamaCppEx.Server.start_link( + model_path: path, + rpc_servers: ["10.100.64.2:50052"], + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.5, 0.5] +) +``` + +## Supervision + +`loginctl enable-linger` succeeds without a password on these boxes, so the +worker runs as a **`systemd --user` transient unit** started with `systemd-run`. +That gets journald capture, `systemctl --user restart`, and survival across +logout, with no root and no unit files to install. `rpc-worker.sh` also starts an +RSS sampler alongside it. + +`LlamaCppEx.RPC.Server` is a GenServer that owns the native server thread, but it +**cannot stop it**: upstream's accept loop is `while (true)` with no shutdown +hook, so the thread and its port outlive the process. `terminate/2` says so +rather than pretending. The VM is the unit of restart. + +## Five things that will cost you an afternoon + +### 1. A peer failure kills the VM + +Every client-side RPC command checks its result with `RPC_STATUS_ASSERT`, which +is `GGML_ABORT`. A worker that crashes, a link that drops, or a malformed +response **terminates the OS process — the BEAM with it**. There is no error +return, no retry, no reconnect, and nothing to `rescue`. + +This is upstream's design, not this binding's, and the API is shaped around it: +*registration* is the one operation that reports instead of aborting, so +`LlamaCppEx.RPC.ping/1` before a load turns "the model silently landed on the +wrong devices" into `{:error, :unreachable}`. After the load, treat the VM as the +unit of restart. Real fault isolation means putting the RPC client in a separate +OS process, which is a different architecture. + +### 2. `devices/0` order is NOT `tensor_split` order + +Two device lists exist and they disagree: + +| list | order | read by | +|---|---|---| +| ggml **registry** | registration order — local first, RPC appended | `LlamaCppEx.devices/0`, and so `gpu_index` | +| llama.cpp **placement** | **RPC first**, then GPUs, then iGPUs | `:tensor_split`, `:main_gpu` | + +Measured on spark-1 with one endpoint registered: + +``` +[0] CUDA0 CUDA igpu gpu_index=0 NVIDIA GB10 +[1] CPU CPU cpu gpu_index=nil CPU +[2] RPC0 RPC gpu gpu_index=1 10.100.64.2:50052 +``` + +…yet placement is `[RPC0, CUDA0]`, so `tensor_split: [0.25, 0.75]` puts 25% on +the **remote** node and `main_gpu: 0` selects it. A backwards split produces +correct tokens and merely benchmarks badly, so nothing warns you. + +**Always pass `:devices`.** It is used verbatim — no reordering, no dedup, no CPU +filtering — and then `:tensor_split` indexes the list you wrote down. + +### 3. The worker leaks, but it plateaus + +Measured across repeated runs against one worker serving a 30 GB share: +349 → 595 MiB on the first client, then 595 → 595 for every subsequent client. +So it retains roughly 245 MiB per model share and never gives it back, but it +does **not** accumulate per run within one worker lifetime. Restart between +experiments anyway; `rpc-worker.sh` runs an RSS sampler that stops the worker at +92% of RAM rather than letting the node OOM. + +### 4. A model that does not fit does not degrade — it OOMs the box + +Loading Qwen3-235B-A22B Q4_K_M (142.1 GB) on **one** node with mmap did not +produce a slow number. It produced this: + +``` +oom-kill: constraint=CONSTRAINT_NONE, global_oom +Out of memory: Killed process 1599 (avahi-daemon) +NVRM: Out of memory [NV_ERR_NO_MEMORY] ... _memdescAllocInternal +``` + +The machine survived, but the OOM killer took out an unrelated system service — +`avahi-daemon`, so the box stopped resolving over mDNS and became unreachable by +name from the control node while remaining perfectly healthy. Unified memory is +the reason: there is no separate VRAM to spill into, so "offload everything" and +"keep it in page cache" compete for the same 130 GB and `mmap` cannot save you. + +That is why `scripts/spark/lib.sh` has `SPARK_HOST_SPARK_1` / `SPARK_HOST_SPARK_2` +overrides — a name-resolution failure should be a one-variable fix: + +```bash +export SPARK_HOST_SPARK_1=192.168.0.164 +``` + +### 5. Both nodes must be byte-identical + +The RPC `HELLO` handshake compares only `major`/`minor` and **ignores `patch`**, +so two builds from drifted trees connect happily and then misinterpret each +other's op codes. `sync.sh` proves byte-identity by content digest on every run. +This is load-bearing, not hygiene. + +## Measured + +### The headline: a model that does not fit on one node + +Qwen3-235B-A22B Q4_K_M, **142.1 GB** of weights against 130.0 GB of unified +memory per node. 512-token prompt, 32 decode steps. + +| run | load s | TTFT ms | prefill t/s | decode t/s | worker RSS | +|---|---|---|---|---|---| +| single-node, mmap overflow | — | — | — | — | **global OOM, killed `avahi-daemon`** | +| two-node 50/50, RDMA | 538.4 | 1281.9 | 423.5 | **13.69** | 344 → 580 MiB | + +There is no percentage to quote here, and that is the point: on one Spark this +model does not run slowly, it takes the machine's memory out from under the OOM +killer. On two it runs at **13.7 tokens/s**, which is a usable interactive speed +for a 235B model. + +The cold load is nine minutes, because ~71 GB of weights cross the fabric. Budget +for it, keep the worker's tensor cache on, and do not restart casually. + +**This is the entire argument for the second Spark.** If your model fits in +130 GB, the numbers below say one node is the answer. + +### The control: a model that does fit + +gpt-oss-120b MXFP4 (63.4 GB — fits one node, so this isolates RPC cost from +memory benefit), 1024-token prompt, 64 decode steps. + +| run | load s | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---|---| +| single-node | 64.8 | 563.6 | 1889.0 | 46.50 | +| two-node 50/50, RDMA | 151.3 | 590.4 | 1797.6 | **48.14** | + +Decode came out **+3.5% on two nodes** for a model that fits on one. That is not +what "the nodes run sequentially" predicts, and it is worth understanding before +reading too much into it: splitting halves each node's per-token weight traffic, +and on a bandwidth-bound part that relief roughly cancels the sequential +penalty. Repeat runs put both configurations in the 45–49 t/s band, so the +honest summary is **"no material difference"**, not "two nodes are faster". + +Load time is the real cost: **2.3× worse cold**, because ~30 GB of weights cross +the network. The worker's content-addressed tensor cache (`-c`, on by default in +`rpc-worker.sh`) took a warm load from 151 s to 139 s — much less than hoped, +because the client still reads and hashes every tensor locally to check the +cache; only the transfer is skipped. + +### RDMA versus TCP + +There is no runtime switch. Transport selection is silent auto-negotiation with +no env var and no endpoint scheme; the only levers are `GGML_RDMA_DEV` pointing +at a device that does not exist, or a `LLAMA_RPC_RDMA=0` build. + +| transport | TTFT ms | prefill t/s | decode t/s | +|---|---|---|---| +| RDMA | 595.7 | 1784.0 | 46.05 | +| TCP (forced) | 805.8 | 1310.5 | 40.90 | +| | **+35%** | **−27%** | **−11%** | + +So RDMA is worth real tokens, and if RDMA ever wedges (upstream issue #24813, +closed as stale one week before our pinned commit), TCP is a working fallback +that costs about a tenth of decode. + +To confirm which one you got, there is exactly one signal — the worker's log with +`GGML_RPC_DEBUG=1`: + +```bash +scripts/spark/rpc-worker.sh start spark-2 --debug +scripts/spark/rpc-worker.sh logs spark-2 200 | grep -E 'RDMA|transport' +# RDMA probed: dev=rocep1s0f1 gid=3 RoCEv2 qpn=33437 inline=316 +# RDMA activated: qpn=33437->33437 mtu=4096 rx_depth=24 +``` + +`RDMA activate failed, staying on TCP` is the line that means you are measuring +the slow path. + +### Concurrency across two nodes + +| `n_parallel` | decode t/s per request | +|---|---| +| 1 | 49.48 | +| 4 | 44.71 | +| 8 | 44.25 | + +Per-token RTT is fixed per *graph*, not per token, so batching amortises it +well: 8 concurrent requests cost 11% of per-request decode for 8× the aggregate. + +### The decode fast path is holding + +Worth checking whenever anything about batching changes. A repeated graph +collapses to a 4-byte `GRAPH_RECOMPUTE`; a miss re-serialises every tensor +descriptor on every token. Measured over a two-node generation: +**92 `graph_recompute`, zero `graph_compute`**. `LlamaCppEx.Server`'s per-tick +batch composition does not break the cache. + +```bash +scripts/spark/rpc-worker.sh logs spark-2 4000 | grep -c graph_recompute +``` + +--- + +# "tp=2" — what it means here, and the verdict + +"tp=2" is vLLM vocabulary. llama.cpp at b10362 has four split modes and only one +of them is what people mean by it: + +| mode | value | status | +|---|---|---| +| `:none` | 0 | single device | +| `:layer` | 1 | contiguous layer ranges, one KV cache per device. **The only working cross-host mode** | +| `:row` | 2 | **dead for CUDA.** ggml-cuda no longer exports `ggml_backend_split_buffer_type`, so the load throws `device CUDA0 does not support split buffers`. Only SYCL still declares one | +| `:tensor` | 3 | real tensor parallelism via a Meta device (#19378, Apr 2026) | + +`:tensor` forces flash attention on, refuses a handful of architectures +(`llm_arch_supports_sm_tensor` is a blocklist, so qwen3 and gpt-oss both pass), +and disables backend sampling. + +## Does `-sm tensor` work across two hosts? + +**Yes — and you should not use it.** No prior report of this combination exists; +here is one. + +Single node first, to price the Meta device itself (Qwen3-8B, 512-token prompt): + +| configuration | prefill t/s | decode t/s | +|---|---|---| +| `:none`, flash on | 3700.1 | 38.88 | +| `:tensor`, one local GPU | 3865.5 | 38.46 (−1.1%) | + +Essentially free. So the Meta device is not the problem. Now two nodes: + +| configuration | prefill t/s | decode t/s | vs layer split | +|---|---|---|---| +| `:layer`, 2 nodes | 3262.4 | 36.58 | — | +| `:tensor`, 2 nodes | 140.9 | 13.30 | **−63.6%** | +| `:tensor`, `GGML_CUDA_ALLREDUCE=none` | 137.4 | 13.59 | −63.2% | +| `:tensor`, `GGML_CUDA_ALLREDUCE=internal` | 277.6 | 10.33 | −71.8% | + +It runs, and the output is byte-identical to the layer-split reference. It is +**2.7× slower on decode and 23× slower on prefill**. + +## Why, and why the comm-mode knob cannot help + +`ggml_backend_cuda_comm_init` returns `nullptr` the moment **any** member backend +is not CUDA. An RPC device is not CUDA, so the CUDA all-reduce — NCCL or the +internal pipeline — never engages at all, which is why all three +`GGML_CUDA_ALLREDUCE` settings land in the same place. + +What runs instead is the meta backend's generic butterfly, which moves data with +`ggml_backend_tensor_{set,get}_2d`. The RPC backend leaves both 2-D hooks `NULL`, +and ggml then falls back to a **loop of `n_copies` separate 1-D transfers**. That +is the cliff: not a failure, just every all-reduce turned into a burst of +individual network round trips, once per layer, per token. + +Note this contradicts the obvious reading of the source, which is that NULL 2-D +hooks would abort. They do not — `ggml-backend.cpp` degrades to the 1-D loop. +That is why this was worth measuring rather than reasoning about. + +## The verdict + +**On two DGX Sparks, use `split_mode: :layer` over the RPC backend.** It is the +only cross-host mode that is both correct and fast, and it buys capacity. + +- `:row` throws at load on CUDA. Do not build on it. +- `:tensor` is in-process tensor parallelism. Its CUDA all-reduce is + `ncclCommInitAll` — single-process, one distinct physical GPU per rank — so it + cannot span hosts as designed. Across hosts it silently falls back to a generic + path that is 2.7× slower. It is the right tool for several GPUs in **one** box, + which a Spark does not have. +- Layer split over RPC buys **capacity, not speed**. If your model fits in + 130 GB, one Spark is the answer. + +Re-check these claims against a future llama.cpp bump using upstream commits +`d6f303004` (`-sm tensor`), `adb541a6a` and `91fef9536`. diff --git a/docs/multi-gpu.md b/docs/multi-gpu.md index e44ed65..b5eff73 100644 --- a/docs/multi-gpu.md +++ b/docs/multi-gpu.md @@ -20,7 +20,9 @@ LlamaCppEx.devices() ``` - `:gpu_index` is 0-based across GPU/IGPU devices and **matches the index space - of `:tensor_split`** (non-GPU devices have `gpu_index: nil`). + of `:tensor_split`** (non-GPU devices have `gpu_index: nil`) — *unless* an RPC + device is registered, which reorders the placement list. See + [Remote devices](#remote-devices-rpc). - `:memory_free`/`:memory_total` are bytes. Device order follows `CUDA_VISIBLE_DEVICES`. @@ -32,7 +34,7 @@ These pass straight through `load/3` (per model) to `Model.load/2` / | Option | Meaning | |---|---| | `:n_gpu_layers` | Layers to offload (`-1` = all, `0` = CPU only) | -| `:split_mode` | `:none` (single GPU), `:layer` (split layers), `:row` (split tensor rows) | +| `:split_mode` | `:none` (single device), `:layer` (split layers), `:row` (**throws on CUDA**), `:tensor` (in-process tensor parallelism) | | `:tensor_split` | A **list of per-device proportions** — one float per GPU, indexed by device order. Zeros exclude a device. | | `:main_gpu` | Primary device: the single GPU under `:none`, or the device holding non-split tensors under `:layer` | @@ -55,6 +57,112 @@ LlamaCppEx.ModelManager.load("embed", {:path, m2}, tensor_split: [0, 0, 0, 0, 1, 1, 1, 1]) ``` +## Remote devices (RPC) + +With a build that has `LLAMA_RPC=1`, `LlamaCppEx.RPC.add_server/1` registers +another host's devices into the same registry `LlamaCppEx.devices/0` reads, and +they take part in placement like any other device. That is how a model larger +than one machine loads at all. + +```elixir +{:ok, 1} = LlamaCppEx.RPC.add_server("10.100.64.2:50052") + +LlamaCppEx.devices() +#=> [%{index: 0, gpu_index: 0, type: :igpu, backend: "CUDA", name: "CUDA0", +# description: "NVIDIA GB10", memory_total: 130_662_940_672, ...}, +# %{index: 1, gpu_index: nil, type: :cpu, backend: "CPU", ...}, +# %{index: 2, gpu_index: 1, type: :gpu, backend: "RPC", name: "RPC0", +# description: "10.100.64.2:50052", ...}] +``` + +Three things about that list are not what you would guess. + +### `gpu_index` stops indexing `:tensor_split` + +**This is the one that will cost you an afternoon.** There are two device +orderings and they are not the same list: + +| list | order | what reads it | +|---|---|---| +| the ggml **registry** | registration order — local backends first, RPC endpoints appended as they register | `LlamaCppEx.devices/0`, and therefore `gpu_index` | +| llama.cpp's **placement** list | **RPC first**, then discrete GPUs, then integrated GPUs | `:tensor_split`, `:main_gpu` | + +llama.cpp rebuilds the second list at load time and inserts RPC devices at the +front of it, with the comment *"to minimize network transfers"*. So with one +local GPU and one endpoint: + +```elixir +# devices/0 says RPC0 has gpu_index: 1 ... +# ... but placement order is [RPC0, CUDA0], so this puts 25% on the REMOTE node. +LlamaCppEx.Model.load(path, split_mode: :layer, tensor_split: [0.25, 0.75]) + +# And main_gpu: 0 selects the REMOTE node for the non-split tensors, which is +# almost certainly not what you want — the output tensor would then cross the +# network on every token. The local GPU is index 1. +LlamaCppEx.Model.load(path, split_mode: :layer, tensor_split: [0.5, 0.5], main_gpu: 1) +``` + +A backwards split still works, produces correct tokens, and simply benchmarks +badly, so nothing tells you. The placement order is not observable from Elixir; +it is derived, at load time, from what is registered. Two rules follow: + +1. With RPC devices registered, **do not use `gpu_index` to build + `:tensor_split`.** Count RPC devices first, then local GPUs. +2. Better: state the order. `LlamaCppEx.Model.load/2` accepts `:devices`, a list + of device names used **verbatim** with no reordering — see below. + +One local wrinkle worth knowing if you are on a DGX Spark or another +integrated-GPU box: the local GB10 reports as `:igpu`, and llama.cpp appends +integrated GPUs only when it found no discrete ones. RPC devices deliberately do +not count as discrete for that test, so the local iGPU is not dropped — the +placement list really is `[RPC0, CUDA0]`. + +### Stating the device order explicitly + +```elixir +# Verbatim: no reordering, no dedup, no CPU filtering. +LlamaCppEx.Model.load(path, + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.6, 0.4]) # 60% local, 40% remote — and it says so +``` + +`:devices` names devices as `LlamaCppEx.devices/0` reports them, and +`:tensor_split` then indexes *that* list. This is the sanctioned way to make +placement deterministic, and with more than one device in play it is the only +way to make it obvious in review. + +### The type is always `:gpu` + +Even when the remote worker serves only a CPU device. Upstream hardcodes it with +a TODO. `:description` carries the endpoint string and is the only reliable way +to tell remote devices apart. + +### The memory numbers are real, the budget's model of them is not + +RPC devices report the remote host's actual free and total memory, so +`LlamaCppEx.devices/0` is accurate. But `ModelManager`'s `:memory_budget` derives +placement from `:split_mode` / `:tensor_split` / `:main_gpu` and has no concept +of a device being a network away — it will budget a remote device as if it were +local VRAM. The numbers are not nonsense, but do not read more into them than +that. + +### It buys capacity, not speed + +Pipeline parallelism is disabled whenever an RPC device participates: the RPC +backend reports `async = false, events = false`, and llama.cpp checks that before +enabling pipelining. The two nodes therefore execute **sequentially**. A model +that already fits on one machine gains nothing from a second one. See +[DGX Spark](dgx-spark.md) for the measurements. + +> #### A peer failure aborts the VM {: .error} +> +> Every client-side RPC command checks its result with `GGML_ABORT`. A peer that +> crashes or a network that drops kills the OS process, BEAM included. There is +> nothing to rescue. Registration is checkable up front — use +> `LlamaCppEx.RPC.ping/1` — and after that the VM is the unit of restart. See +> `LlamaCppEx.RPC`. + ## Placement-aware memory budget `:memory_budget` knows whether a model lands in RAM or on specific GPUs and diff --git a/docs/release-guide.md b/docs/release-guide.md index 7d22316..a7056e0 100644 --- a/docs/release-guide.md +++ b/docs/release-guide.md @@ -36,15 +36,38 @@ git -C vendor/llama.cpp rev-parse HEAD Before building, verify the llama.cpp APIs used by the NIF haven't changed: +Re-derive the header list if you add an include -- a hand-kept list drifts, and +that is exactly how the `common/speculative.h` break below reached a build: + ```bash -# Diff the public header between old and new commits -git -C vendor/llama.cpp diff .. -- include/llama.h +grep -hoE '^#include [<"](llama|ggml|chat|json-schema|speculative)[^">]*' \ + c_src/llama_cpp_ex/*.cpp c_src/llama_cpp_ex/*.h | sort -u +``` -# Diff common headers used by the NIF -git -C vendor/llama.cpp diff .. -- common/chat.h -git -C vendor/llama.cpp diff .. -- common/json-schema-to-grammar.h +```bash +for h in include/llama.h \ + ggml/include/ggml-backend.h ggml/include/ggml-rpc.h \ + common/chat.h common/json-schema-to-grammar.h common/speculative.h; do + echo "##### $h" + git -C vendor/llama.cpp diff .. -- "$h" +done ``` +A signature change is the easy case -- it fails to compile with a clear message. +Watch for two harder ones: + +- **A function removed outright.** The compiler's "did you mean" is actively + misleading: when `common_speculative_need_embd` was deleted in `f785fc9ea`, + both GCC and clang suggested the unrelated `common_speculative_n_max`, whose + first parameter happens to be a different pointer type. Check the header diff + for `-` lines before believing the suggestion. +- **A default value changed** with the signature intact. Nothing fails to + compile. `llama_model_default_params()` moved `load_mode` from + `LLAMA_LOAD_MODE_MMAP` to `LLAMA_LOAD_MODE_AUTO`; because the NIF always sets + that field explicitly, behaviour did not change -- but a field we left at its + default would have shifted silently. Diff + `llama_model_default_params` / `llama_context_default_params` on every bump. + The NIF uses these key APIs (grep `llama_nif.cpp` for the full list): - `llama_model_*`, `llama_context_*`, `llama_vocab_*` — model/context/vocab management - `llama_tokenize`, `llama_detokenize`, `llama_token_to_piece` — tokenization @@ -55,9 +78,43 @@ The NIF uses these key APIs (grep `llama_nif.cpp` for the full list): - `llama_chat_apply_template` — legacy chat templates - `common_chat_templates_init`, `common_chat_templates_apply` — Jinja chat templates - `json_schema_to_grammar` — grammar generation +- `common_speculative_*` — speculative decoding and MTP draft models +- `ggml_backend_dev_*`, `ggml_backend_reg_*` — device enumeration for `:devices` +- `ggml_backend_rpc_add_server`, `ggml_backend_rpc_start_server` — RPC backend If any signatures changed, update `c_src/llama_cpp_ex/llama_nif.cpp` and/or `llama_nif.h`. +### Upstream defects we work around + +Three known llama.cpp defects have workarounds in this repo. A bump is the only +time anyone looks at them, so check each one here — if upstream has fixed it, the +workaround should come out rather than quietly accumulate. + +Each was measured against `4801e3c567d5` (b10362) on NVIDIA DGX Spark (GB10, +aarch64, GCC 13.3, CUDA 13.0). Full reports, with reproductions and suggested +upstream fixes, are drafted in +`.claude/plans/dgx-spark-2node/upstream-issues.md` — not yet filed, so there are +no issue URLs to link. **When they are filed, put the URLs in this table.** + +Re-checked at `a94d563ed801` (61 commits later): all three still stand. That +check was a source diff, not a re-measurement — the files each defect lives in +(`ggml/src/ggml-cpu/CMakeLists.txt`, `ggml_backend_cuda_comm_init`, and +`ggml_backend_rpc_start_server`) were untouched by the bump. A source diff is +enough to say a defect is *still there*; it is not enough to say it is *gone*, +so if a diff ever shows movement, run the command in the last column. + +| # | Upstream defect | Our workaround | Still needed? | +|---|---|---|---| +| 1 | `GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to **base ARMv8-A** on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel. | `LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See [DGX Spark](dgx-spark.md) and [Cross-Platform Builds](cross-platform-builds.md). | `scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe. | +| 2 | `-sm tensor` with a non-CUDA device in the set **runs and is correct but ~2.7× slower on decode**: `ggml_backend_cuda_comm_init` returns `nullptr` on any non-CUDA member, so the generic meta-backend butterfly runs instead, and the RPC backend's `NULL` 2-D tensor hooks degrade it to a loop of 1-D transfers. | Documented, not coded around: `Model.load/2` maps `:tensor` to its upstream value and the docs say to use `:layer` across hosts. See the tp=2 verdict in [DGX Spark](dgx-spark.md). | `mix run bench/spark_tensor_split.exs remote`. If `:tensor` comes within range of `:layer`, upstream implemented the 2-D hooks or the all-reduce — update the verdict section. | +| 3 | `ggml_backend_rpc_start_server` returns `void`, never returns on success, and prints failures to stderr, so an **embedded** caller cannot tell "listening" from "port in use". | `rpc_start_server` in `llama_nif.cpp` pre-`bind()`s the endpoint for a real `errno`, then polls `connect()` until something accepts. A TOCTOU window and one wasted connection per start. | Check whether the signature gained a return value or a listening callback. If so, delete `rpc_preflight_bind` and `rpc_wait_until_listening` and drop the poll. | + +Not a defect and not going away: `RPC_STATUS_ASSERT` is `GGML_ABORT` +(`ggml-rpc.cpp:30`), so a peer failure terminates the client process — the BEAM +included. That is upstream's deliberate design. `LlamaCppEx.RPC` documents it; +see also the `:row` split mode, which throws on CUDA at this version and which we +deliberately do not work around. + ## 3. Build and test ```bash diff --git a/lib/llama_cpp_ex/model.ex b/lib/llama_cpp_ex/model.ex index e51acc1..4ac5b67 100644 --- a/lib/llama_cpp_ex/model.ex +++ b/lib/llama_cpp_ex/model.ex @@ -15,7 +15,9 @@ defmodule LlamaCppEx.Model do :use_mmap, :use_mlock, :use_direct_io, - :check_tensors + :check_tensors, + :rpc_servers, + :devices ] @structural_option_keys [:n_gpu_layers, :vocab_only, :load_mtp] @@ -47,8 +49,9 @@ defmodule LlamaCppEx.Model do Defaults to `99` (offload all layers). * `:use_mmap` - Whether to memory-map the model file. Defaults to `true`. * `:main_gpu` - GPU device index for single-GPU mode. Defaults to `0`. - * `:split_mode` - How to split the model across GPUs: `:none`, `:layer`, or `:row`. - Defaults to `:none`. + * `:split_mode` - How to split the model across devices: `:none`, `:layer`, + `:row` or `:tensor`. Defaults to `:none`. Only `:none` and `:layer` are + generally usable at this llama.cpp version — see the note below. * `:tensor_split` - List of floats specifying the proportion of work per GPU (e.g. `[0.5, 0.5]` for two GPUs). Defaults to `[]`. * `:use_mlock` - Pin model memory in RAM to prevent swapping. Implies `:use_mmap`. @@ -63,6 +66,36 @@ defmodule LlamaCppEx.Model do who are not doing speculative decoding do not pay for the extra tensors. Required for `LlamaCppEx.MTP.init/2`, which refuses a model loaded without it — the layers cannot be added after the fact. + * `:rpc_servers` - Endpoints (`"host:port"`) to register before loading, so + their remote devices can hold part of the model. Defaults to `[]`. Requires + a build with `LLAMA_RPC=1`. See `LlamaCppEx.RPC`. Note that llama.cpp puts + remote devices **first** in its automatic placement list — which is not the + order `LlamaCppEx.devices/0` reports — so `tensor_split: [0.25, 0.75]` + gives 25% to the first remote endpoint. Pass `:devices` to avoid guessing. + * `:devices` - Device names, e.g. `["CUDA0", "RPC0"]`, used **verbatim** as + the placement list: no reordering, no dedup, no CPU filtering. Defaults to + `[]`, which lets llama.cpp build the list itself. Set this whenever more + than one device is in play, because the automatic list is **not** the order + `LlamaCppEx.devices/0` reports — it puts RPC devices first — so + `:tensor_split` and `:main_gpu` would index a list you never saw. With + `:devices` set, they index this one. + + > #### Split modes at llama.cpp b10362 {: .warning} + > + > `:layer` splits contiguous layer ranges across devices, one KV cache per + > device, and is the only mode that works across hosts. + > + > `:row` throws at load time on CUDA: ggml-cuda no longer exports + > `ggml_backend_split_buffer_type`, so `llama_model_load` raises + > `device CUDA0 does not support split buffers`. It is kept mapped to its + > upstream value rather than removed, because the enum is upstream's, but do + > not build on it. Only SYCL still declares a split buffer type. + > + > `:tensor` is real tensor parallelism via a Meta device, added in + > llama.cpp #19378. It forces flash attention on, refuses some architectures, + > disables backend sampling, and its CUDA all-reduce is `ncclCommInitAll` — + > a single-process, all-local-GPUs API. **It cannot span hosts**, so it is not + > "tp=2 across two machines". See `docs/dgx-spark.md` for the measurements. > #### Load mode {: .info} > @@ -102,28 +135,59 @@ defmodule LlamaCppEx.Model do vocab_only = Keyword.get(opts, :vocab_only, false) check_tensors = Keyword.get(opts, :check_tensors, false) load_mtp = Keyword.get(opts, :load_mtp, false) + # Read here rather than in a private helper: test/option_forwarding_test.exs + # checks both that `load/2` accepts the key and that it is one of + # `tuning_option_keys/0`, and a helper satisfies only half of that. + rpc_servers = Keyword.get(opts, :rpc_servers, []) + devices = Keyword.get(opts, :devices, []) + + # Registration has to happen before the load, not during it: tensor + # placement is computed from the devices that exist when + # llama_model_load_from_file runs. An unreachable endpoint is reported here + # rather than silently dropped, because ggml_backend_register no-ops on a + # null registration and the model would then load onto the wrong devices. + with :ok <- register_rpc_servers(rpc_servers), + {:ok, ref} <- + LlamaCppEx.NIF.model_load( + path, + n_gpu_layers, + use_mmap, + main_gpu, + split_mode, + tensor_split, + use_mlock, + use_direct_io, + vocab_only, + check_tensors, + load_mtp, + devices + ) do + {:ok, %__MODULE__{ref: ref, load_mtp: load_mtp}} + end + end + + defp register_rpc_servers([]), do: :ok - case LlamaCppEx.NIF.model_load( - path, - n_gpu_layers, - use_mmap, - main_gpu, - split_mode, - tensor_split, - use_mlock, - use_direct_io, - vocab_only, - check_tensors, - load_mtp - ) do - {:ok, ref} -> {:ok, %__MODULE__{ref: ref, load_mtp: load_mtp}} - {:error, _} = error -> error + defp register_rpc_servers(endpoints) do + case LlamaCppEx.RPC.add_servers(endpoints) do + {:ok, _n} -> :ok + {:error, {endpoint, reason}} -> {:error, "RPC endpoint #{endpoint}: #{reason}"} end end - defp encode_split_mode(:none), do: 0 - defp encode_split_mode(:layer), do: 1 - defp encode_split_mode(:row), do: 2 + @doc false + # Exposed for tests: the mapping is upstream's `llama_split_mode` enum and a + # silent drift here would place tensors somewhere nobody asked for. + # 0 none, 1 layer, 2 row, 3 tensor (llama.h). + def encode_split_mode(:none), do: 0 + def encode_split_mode(:layer), do: 1 + def encode_split_mode(:row), do: 2 + def encode_split_mode(:tensor), do: 3 + + def encode_split_mode(other) do + raise ArgumentError, + "unknown split_mode #{inspect(other)}, expected :none, :layer, :row or :tensor" + end @doc "Returns the training context size of the model." @spec n_ctx_train(t()) :: integer() diff --git a/lib/llama_cpp_ex/nif.ex b/lib/llama_cpp_ex/nif.ex index bcfa3f0..b816e7f 100644 --- a/lib/llama_cpp_ex/nif.ex +++ b/lib/llama_cpp_ex/nif.ex @@ -24,6 +24,14 @@ defmodule LlamaCppEx.NIF do # Devices def device_list, do: :erlang.nif_error(:not_loaded) + # RPC. Present in every build; they return {:error, :rpc_unsupported} unless + # the NIF was compiled with LLAMA_RPC=1. + def rpc_supported, do: :erlang.nif_error(:not_loaded) + def rpc_add_server(_endpoint), do: :erlang.nif_error(:not_loaded) + + def rpc_start_server(_endpoint, _cache_dir, _n_threads, _device_names), + do: :erlang.nif_error(:not_loaded) + # Model def model_load( _path, @@ -36,7 +44,8 @@ defmodule LlamaCppEx.NIF do _use_direct_io, _vocab_only, _check_tensors, - _load_mtp + _load_mtp, + _devices ), do: :erlang.nif_error(:not_loaded) diff --git a/lib/llama_cpp_ex/rpc.ex b/lib/llama_cpp_ex/rpc.ex new file mode 100644 index 0000000..06ac544 --- /dev/null +++ b/lib/llama_cpp_ex/rpc.ex @@ -0,0 +1,178 @@ +defmodule LlamaCppEx.RPC do + @moduledoc """ + Remote ggml devices over the llama.cpp RPC backend. + + An RPC *worker* runs `LlamaCppEx.RPC.Server` and exposes its local devices on a + TCP endpoint. A *client* registers that endpoint with `add_server/1`, after + which the remote devices appear in `LlamaCppEx.devices/0` and can hold part of + a model — so a model larger than one machine's memory can be loaded across two. + + # on the worker + {:ok, _} = LlamaCppEx.RPC.Server.start_link(endpoint: "10.100.64.2:50052") + + # on the client, before loading + {:ok, 1} = LlamaCppEx.RPC.add_server("10.100.64.2:50052") + {:ok, model} = LlamaCppEx.Model.load(path, split_mode: :layer, tensor_split: [0.5, 0.5]) + + `LlamaCppEx.Model.load/2` accepts `:rpc_servers` and does the registration for + you, in the right order. + + ## Build requirement + + The RPC backend is not compiled in by default. Build with `LLAMA_RPC=1`; + otherwise every function here returns `{:error, :rpc_unsupported}`. + + LLAMA_RPC=1 LLAMA_BACKEND=cuda mix compile + + On Linux the transport auto-negotiates RDMA when both peers have a usable HCA, + which is a build-time choice (`LLAMA_RPC_RDMA`, default on) with no runtime + switch. See `LlamaCppEx.RPC.Server` for how to tell which transport you got. + + > #### A peer failure kills the VM {: .error} + > + > This is a property of upstream llama.cpp, not of this binding, and stating it + > is better than hiding it. Every client-side RPC command checks its result + > with `RPC_STATUS_ASSERT`, which is `GGML_ABORT` — so a peer that crashes, + > a network that drops, or a malformed response **terminates the OS process**, + > taking the BEAM with it. There is no error return, no retry and no reconnect + > to catch. + > + > Registration is the exception and the reason this module is shaped the way it + > is: an unreachable endpoint is reported as `{:error, :unreachable}` rather + > than aborting, so a two-node setup can be *checked* before a model load and + > is only *fatal* during one. Check with `ping/1` before you load; treat the VM + > as the unit of restart afterwards. Real fault isolation would mean running + > the RPC client in a separate OS process, which is a different architecture. + + ## Two device orderings, and they disagree + + `LlamaCppEx.devices/0` reports the ggml **registry**, which is registration + order: local backends first, RPC endpoints appended as they register. + llama.cpp builds a **different** list for placement at load time and inserts + RPC devices at the **front** of it, to minimise network transfers. So with one + local GPU and one endpoint, `devices/0` shows `[CUDA0, CPU, RPC0]` while + `tensor_split` indexes `[RPC0, CUDA0]` — `tensor_split: [0.25, 0.75]` puts 25% + on the **remote** node, and `main_gpu: 0` selects it. + + A backwards split still produces correct tokens and simply runs badly, so + nothing tells you. `:gpu_index` from `devices/0` does **not** index + `:tensor_split` once a remote device exists. Pass `:devices` to + `LlamaCppEx.Model.load/2` — it is used verbatim — and stop guessing: + + LlamaCppEx.Model.load(path, + rpc_servers: ["10.100.64.2:50052"], + devices: ["CUDA0", "RPC0"], + split_mode: :layer, + tensor_split: [0.6, 0.4]) + + See `docs/multi-gpu.md` for the worked example. + """ + + alias LlamaCppEx.NIF + + @type endpoint :: String.t() + @type error :: :rpc_unsupported | :unreachable + + @doc """ + Whether this build has the RPC backend compiled in. + + `false` means the NIF was built without `LLAMA_RPC=1`, and every other function + in this module will return `{:error, :rpc_unsupported}`. + + iex> LlamaCppEx.RPC.supported?() + false + + Worth checking rather than inferring from an error, because + `:rpc_unsupported` and `:unreachable` are easy to confuse and mean very + different things: the first is a build problem, the second is a network or + version-mismatch problem. Code that treats them alike will eventually paper + over an artifact built with the wrong flags. + """ + @spec supported?() :: boolean() + def supported?, do: NIF.rpc_supported() + + @doc """ + Registers a remote endpoint's devices in the global device registry. + + Returns the number of devices the endpoint contributed. Idempotent: upstream + memoizes the registration per endpoint, so registering the same endpoint twice + succeeds and reports `0` added the second time. + + Must be called **before** `LlamaCppEx.Model.load/2`, because tensor placement + is computed from the devices that exist at load time. + + iex> LlamaCppEx.RPC.add_server("10.100.64.2:50052") + {:ok, 1} + + ## Errors + + * `{:error, :rpc_unsupported}` — the NIF was built without `LLAMA_RPC=1`. + * `{:error, :unreachable}` — nothing answered, or the peer's RPC protocol + major/minor did not match ours. Upstream collapses both to a null + registration, and a null registration is silently ignored by + `ggml_backend_register`, so this check is the only thing standing between + you and a model that quietly loads onto the wrong devices. + """ + @spec add_server(endpoint()) :: {:ok, non_neg_integer()} | {:error, error()} + def add_server(endpoint) when is_binary(endpoint) do + NIF.backend_init() + NIF.rpc_add_server(endpoint) + end + + @doc """ + Registers several endpoints, in order. + + Stops at the first failure and reports which endpoint failed, because a + partially registered set would place tensors somewhere nobody intended. + + iex> LlamaCppEx.RPC.add_servers(["10.100.64.2:50052", "10.100.64.3:50052"]) + {:ok, 2} + """ + @spec add_servers([endpoint()]) :: + {:ok, non_neg_integer()} | {:error, {endpoint(), error()}} + def add_servers(endpoints) when is_list(endpoints) do + Enum.reduce_while(endpoints, {:ok, 0}, fn endpoint, {:ok, total} -> + case add_server(endpoint) do + {:ok, n} -> {:cont, {:ok, total + n}} + {:error, reason} -> {:halt, {:error, {endpoint, reason}}} + end + end) + end + + @doc """ + The registered remote devices, in placement order. + + A filtered view of `LlamaCppEx.devices/0`. Two things to know about what comes + back: + + * `:type` is always `:gpu`, even when the remote server exposes only a CPU + device — upstream hardcodes it with a TODO. + * `:description` is the endpoint string, which is the only way to tell two + remote devices apart. + + """ + @spec devices() :: [map()] + def devices do + Enum.filter(LlamaCppEx.devices(), &(&1.backend == "RPC")) + end + + @doc """ + Reports whether an endpoint is reachable and speaks a compatible protocol. + + This is `add_server/1` under a name that says what it is good for. It has the + same side effect — a reachable endpoint stays registered — and it is **not** a + health check you can repeat to monitor a peer: the answer is memoized after the + first success, and once a model is loaded a dead peer aborts the VM rather than + failing a probe. + + Use it once, before loading, to turn "the model landed on the wrong devices" + into a clear error. + """ + @spec ping(endpoint()) :: :ok | {:error, error()} + def ping(endpoint) when is_binary(endpoint) do + case add_server(endpoint) do + {:ok, _} -> :ok + {:error, reason} -> {:error, reason} + end + end +end diff --git a/lib/llama_cpp_ex/rpc/server.ex b/lib/llama_cpp_ex/rpc/server.ex new file mode 100644 index 0000000..4bcfe68 --- /dev/null +++ b/lib/llama_cpp_ex/rpc/server.ex @@ -0,0 +1,159 @@ +defmodule LlamaCppEx.RPC.Server do + @moduledoc """ + The worker side of the llama.cpp RPC backend: serves this node's devices to a + remote client. + + children = [ + {LlamaCppEx.RPC.Server, + endpoint: "10.100.64.2:50052", + cache_dir: Path.expand("~/.cache/llama.cpp/rpc"), + n_threads: 10} + ] + + ## Bind to the fabric address, not localhost + + `endpoint` is required and there is deliberately no default. Upstream's default + is `127.0.0.1`, which is useless here twice over: a remote client cannot reach + it, and **RDMA can never engage on it**, because the transport selects an HCA + by matching a GID against the socket's *local* address. + + Nothing about this endpoint is authenticated or encrypted. It is a plain TCP + port that accepts commands to allocate memory and execute compute graphs. Bind + it to a point-to-point fabric address, never to a routable interface. + + ## The tensor cache is worth having + + `:cache_dir` enables upstream's content-addressed cache for tensors over + 10 MiB. Without it every model load re-pushes the whole remote share across the + network; with it a warm load is close to free. There is no default — pass a + path or accept the cost knowingly. + + ## Which transport did I get? + + Transport selection is silent auto-negotiation. There is no env var, no + endpoint scheme and no return value that tells you. The only signal is the + worker's own log with `GGML_RPC_DEBUG=1`: the absence of + `RDMA activate failed, staying on TCP` means RDMA is live. To force TCP for an + A/B, set `GGML_RDMA_DEV` to a device name that does not exist. + + ## This process cannot stop the server + + `ggml_backend_rpc_start_server` never returns — its accept loop is + `while (true)` and the cleanup after it is unreachable — so the native server + runs on a detached thread that outlives this GenServer. `terminate/2` therefore + does nothing but say so. **The OS process is the unit of restart**, and that + matters in practice: upstream's RPC worker is reported by multiple independent + users to grow RSS during inference and never release it, so a long-lived worker + wants restarting between runs and watching during them. + + Starting it twice on one endpoint fails cleanly at the bind, which is the one + guard rail this design does provide — and it is also why this child is + **`restart: :temporary`**. A restart could not succeed: the listening socket + lives on that detached thread inside the same OS process, so it survives this + GenServer dying, and the next `init/1` would fail `EADDRINUSE` at the + pre-flight bind (`SO_REUSEADDR` does not let you bind over a socket that is + actively listening). Under the default `:permanent` a single crash would retry, + fail identically, exhaust `max_restarts` and take the whole supervision subtree + down — reporting a bind error rather than the original cause. `:temporary` is + what "the OS process is the unit of restart" means as a child spec. + + This module exists to own a native resource's lifecycle and to be supervised, + which is the only reason to add a process. It carries no state a function could + not compute. + """ + + use GenServer, restart: :temporary + + require Logger + + alias LlamaCppEx.NIF + + @type option :: + {:endpoint, String.t()} + | {:cache_dir, String.t() | nil} + | {:n_threads, pos_integer()} + | {:devices, [String.t()]} + | {:name, GenServer.name()} + + @doc """ + Starts the RPC worker. + + ## Options + + * `:endpoint` — **required**, `"host:port"`. Bind to a fabric address. + * `:cache_dir` — content-addressed tensor cache directory. Default `nil` + (no cache). + * `:n_threads` — CPU threads for the served backends. Default `4`, matching + upstream. On a DGX Spark use `10` and pin the process to cores `5-9,15-19`; + the performance and efficiency clusters are interleaved, so the default + affinity spreads work across both. + * `:devices` — device names to serve, e.g. `["CUDA0"]`. Default: every + non-CPU device, falling back to the CPU device. + + Returns `{:error, {:rpc, reason}}` rather than starting when the endpoint + cannot be served. Notable reasons: + + * `:rpc_unsupported` — built without `LLAMA_RPC=1`. + * `:bind_timeout` — the native thread never reached `listen`. + * `:no_devices` — nothing to serve. + * a string — a bad endpoint, an unknown device name, or the bind's `errno`. + """ + @spec start_link([option()]) :: GenServer.on_start() + def start_link(opts) do + {name, opts} = Keyword.pop(opts, :name) + GenServer.start_link(__MODULE__, opts, if(name, do: [name: name], else: [])) + end + + @doc "The endpoint this worker is serving, and the device names it exposes." + @spec info(GenServer.server()) :: %{endpoint: String.t(), devices: [String.t()]} + def info(server), do: GenServer.call(server, :info) + + @impl true + def init(opts) do + # Without this, `terminate/2` below never runs in the case it was written + # for. GenServer only turns a parent exit signal into a `terminate/2` call + # when the process traps exits, and a Supervisor shuts a child down with + # `exit(pid, :shutdown)` — so the supervised deployment in this module's own + # child-spec example died silently and the "the native server keeps running" + # warning never reached the operator who most needs it. + Process.flag(:trap_exit, true) + + endpoint = Keyword.fetch!(opts, :endpoint) + cache_dir = Keyword.get(opts, :cache_dir) + n_threads = Keyword.get(opts, :n_threads, 4) + devices = Keyword.get(opts, :devices, []) + + if cache_dir, do: File.mkdir_p!(cache_dir) + + NIF.backend_init() + + case NIF.rpc_start_server(endpoint, cache_dir || "", n_threads, devices) do + {:ok, served} -> + Logger.info( + "RPC server listening on #{endpoint}, serving #{Enum.join(served, ", ")} " <> + "(#{n_threads} threads, cache #{cache_dir || "disabled"})" + ) + + {:ok, %{endpoint: endpoint, devices: served}} + + {:error, reason} -> + {:stop, {:rpc, reason}} + end + end + + @impl true + def handle_call(:info, _from, state), do: {:reply, state, state} + + @impl true + def terminate(_reason, state) do + # Honest rather than reassuring: upstream's accept loop never returns and + # there is no shutdown hook, so the native thread and its port outlive this + # process. Only exiting the VM frees them. + Logger.warning( + "RPC server process stopping, but the native server on #{state.endpoint} keeps running. " <> + "Restart the VM to release the port." + ) + + :ok + end +end diff --git a/lib/llama_cpp_ex/server.ex b/lib/llama_cpp_ex/server.ex index dd71be7..7002eac 100644 --- a/lib/llama_cpp_ex/server.ex +++ b/lib/llama_cpp_ex/server.ex @@ -312,9 +312,12 @@ defmodule LlamaCppEx.Server do from the options above and cannot be overridden here. * Model loading options are forwarded to `LlamaCppEx.Model.load/2` — `:main_gpu`, `:split_mode`, `:tensor_split`, `:use_mmap`, `:use_mlock`, - `:use_direct_io` and `:check_tensors`. The three load flags collapse into - llama.cpp's single `load_mode`: `:use_direct_io` wins outright, otherwise - `:use_mlock` and `:use_mmap` combine; see `LlamaCppEx.Model.load/2`. + `:use_direct_io`, `:check_tensors` and `:rpc_servers`. The three load flags + collapse into llama.cpp's single `load_mode`: `:use_direct_io` wins + outright, otherwise `:use_mlock` and `:use_mmap` combine; see + `LlamaCppEx.Model.load/2`. `:rpc_servers` registers remote endpoints before + the load so their devices can hold part of the model — see + `LlamaCppEx.RPC`, including the caveat that a peer failure aborts the VM. * GenServer options like `:name`. """ diff --git a/mix.exs b/mix.exs index 96cd81a..b548849 100644 --- a/mix.exs +++ b/mix.exs @@ -233,6 +233,7 @@ defmodule LlamaCppEx.MixProject do "LICENSE", "docs/architecture.md", "docs/cross-platform-builds.md", + "docs/dgx-spark.md", "docs/adr/001-cpp-nif-over-rustler.md", "docs/adr/002-fine-for-nif-ergonomics.md", "docs/adr/003-static-linking.md", @@ -241,7 +242,9 @@ defmodule LlamaCppEx.MixProject do "docs/adr/006-continuous-batching.md", "docs/adr/007-prefix-caching.md", "docs/adr/008-batching-strategies.md", + "docs/adr/009-multi-model-manager.md", "docs/examples.md", + "docs/multi-gpu.md", "docs/performance.md", "docs/release-guide.md" ], @@ -277,16 +280,38 @@ defmodule LlamaCppEx.MixProject do # the inherited environment anyway, but listing them keeps the build's input # contract in one place: # - # LLAMA_BACKEND auto | metal | cuda | vulkan | cpu - # LLAMA_CMAKE_ARGS extra flags appended to the llama.cpp cmake invocation - # LLAMA_PORTABLE 1 to drop -march=native, set by the precompile workflow - # LLAMA_CUDA_NCCL 1 to build and link ggml's NCCL multi-GPU collectives, - # which also makes libnccl.so.2 a load-time requirement + # LLAMA_BACKEND auto | metal | cuda | vulkan | cpu + # LLAMA_CMAKE_ARGS extra flags appended to the llama.cpp cmake invocation + # LLAMA_PORTABLE 1 to drop -march=native, set by the precompile workflow + # LLAMA_CUDA_NCCL 1 to build and link ggml's NCCL multi-GPU collectives, + # which also makes libnccl.so.2 a load-time requirement + # LLAMA_CPU_ARM_ARCH the ARM architecture string for ggml's CPU backend, for + # hosts where -mcpu=native degrades silently (GB10 on GCC + # 13.3). Requires LLAMA_CUDA_ARCH on a CUDA build; the + # Makefile errors otherwise, because reaching this flag + # needs GGML_NATIVE=OFF and that turns one CUDA arch into + # seven. + # LLAMA_CUDA_ARCH CMAKE_CUDA_ARCHITECTURES, e.g. 121a-real for GB10 + # LLAMA_RPC 1 to build the ggml RPC backend, which lets a model's + # layers live on another host. Off by default: it is a + # networked surface and a protocol version coupling. + # LLAMA_RPC_RDMA 1 (default) to use RDMA for the RPC transport on Linux. + # Declared rather than auto-detected, and paired with + # -libverbs on the link line. + # + # None of these need to appear in make_force_build: each is part of the + # Makefile's build-directory key, so changing one lands in a different tree + # with its own CMakeCache.txt and rebuilds on its own. That is a better answer + # than forcing a rebuild, because switching back is still a cache hit. @make_env_passthrough [ "LLAMA_BACKEND", "LLAMA_CMAKE_ARGS", "LLAMA_PORTABLE", - "LLAMA_CUDA_NCCL" + "LLAMA_CUDA_NCCL", + "LLAMA_CPU_ARM_ARCH", + "LLAMA_CUDA_ARCH", + "LLAMA_RPC", + "LLAMA_RPC_RDMA" ] defp make_env do diff --git a/scripts/spark/bootstrap.sh b/scripts/spark/bootstrap.sh new file mode 100755 index 0000000..b230419 --- /dev/null +++ b/scripts/spark/bootstrap.sh @@ -0,0 +1,182 @@ +#!/usr/bin/env bash +# Idempotent per-node provisioning for the DGX Sparks. No sudo, ever. +# +# scripts/spark/bootstrap.sh # both nodes +# scripts/spark/bootstrap.sh spark-2 # one node +# scripts/spark/bootstrap.sh --facts-only # just print the fact sheet +# +# Neither box has passwordless sudo, so nothing here installs a package, writes +# a sysctl, touches the kernel cmdline or adds a system unit. It also does not +# touch ~/.ssh on either node. +# +# The one substantive job is the toolchain. spark-2 has no asdf at all and we +# cannot `apt install` the OTP build dependencies, so `asdf install erlang` +# there is not a plan. The two boxes are identical — same distro, kernel, arch — +# and kerl-built OTP links only against libraries present on both, so the +# primary path is to copy spark-1's ~/.asdf across the fabric. The fallback is +# reported for the user to action with their password rather than silently +# degrading to Ubuntu's Elixir 1.14 (mix.exs:143 requires ~> 1.18). + +set -euo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +facts_only=0 +target=all + +while [ $# -gt 0 ]; do + case "$1" in + --facts-only) facts_only=1; shift ;; + -h | --help) sed -n '2,19p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) spark_die "unknown option '$1' (try --help)" ;; + *) target=$1; shift ;; + esac +done + +nodes=() +while IFS= read -r _n; do nodes+=("$_n"); done < <(spark_resolve_nodes "$target") + +erlang_bin=".asdf/installs/erlang/${SPARK_ERLANG_VERSION}/bin" +elixir_bin=".asdf/installs/elixir/${SPARK_ELIXIR_VERSION}/bin" + +# --- Toolchain --------------------------------------------------------------- + +toolchain_present() { + ssh "$(spark_host "$1")" "test -x ~/$erlang_bin/erl && test -x ~/$elixir_bin/elixir" +} + +# Copy the donor's ~/.asdf over the 10.100.64 fabric. The nodes have no keys for +# each other and the plan forbids adding any, so authentication rides a +# forwarded agent. ControlPath=none is required: the multiplexed master was +# opened without -A, and -A on a session that reuses it does nothing. +copy_toolchain_over_fabric() { + local donor=$1 target_node=$2 donor_ip + donor_ip=$(spark_fabric_ip "$donor" 0) + + ssh -o ControlPath=none -A "$(spark_host "$target_node")" \ + "rsync -a --delete -e 'ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' \ + '$donor_ip:.asdf/' '.asdf/' && rsync -a -e 'ssh -o BatchMode=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null' \ + '$donor_ip:.tool-versions' '.tool-versions'" +} + +# Fallback when the agent is not forwardable: relay through the control node. +# 431 MB, so it is the slow path, not the wrong one. +copy_toolchain_via_control_node() { + local donor=$1 target_node=$2 tmp + tmp=$(mktemp -d) + # shellcheck disable=SC2064 # $tmp must expand now, not at trap time + trap "rm -rf '$tmp'" RETURN + rsync -a "$(spark_host "$donor"):.asdf/" "$tmp/asdf/" + rsync -a "$(spark_host "$donor"):.tool-versions" "$tmp/tool-versions" + rsync -a --delete "$tmp/asdf/" "$(spark_host "$target_node"):.asdf/" + rsync -a "$tmp/tool-versions" "$(spark_host "$target_node"):.tool-versions" +} + +verify_toolchain() { + local node=$1 out + out=$(ssh "$(spark_host "$node")" "PATH=\$HOME/$erlang_bin:\$HOME/$elixir_bin:\$PATH; \ + elixir -v 2>&1 | tail -1; \ + erl -noshell -eval 'io:format(\"otp=~s nif=~s~n\",[erlang:system_info(otp_release),erlang:system_info(nif_version)]),halt().'" 2>&1) || return 1 + printf '%s\n' "$out" + printf '%s' "$out" | grep -q "Elixir ${SPARK_ELIXIR_VERSION%%-*}" || return 1 + printf '%s' "$out" | grep -q 'otp=29 nif=2.18' || return 1 +} + +provision_toolchain() { + local node=$1 + + if toolchain_present "$node"; then + spark_log "$node: toolchain already installed" + elif [ "$node" = "$SPARK_TOOLCHAIN_DONOR" ]; then + spark_die "$node is the toolchain donor but has no erlang ${SPARK_ERLANG_VERSION} / elixir ${SPARK_ELIXIR_VERSION}. + Install them there first, or point SPARK_TOOLCHAIN_DONOR at a node that has them." + else + toolchain_present "$SPARK_TOOLCHAIN_DONOR" || + spark_die "donor $SPARK_TOOLCHAIN_DONOR has no usable toolchain to copy" + + spark_log "$node: copying ~/.asdf from $SPARK_TOOLCHAIN_DONOR over the fabric" + if ! copy_toolchain_over_fabric "$SPARK_TOOLCHAIN_DONOR" "$node"; then + spark_warn "$node: fabric copy failed (no forwardable agent?), relaying through the control node" + copy_toolchain_via_control_node "$SPARK_TOOLCHAIN_DONOR" "$node" + fi + fi + + if ! verify_toolchain "$node"; then + spark_warn "$node: the copied toolchain does not run. Fallback path: + + ssh $node + asdf plugin add erlang && asdf install erlang ${SPARK_ERLANG_VERSION} + + That needs the OTP build dependencies, which need your password: + + sudo apt install build-essential autoconf m4 libncurses-dev \\ + libssl-dev libwxgtk3.2-dev libgl1-mesa-dev libglu1-mesa-dev libpng-dev + + Do NOT fall back to the distro's Elixir: Ubuntu 24.04 ships 1.14 / OTP 25 and + mix.exs:143 requires ~> 1.18." + return 1 + fi +} + +# --- Fact sheet -------------------------------------------------------------- +# +# Printed so a later session can diff it. nvidia-smi cannot report GPU memory on +# GB10 (memory.total = [N/A], ATS addressing mode) — `free -h` is the number, +# and it is simultaneously host RAM and GPU memory. +fact_sheet() { + local node=$1 + ssh "$(spark_host "$node")" "bash -s" <<'FACTS' +set -u +printf 'hostname %s\n' "$(hostname)" +printf 'kernel %s\n' "$(uname -r)" +printf 'distro %s\n' "$(. /etc/os-release && echo "$PRETTY_NAME")" +printf 'driver %s\n' "$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -1)" +printf 'gpu %s\n' "$(nvidia-smi --query-gpu=name,compute_cap --format=csv,noheader 2>/dev/null | head -1)" +printf 'cuda %s\n' "$(/usr/local/cuda/bin/nvcc --version 2>/dev/null | sed -n 's/.*release \([0-9.]*\).*/\1/p')" +printf 'memory %s\n' "$(free -h | awk '/^Mem:/{print $2" total, "$7" available"}')" +printf 'disk %s\n' "$(df -h "$HOME" | awk 'NR==2{print $4" free of "$2}')" +printf 'cores %s online\n' "$(nproc)" +printf 'gcc %s\n' "$(gcc -dumpfullversion 2>/dev/null)" +printf 'cmake %s\n' "$(cmake --version 2>/dev/null | head -1 | awk '{print $3}')" +for iface in $(ip -o -4 addr show | awk '$4 ~ /^10\.100\./ {print $2}'); do + addr=$(ip -o -4 addr show dev "$iface" | awk '{print $4}') + mtu=$(cat "/sys/class/net/$iface/mtu") + state=$(cat "/sys/class/net/$iface/operstate") + printf 'fabric %-16s %-16s mtu %s %s\n' "$iface" "$addr" "$mtu" "$state" +done +for dev in /sys/class/infiniband/*; do + [ -e "$dev" ] || continue + printf 'hca %-10s width %s rate %s\n' "$(basename "$dev")" \ + "$(cat "$dev/ports/1/rate" 2>/dev/null | tr -s ' ')" \ + "$(cat "$dev/ports/1/phys_state" 2>/dev/null)" +done +printf 'toolchain %s\n' "$(ls -d "$HOME"/.asdf/installs/*/* 2>/dev/null | sed "s|$HOME/.asdf/installs/||" | tr '\n' ' ')" +FACTS + # Cluster topology comes from the local table, not from a probe: the point of + # printing it is that `taskset -c 0-9` is a trap, and that is a fact about the + # chip rather than about this boot. + printf 'cpu clusters X925 (big) %s, A725 (little) %s\n' "$SPARK_BIG_CORES" "$SPARK_LITTLE_CORES" +} + +# --- Main -------------------------------------------------------------------- + +status=0 + +for node in "${nodes[@]}"; do + printf '\n===== %s =====\n' "$node" + + ssh -o BatchMode=yes -o ConnectTimeout=10 "$(spark_host "$node")" true || + spark_die "$node is not reachable over ssh" + + if [ "$facts_only" -eq 0 ]; then + provision_toolchain "$node" || status=1 + + spark_log "$node: creating ~/$SPARK_MODELS_DIR, ~/$SPARK_RPC_CACHE_DIR, ~/$SPARK_REMOTE_DIR" + ssh "$(spark_host "$node")" "mkdir -p ~/$SPARK_MODELS_DIR ~/$SPARK_RPC_CACHE_DIR ~/$SPARK_REMOTE_DIR" + fi + + fact_sheet "$node" +done + +exit "$status" diff --git a/scripts/spark/cpuidle-matrix.sh b/scripts/spark/cpuidle-matrix.sh new file mode 100755 index 0000000..f694380 --- /dev/null +++ b/scripts/spark/cpuidle-matrix.sh @@ -0,0 +1,93 @@ +#!/usr/bin/env bash +# Measure decode latency under four process-placement conditions. +# +# scripts/spark/cpuidle-matrix.sh spark-1 +# +# The conditions, and why each is here: +# +# a-default no pinning, no poller. The number you get by doing nothing. +# b-poller a `nice -19` busy loop pinned to every cpu, holding the +# cores out of deep idle. This is the sudo-free stand-in for +# `idle=poll` on the kernel cmdline. On these boxes it takes +# ICMP RTT from 1.2 ms to 0.028 ms — a 43x effect on the +# network path. Whether it buys anything for *decode* is the +# question; if it does, that is the argument for asking the +# user to set cpuidle limits with their password. +# c-beam-busy BEAM scheduler busy-wait: +sbwt very_long and friends. Keeps +# Erlang's own schedulers spinning instead of sleeping, which +# is the in-VM equivalent of the poller and costs no cores. +# d-big-cores bound to the Cortex-X925 cluster, 5-9,15-19. NOT 0-9: the +# clusters are interleaved and 0-9 is mostly little cores. +# e-big+beam d plus c, since they address different sleeps. +# +# Output is a markdown table, ready to paste into bench/results/. + +set -euo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +node=${1:-spark-1} +spark_is_node "$node" || spark_die "unknown node '$node'" +shift || true + +here=$(dirname "${BASH_SOURCE[0]}") + +run() { + local label=$1 prefix=$2 erl_flags=$3 + spark_log "$label" + + "$here/remote.sh" --env MIX_ENV=bench --env "SPARK_COND=$label" \ + ${erl_flags:+--env "ELIXIR_ERL_OPTIONS=$erl_flags"} \ + "$node" bash -c "$prefix mix run bench/spark_cpuidle.exs" 2>&1 | + grep '^RESULT' || spark_warn "$label produced no result" +} + +# The poller: one `nice -19` spinner per cpu. nice -19 means it yields to +# anything real, so it costs throughput ~nothing while still preventing the +# core from entering a deep C-state. Started, measured against, and reaped — +# `pkill -f` would match this ssh session's own command line, so the pattern is +# escaped and the pids are tracked instead. +poller_start() { + ssh "$(spark_host "$node")" 'bash -s' <<'POLLER' +ncpu=$(nproc) +mkdir -p ~/.cache/spark-poller +: > ~/.cache/spark-poller/pids +for cpu in $(seq 0 $((ncpu - 1))); do + setsid nice -n 19 taskset -c "$cpu" bash -c 'while :; do :; done' >/dev/null 2>&1 & + echo $! >> ~/.cache/spark-poller/pids +done +echo "started $ncpu pollers" +POLLER +} + +poller_stop() { + ssh "$(spark_host "$node")" 'while read -r pid; do kill "$pid" 2>/dev/null || true; done < ~/.cache/spark-poller/pids; \ + rm -f ~/.cache/spark-poller/pids; echo "pollers stopped"' +} + +# Registered before any poller starts, and in the parent shell rather than inside +# the subshell below, so it survives the subshell dying. `run` can fail under +# `set -euo pipefail` (its `grep` tolerates no match, but remote.sh itself can +# fail), and without this the `b-poller` branch would leave one spinning process +# per cpu behind — silently contaminating every later measurement on the box. +# poller_stop is idempotent, so running it on the happy path too is free. +trap 'poller_stop >&2' EXIT + +results=$( + run "a-default" "" "" + + poller_start >&2 + # Give the pollers a moment to actually be scheduled everywhere. + sleep 2 + run "b-poller" "" "" + poller_stop >&2 + + run "c-beam-busy" "" "+sbwt very_long +sbwtdcpu very_long +sbwtdio very_long" + run "d-big-cores" "taskset -c $SPARK_BIG_CORES" "" + run "e-big+beam" "taskset -c $SPARK_BIG_CORES" "+sbwt very_long +sbwtdcpu very_long +sbwtdio very_long" +) + +printf '\n| condition | TTFT median ms | TTFT worst ms | decode t/s | ms/token |\n' +printf -- '|---|---|---|---|---|\n' +printf '%s\n' "$results" | awk -F'\t' '{ printf "| %s | %s | %s | %s | %s |\n", $2, $3, $4, $5, $6 }' diff --git a/scripts/spark/cpuidle-two-node.sh b/scripts/spark/cpuidle-two-node.sh new file mode 100755 index 0000000..039812b --- /dev/null +++ b/scripts/spark/cpuidle-two-node.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +# B5 — does holding the cores out of deep idle help a TWO-NODE run? +# +# scripts/spark/cpuidle-two-node.sh +# +# Phase 3 measured this single-node and found nothing: every condition landed +# within 2%, and the poller actively hurt TTFT. That is the expected answer for +# a GPU-bound decode loop on a CPU that never gets deeply idle anyway. +# +# Two nodes is a different question. Every token now involves a network wake on +# the far side, and LPI-3 exit latency on these boxes is 433 us — the effect +# that makes ICMP read 1.2 ms on a link whose real RTT is 1.39 us. If cpuidle +# costs anything anywhere, it costs it here. +# +# Pollers run on BOTH nodes, because the wake that matters is the worker's. +# `nice -19` so they yield to the real work; this is the sudo-free stand-in for +# `idle=poll`, which would need the kernel cmdline and a password. + +set -euo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +here=$(dirname "${BASH_SOURCE[0]}") +client=${SPARK_CLIENT_NODE:-spark-1} +worker=${SPARK_WORKER_NODE:-spark-2} + +poller_start() { + local node=$1 + ssh "$(spark_host "$node")" 'bash -s' <<'POLLER' >/dev/null +mkdir -p ~/.cache/spark-poller +: > ~/.cache/spark-poller/pids +for cpu in $(seq 0 $(($(nproc) - 1))); do + setsid nice -n 19 taskset -c "$cpu" bash -c 'while :; do :; done' >/dev/null 2>&1 & + echo $! >> ~/.cache/spark-poller/pids +done +POLLER + spark_log "$node: pollers up" +} + +poller_stop() { + local node=$1 + ssh "$(spark_host "$node")" 'while read -r pid; do kill "$pid" 2>/dev/null || true; done < ~/.cache/spark-poller/pids 2>/dev/null; \ + rm -f ~/.cache/spark-poller/pids' || true + spark_log "$node: pollers down" +} + +# Restart the worker between conditions: upstream's worker leaks, and a run that +# starts from a different RSS is not the same run. +measure() { + local label=$1 + "$here/rpc-worker.sh" restart "$worker" >/dev/null 2>&1 + spark_log "measuring $label" + + "$here/remote.sh" --env MIX_ENV=bench --env LLAMA_RPC=1 --big-cores --forward-agent \ + "$client" mix run bench/spark_two_node.exs b3 2>&1 | + grep '^| 120b' | sed "s/^| 120b two-node over RDMA/| $label/" +} + +a=$(measure "a-default") + +# The pollers are `setsid`'d so they survive the ssh session, and nothing reaps +# them but poller_stop. Under `set -euo pipefail` this script can die before +# reaching it in at least two ordinary ways -- rpc-worker.sh failing inside +# measure(), or the `grep` at the tail of that pipeline matching nothing -- and +# the result would be 20 spinning processes per node, on two machines, +# indefinitely, quietly contaminating every later measurement on those boxes. +# The trap is registered before the first poller starts and is idempotent. +trap 'poller_stop "$client"; poller_stop "$worker"' EXIT + +poller_start "$client" +poller_start "$worker" +sleep 2 +b=$(measure "b-pollers on both nodes") +poller_stop "$client" +poller_stop "$worker" +trap - EXIT + +printf '\n| condition | load s | prompt | TTFT ms | prefill t/s | decode t/s | worker RSS MiB |\n' +printf -- '|---|---|---|---|---|---|---|\n' +printf '%s\n%s\n' "$a" "$b" diff --git a/scripts/spark/fetch_models.exs b/scripts/spark/fetch_models.exs new file mode 100644 index 0000000..4377a1f --- /dev/null +++ b/scripts/spark/fetch_models.exs @@ -0,0 +1,171 @@ +# Fetch the benchmark models onto a Spark. +# +# scripts/spark/remote.sh spark-1 mix run scripts/spark/fetch_models.exs +# scripts/spark/remote.sh spark-1 mix run scripts/spark/fetch_models.exs 8b 30b +# scripts/spark/remote.sh spark-1 mix run scripts/spark/fetch_models.exs --list +# +# Uses LlamaCppEx.Hub.download/3 rather than curl or the huggingface CLI, so the +# download path this library ships is the one that gets exercised — including +# the SHA-256 verification, which is fail-closed by default. +# +# Files land at $LLAMA_CACHE_DIR///; remote.sh sets +# LLAMA_CACHE_DIR to ~/models. + +defmodule FetchModels do + # Sizes are the exact byte counts HuggingFace reports, resolved 2026-08-12. + # 121 GiB of unified memory is 130.0 GB, so: + # - gpt-oss-120b at 63.4 GB fits one node with room for KV. It is the A/B + # control for measuring pure RPC overhead: same model, one node vs two. + # - Qwen3-235B-A22B Q4_K_M is 142.1 GB across three shards. It does NOT fit + # one node, which is the entire justification for the second Spark. The + # plan estimated ~133 GB; the real number is 142.1 GB, which only makes + # the case stronger. + @models [ + %{ + label: "8b", + repo: "Qwen/Qwen3-8B-GGUF", + files: ["Qwen3-8B-Q4_K_M.gguf"], + bytes: 5_027_782_656, + note: "dense sanity check; external reference 43.7 t/s tg, 3167 t/s pp512" + }, + %{ + label: "30b", + repo: "Qwen/Qwen3-30B-A3B-GGUF", + files: ["Qwen3-30B-A3B-Q4_K_M.gguf"], + bytes: 18_565_509_120, + note: "MoE; external reference 89.3 t/s tg, 2541 t/s pp512" + }, + %{ + label: "120b", + repo: "ggml-org/gpt-oss-120b-GGUF", + files: ["gpt-oss-120b-MXFP4.gguf"], + bytes: 63_390_146_560, + note: "big but fits one node — the controlled A/B for RPC overhead" + }, + %{ + label: "235b", + repo: "unsloth/Qwen3-235B-A22B-GGUF", + files: [ + "Q4_K_M/Qwen3-235B-A22B-Q4_K_M-00001-of-00003.gguf", + "Q4_K_M/Qwen3-235B-A22B-Q4_K_M-00002-of-00003.gguf", + "Q4_K_M/Qwen3-235B-A22B-Q4_K_M-00003-of-00003.gguf" + ], + bytes: 142_100_000_000, + note: "142.1 GB across 3 shards — does not fit 121 GiB; the two-node headline" + }, + # Qwen3.6, the current generation. Two shapes (dense 27B, MoE 35B-A3B) and + # each in a plain and an MTP build. The MTP repos are the same weights plus + # the Multi-Token Prediction head, which is why they are a few hundred MB + # larger — llama.cpp reads those layers only when the model is loaded with + # `load_mtp: true`, so the plain and MTP files are a clean A/B for what + # speculative decoding buys on this hardware. + %{ + label: "q36-27b", + repo: "unsloth/Qwen3.6-27B-GGUF", + files: ["Qwen3.6-27B-Q4_K_M.gguf"], + bytes: 16_820_000_000, + note: "Qwen3.6 dense 27B" + }, + %{ + label: "q36-27b-mtp", + repo: "unsloth/Qwen3.6-27B-MTP-GGUF", + files: ["Qwen3.6-27B-Q4_K_M.gguf"], + bytes: 17_110_000_000, + note: "Qwen3.6 dense 27B with the MTP head — needs load_mtp: true" + }, + %{ + label: "q36-35b", + repo: "unsloth/Qwen3.6-35B-A3B-GGUF", + files: ["Qwen3.6-35B-A3B-UD-Q4_K_M.gguf"], + bytes: 22_130_000_000, + note: "Qwen3.6 MoE 35B-A3B" + }, + %{ + label: "q36-35b-mtp", + repo: "unsloth/Qwen3.6-35B-A3B-MTP-GGUF", + files: ["Qwen3.6-35B-A3B-UD-Q4_K_M.gguf"], + bytes: 22_660_000_000, + note: "Qwen3.6 MoE 35B-A3B with the MTP head — needs load_mtp: true" + } + ] + + def models, do: @models + + def gb(bytes), do: Float.round(bytes / 1_000_000_000, 1) + + def run(labels) do + selected = + case labels do + [] -> @models + _ -> Enum.filter(@models, &(&1.label in labels)) + end + + if selected == [] do + IO.puts(:stderr, "no model matches #{inspect(labels)}; try --list") + System.halt(2) + end + + total = selected |> Enum.map(& &1.bytes) |> Enum.sum() + IO.puts("fetching #{length(selected)} model(s), #{gb(total)} GB total\n") + + results = Enum.flat_map(selected, &fetch/1) + + IO.puts("") + + Enum.each(results, fn + {:ok, path} -> IO.puts(" ok #{path}") + {:error, what, reason} -> IO.puts(" FAIL #{what}: #{reason}") + end) + + if Enum.any?(results, &match?({:error, _, _}, &1)), do: System.halt(1) + end + + defp fetch(model) do + Enum.map(model.files, fn file -> + IO.puts("#{model.label} #{model.repo}/#{file}") + started = System.monotonic_time(:millisecond) + + # verify_checksum defaults to true and fails closed. Left at the default + # deliberately: a 142 GB download that silently truncates is exactly the + # failure this check exists for. + case LlamaCppEx.Hub.download(model.repo, file) do + {:ok, path} -> + elapsed = (System.monotonic_time(:millisecond) - started) / 1000 + size = File.stat!(path).size + + IO.puts( + " #{gb(size)} GB in #{Float.round(elapsed, 1)}s" <> + if(elapsed > 1, + do: " (#{Float.round(size / elapsed / 1_000_000, 1)} MB/s)", + else: " (cached)" + ) + ) + + {:ok, path} + + {:error, reason} -> + IO.puts(" FAILED: #{reason}") + {:error, "#{model.repo}/#{file}", reason} + end + end) + end + + def list do + IO.puts("cache dir: #{System.get_env("LLAMA_CACHE_DIR") || "~/.cache/llama_cpp_ex/models"}\n") + + Enum.each(@models, fn m -> + IO.puts( + "#{String.pad_trailing(m.label, 6)} #{String.pad_leading("#{gb(m.bytes)} GB", 9)} #{m.repo}" + ) + + IO.puts(" #{m.note}") + Enum.each(m.files, &IO.puts(" - #{&1}")) + IO.puts("") + end) + end +end + +case System.argv() do + ["--list"] -> FetchModels.list() + labels -> FetchModels.run(labels) +end diff --git a/scripts/spark/lib.sh b/scripts/spark/lib.sh new file mode 100755 index 0000000..69e1e0a --- /dev/null +++ b/scripts/spark/lib.sh @@ -0,0 +1,241 @@ +#!/usr/bin/env bash +# Shared configuration and helpers for scripts/spark/*. Sourced, never executed. +# +# Every constant below is a measured fact about the two DGX Sparks rather than a +# guess, and it lives here so a hardware change is a one-line edit instead of a +# grep. The measurements behind them are in +# .claude/plans/dgx-spark-2node/research/ and docs/dgx-spark.md. + +# --- Nodes ------------------------------------------------------------------- + +# ssh aliases, configured on the control node. The scripts never assume more +# than "ssh works and lands in the same $HOME on both". +SPARK_NODES=(spark-1 spark-2) + +# Fabric addresses. Two independent point-to-point ConnectX-7 links, MTU 9000, +# RoCE v2. They are separate subnets, not a bond: ggml-rpc opens one queue pair +# per socket (ggml/src/ggml-rpc/transport.cpp:292) and picks the HCA by matching +# a GID against the socket's local address, so one peer connection can only ever +# use one link. Link 0 is the one everything uses; link 1 is listed so the fact +# sheet can prove both trained. +spark_fabric_ip() { + case "$1:${2:-0}" in + spark-1:0) printf '10.100.64.1' ;; + spark-1:1) printf '10.100.65.1' ;; + spark-2:0) printf '10.100.64.2' ;; + spark-2:1) printf '10.100.65.2' ;; + *) return 1 ;; + esac +} + +# The ssh target for a node. Normally the alias, which resolves through mDNS +# (`spark-f11e.local`). +# +# mDNS is the least reliable thing in this setup: it went away for good on a box +# that was otherwise perfectly healthy — 10-day uptime, 117 GiB free, answering +# pings from its neighbour — after a heavy run, and no amount of cache flushing +# brought it back. The override exists so a name-resolution problem is a +# one-variable fix instead of a dead afternoon: +# +# export SPARK_HOST_SPARK_1=192.168.0.164 +# +# The value is passed to ssh and rsync in place of the alias, so it can be an IP, +# another alias, or a user@host. +spark_host() { + case "$1" in + spark-1) printf '%s' "${SPARK_HOST_SPARK_1:-spark-1}" ;; + spark-2) printf '%s' "${SPARK_HOST_SPARK_2:-spark-2}" ;; + *) printf '%s' "$1" ;; + esac +} + +# The node that owns a working asdf install and acts as the toolchain donor. +SPARK_TOOLCHAIN_DONOR="${SPARK_TOOLCHAIN_DONOR:-spark-1}" + +# --- CPU topology ------------------------------------------------------------ + +# The clusters are INTERLEAVED. X925 (performance) is 5-9,15-19 and A725 +# (efficiency) is 0-4,10-14, so the obvious `taskset -c 0-9` pins a job entirely +# to little cores. Measured: 1-byte RDMA ping-pong p50 21.5 us on cpu19 versus +# 29.1 us on cpu0. +SPARK_BIG_CORES="5-9,15-19" +SPARK_LITTLE_CORES="0-4,10-14" +SPARK_BIG_CORE_COUNT=10 + +# --- Build flags ------------------------------------------------------------- + +# GB10 is sm_121a. GCC 13.3 predates Cortex-X925/A725 and rejects +# -mcpu=cortex-x925 outright, so ggml's -mcpu=native probe degrades to base +# ARMv8-A *silently* (soft CMake warning, exit 0) and libggml-cpu.a comes out +# with 0 sdot, 0 smmla and no SVE — i.e. no quantized matmul kernels at all. +# Naming the architecture is the only way to get them, it requires +# GGML_NATIVE=OFF, and GGML_NATIVE=OFF in turn drags CUDA from one arch into a +# 7-arch fat binary unless the CUDA arch is pinned too. The Makefile enforces +# that pairing with an $(error); these are the values it wants. +SPARK_CPU_ARM_ARCH="${SPARK_CPU_ARM_ARCH:-armv9.2-a+dotprod+i8mm+fp16+bf16+sve2}" +SPARK_CUDA_ARCH="${SPARK_CUDA_ARCH:-121a-real}" +SPARK_LLAMA_BACKEND="${SPARK_LLAMA_BACKEND:-cuda}" + +# --- Remote layout ----------------------------------------------------------- +# All relative to the remote $HOME so they survive tilde expansion in ssh. + +SPARK_REMOTE_DIR="${SPARK_REMOTE_DIR:-src/llama_cpp_ex}" +SPARK_MODELS_DIR="${SPARK_MODELS_DIR:-models}" +SPARK_RPC_CACHE_DIR="${SPARK_RPC_CACHE_DIR:-.cache/llama.cpp/rpc}" + +# --- What the working tree sync carries -------------------------------------- +# +# Each exclusion has a reason, and the last one is the only non-obvious entry: +# +# .git history, not sources +# _build the control node's tree is aarch64-apple-darwin; +# sharing it poisons the remote build +# deps fetched remotely — hex.pm is reachable from both +# doc, .elixir_ls, .expert editor and docs output +# priv/llama_cpp_ex_nif.so the mac's NIF; the remote builds its own +# priv/plts dialyzer PLTs are OTP+arch specific +# tmp, erl_crash.dump scratch +# vendor/llama.cpp/.git 36 MB of history the build does not need. +# Makefile:29 falls back to LLAMA_COMMIT when +# vendor/llama.cpp/.git is absent, so LLAMA_SHA +# still resolves — sync.sh verifies that on every +# run rather than trusting it. +# models .gitignore treats ./models as the local GGUF +# directory and the nodes keep their own at +# ~/models. Without this, a developer with models +# checked out locally rsyncs tens or hundreds of GB +# to both Sparks on every sync — over the LAN, not +# the fabric — and --delete makes the interaction +# with the nodes' own ~/models worth not finding out +# about. +SPARK_SYNC_EXCLUDES=( + .git + _build + deps + doc + models + .elixir_ls + .expert + priv/llama_cpp_ex_nif.so + priv/.llama_cpp_ex_nif.built + priv/plts + tmp + erl_crash.dump + vendor/llama.cpp/.git + .sync-stamp +) + +# A content digest over exactly the synced file set, computed identically on the +# control node and on each Spark. Both nodes must end byte-identical: the RPC +# HELLO handshake compares only major/minor and ignores patch +# (ggml/src/ggml-rpc/ggml-rpc.h:9-15), so two builds from drifted trees connect +# happily and then misinterpret each other's op codes. This check is +# load-bearing, not hygiene. +spark_tree_digest_cmd() { + local prunes='' excl + for excl in "${SPARK_SYNC_EXCLUDES[@]}"; do + prunes+=" -path ./${excl} -o" + done + cat </dev/null 2>&1 && echo sha256sum || echo 'shasum -a 256') +find . \\( ${prunes% -o} \\) -prune -o -type f -print0 \\ + | LC_ALL=C sort -z \\ + | xargs -0 \$H \\ + | \$H \\ + | cut -d' ' -f1 +DIGEST +} + +# --- Toolchain --------------------------------------------------------------- + +# asdf 0.19's shims are `exec asdf exec `, and the asdf binary is in +# /usr/bin on spark-1 and absent on spark-2 — no sudo, no apt, so it stays +# absent. Putting the real install bin directories on PATH drops the dependency +# on the CLI entirely and behaves identically on both nodes. +SPARK_ERLANG_VERSION="${SPARK_ERLANG_VERSION:-29.0.2}" +SPARK_ELIXIR_VERSION="${SPARK_ELIXIR_VERSION:-1.20.2-otp-29}" + +SPARK_CUDA_HOME="${SPARK_CUDA_HOME:-/usr/local/cuda}" + +# --- Models ------------------------------------------------------------------ +# LLAMA_CACHE_DIR points Hub.download at ~/models, and its layout is +# /// (hub.ex:415-427) — the revision +# segment is part of the cache key, so "main" is in the path. remote.sh exports +# the two env vars the test suite and the benches read, but only for files that +# are actually present. +# +# scripts/spark/fetch_models.exs is the fetcher and the single source of truth +# for repo/filename pairs; these two are the ones the default workflow wants. +SPARK_SMOKE_GEN_MODEL="${SPARK_SMOKE_GEN_MODEL:-Qwen/Qwen3-8B-GGUF/main/Qwen3-8B-Q4_K_M.gguf}" +SPARK_BENCH_MODEL="${SPARK_BENCH_MODEL:-Qwen/Qwen3-8B-GGUF/main/Qwen3-8B-Q4_K_M.gguf}" + +if [ -t 2 ]; then + _spark_dim=$'\033[2m'; _spark_red=$'\033[31m'; _spark_yel=$'\033[33m'; _spark_rst=$'\033[0m' +else + _spark_dim=''; _spark_red=''; _spark_yel=''; _spark_rst='' +fi + +spark_log() { printf '%s==>%s %s\n' "$_spark_dim" "$_spark_rst" "$*" >&2; } +spark_warn() { printf '%swarning:%s %s\n' "$_spark_yel" "$_spark_rst" "$*" >&2; } +spark_die() { printf '%serror:%s %s\n' "$_spark_red" "$_spark_rst" "$*" >&2; exit 1; } + +# --- Helpers ----------------------------------------------------------------- + +# Repository root, derived from this file's location rather than $PWD so the +# scripts work from anywhere. +spark_repo_root() { + cd "$(dirname "${BASH_SOURCE[0]}")/../.." && pwd +} + +spark_shquote() { + printf "'%s'" "$(printf '%s' "$1" | sed "s/'/'\\\\''/g")" +} + +spark_is_node() { + local candidate=$1 node + for node in "${SPARK_NODES[@]}"; do + [ "$node" = "$candidate" ] && return 0 + done + return 1 +} + +# Resolve a node argument to a list of nodes. "all" or "both" means every node. +spark_resolve_nodes() { + case "$1" in + all | both) printf '%s\n' "${SPARK_NODES[@]}" ;; + *) + spark_is_node "$1" || spark_die "unknown node '$1' (known: ${SPARK_NODES[*]}, all)" + printf '%s\n' "$1" + ;; + esac +} + +# The environment preamble every remote command runs under. Emitted as shell +# source text so ssh, rsync-driven bootstrap and the worker scripts all agree. +# +# Deliberately a NON-login shell contract: this is what `ssh host cmd`, a +# systemd unit and any future CI runner get, and it is what the build spike +# validated. DGX OS puts nvcc on the login PATH only, via +# /etc/profile.d/nv_paths.sh, which is exactly the blind spot the Makefile's +# CUDA_HOME discovery exists to cover — so we set CUDA_HOME rather than relying +# on the profile script. +# The build flags are part of the contract too: LLAMA_BACKEND=cuda also flips +# mix.exs's make_force_build, which is what we want here — there is no +# aarch64-linux precompiled artifact, so the Spark always source-builds, and +# make itself stays incremental behind its stamp file. +# +# The two model exports are `|| true`-guarded because callers run under +# `set -e` and a missing model file is normal before Phase 3 provisioning. +spark_remote_env_preamble() { + local models="\$HOME/${SPARK_MODELS_DIR}" + cat < remote working directory, relative to $HOME + (default: the synced repo; "-" for $HOME) + --env K=V extra environment variable, repeatable + --big-cores wrap the command in `taskset -c 5-9,15-19` (Cortex-X925) + --tty allocate a remote tty + --print print the remote script instead of running it +USAGE +} + +login=0 +tty=0 +print_only=0 +big_cores=0 +forward_agent=0 +dir="$SPARK_REMOTE_DIR" +extra_env=() + +while [ $# -gt 0 ]; do + case "$1" in + --login) login=1; shift ;; + --tty) tty=1; shift ;; + --print) print_only=1; shift ;; + --big-cores) big_cores=1; shift ;; + --forward-agent) forward_agent=1; shift ;; + --dir) dir="${2:?--dir needs a path}"; shift 2 ;; + --env) extra_env+=("${2:?--env needs K=V}"); shift 2 ;; + -h | --help) usage; exit 0 ;; + --) shift; break ;; + -*) spark_die "unknown option '$1' (try --help)" ;; + *) break ;; + esac +done + +[ $# -ge 2 ] || { usage >&2; exit 2; } + +node=$1; shift +spark_is_node "$node" || spark_die "unknown node '$node' (known: ${SPARK_NODES[*]})" + +prefix=() +[ "$big_cores" -eq 1 ] && prefix=(taskset -c "$SPARK_BIG_CORES") + +# Assemble the remote script. Every word is quoted here rather than left to +# ssh's own re-splitting, which joins its arguments with spaces and would mangle +# any argument containing one. +build_script() { + printf 'set -euo pipefail\n' + spark_remote_env_preamble + + local kv + for kv in ${extra_env[@]+"${extra_env[@]}"}; do + case "$kv" in + *=*) printf 'export %s=%s\n' "${kv%%=*}" "$(spark_shquote "${kv#*=}")" ;; + *) spark_die "--env expects K=V, got '$kv'" ;; + esac + done + + # Forward MIX_ENV from the caller: `MIX_ENV=bench remote.sh ...` should mean + # what it says. + [ -n "${MIX_ENV:-}" ] && printf 'export MIX_ENV=%s\n' "$(spark_shquote "$MIX_ENV")" + + if [ "$dir" = "-" ]; then + printf 'cd "$HOME"\n' + else + printf 'cd "$HOME/%s" 2>/dev/null || { echo "remote: ~/%s is missing; run scripts/spark/sync.sh first" >&2; exit 1; }\n' \ + "$dir" "$dir" + fi + + local arg + printf 'exec' + for arg in ${prefix[@]+"${prefix[@]}"} "$@"; do printf ' %s' "$(spark_shquote "$arg")"; done + printf '\n' +} + +script=$(build_script "$@") + +if [ "$print_only" -eq 1 ]; then + printf '%s\n' "$script" + exit 0 +fi + +ssh_opts=() +[ "$tty" -eq 1 ] && ssh_opts+=(-t) + +# The nodes hold no keys for each other and the plan forbids adding any, so a +# command that needs to reach the *other* Spark (reading the RPC worker's RSS, +# say) rides a forwarded agent. ControlPath=none is required: the multiplexed +# master was opened without -A and reusing it would silently drop the +# forwarding. +if [ "$forward_agent" -eq 1 ]; then + ssh_opts+=(-A -o ControlPath=none) +fi +[ "$login" -eq 1 ] && shell="bash -lc" || shell="bash -c" + +exec ssh ${ssh_opts[@]+"${ssh_opts[@]}"} "$(spark_host "$node")" "$shell $(spark_shquote "$script")" diff --git a/scripts/spark/rpc-worker.sh b/scripts/spark/rpc-worker.sh new file mode 100755 index 0000000..3c337dd --- /dev/null +++ b/scripts/spark/rpc-worker.sh @@ -0,0 +1,256 @@ +#!/usr/bin/env bash +# Manage the ggml RPC worker on a Spark. +# +# scripts/spark/rpc-worker.sh start spark-2 +# scripts/spark/rpc-worker.sh start spark-2 --upstream --debug +# scripts/spark/rpc-worker.sh status spark-2 +# scripts/spark/rpc-worker.sh logs spark-2 +# scripts/spark/rpc-worker.sh rss spark-2 +# scripts/spark/rpc-worker.sh stop spark-2 +# +# Supervision is `systemd --user` via systemd-run. `loginctl enable-linger` +# succeeds without a password on these boxes (verified), so a user manager +# survives logout and journald captures the worker's output — no nohup, no +# stray log files, and `systemctl --user restart` is the restart story. +# +# ## Restart the worker between runs +# +# Two independent reporters describe upstream's RPC worker growing RSS during +# inference and never releasing it; killing the client does not free it. So the +# `rss` subcommand exists, `start` launches a sampler alongside the worker, and +# the sampler stops the worker rather than letting the node OOM. +# +# ## Bind to the fabric address +# +# Upstream defaults to 127.0.0.1, which is wrong here twice: a remote client +# cannot reach it, and RDMA can never engage on it because the transport picks +# an HCA by matching a GID against the socket's *local* address. This script +# always binds a fabric address and has no option to bind loopback. +# +# Nothing about this port is authenticated. It accepts commands to allocate +# memory and execute compute graphs. The fabric is point-to-point; keep it there. + +set -euo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +UNIT=llama-rpc-worker +RSS_UNIT=llama-rpc-rss + +usage() { sed -n '2,31p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; } + +[ $# -ge 2 ] || { usage >&2; exit 2; } + +command=$1; shift +node=$1; shift +spark_is_node "$node" || spark_die "unknown node '$node' (known: ${SPARK_NODES[*]})" + +port=50052 +link=0 +threads=$SPARK_BIG_CORE_COUNT +upstream=0 +debug=0 +cache=1 +# Fraction of total memory at which the sampler stops the worker. The worker +# legitimately holds tens of GB of weights, so an absolute default would be +# either useless or a footgun; this catches the leak, not the workload. +rss_fraction=0.92 +# Force the TCP path. There is no runtime switch for the transport: selection is +# silent auto-negotiation, and GGML_RDMA_DEV pointing at a device that does not +# exist is the only lever short of a -DGGML_RPC_RDMA=OFF build. Both ends have +# to agree, so pass --tcp here and GGML_RDMA_DEV to the client too. +tcp=0 +# `logs` takes a bare line count. +lines=100 + +while [ $# -gt 0 ]; do + case "$1" in + --port) port="${2:?}"; shift 2 ;; + --link) link="${2:?}"; shift 2 ;; + --threads) threads="${2:?}"; shift 2 ;; + --rss-fraction) rss_fraction="${2:?}"; shift 2 ;; + --upstream) upstream=1; shift ;; + --debug) debug=1; shift ;; + --no-cache) cache=0; shift ;; + --tcp) tcp=1; shift ;; + -h | --help) usage; exit 0 ;; + [0-9]*) lines=$1; shift ;; + *) spark_die "unknown option '$1' (try --help)" ;; + esac +done + +ip=$(spark_fabric_ip "$node" "$link") || spark_die "no link $link on $node" +endpoint="$ip:$port" + +remote() { ssh "$(spark_host "$node")" "bash -c $(spark_shquote "$1")"; } + +start() { + # Linger keeps the user manager alive after the ssh session ends. It is + # idempotent and, on these boxes, needs no password. + remote "loginctl enable-linger >/dev/null 2>&1 || true" + + if remote "systemctl --user is-active --quiet $UNIT"; then + spark_die "$node already has a worker running. Stop it first — and do stop it + between benchmark runs, because upstream's worker leaks." + fi + + # The sampler outlives a worker that dies on its own (it waits for a MainPID), + # so a previous failed start leaves it running and systemd-run then refuses + # the name. Clear both before claiming them. + remote "systemctl --user stop $RSS_UNIT >/dev/null 2>&1 || true; \ + systemctl --user reset-failed $UNIT $RSS_UNIT >/dev/null 2>&1 || true" + + local env_lines cache_dir inner + cache_dir="\$HOME/$SPARK_RPC_CACHE_DIR" + env_lines=$(spark_remote_env_preamble) + + # The `|| true` inside both substitutions below is load-bearing, not noise. An + # assignment whose value contains a command substitution takes that + # substitution's exit status, and `set -euo pipefail` above then kills the + # shell -- so with `--no-cache` (cache=0) the `[ ... -eq 1 ]` test failed, the + # assignment returned 1, and `start` exited silently *after* having already + # stopped the running worker. Do not remove them. + if [ "$upstream" -eq 1 ]; then + # Upstream's standalone binary: the A/B reference. When a two-node run + # misbehaves the first question is whether the fault is ours or upstream's, + # and this is how that gets answered. + inner="BIN=\$(ls -d _build/*/lib/llama_cpp_ex/obj/rpc_server_build/bin/ggml-rpc-server 2>/dev/null | head -1) +[ -n \"\$BIN\" ] || { echo 'no ggml-rpc-server; run: LLAMA_RPC=1 make rpc-server' >&2; exit 1; } +exec taskset -c $SPARK_BIG_CORES \"\$BIN\" -H $ip -p $port -t $threads$([ "$cache" -eq 1 ] && echo ' -c' || true)" + else + inner="export SPARK_RPC_ENDPOINT=$endpoint +export SPARK_RPC_THREADS=$threads +$([ "$cache" -eq 1 ] && echo "export SPARK_RPC_CACHE=$cache_dir" || true) +mkdir -p $cache_dir +exec taskset -c $SPARK_BIG_CORES mix run scripts/spark/rpc_worker.exs" + fi + + local script + script=$( + printf 'set -euo pipefail\n' + printf '%s\n' "$env_lines" + # Not optional for a worker, and it has to be in the unit's own environment: + # LLAMA_BACKEND is set, so mix force-builds, and without this the worker + # would happily rebuild the *non*-RPC tree and then fail at start_link with + # :rpc_unsupported. It is also part of the build-directory key, so the two + # trees coexist and switching back is a cache hit. + printf 'export LLAMA_RPC=1\n' + [ "$debug" -eq 1 ] && printf 'export GGML_RPC_DEBUG=1\n' + [ "$tcp" -eq 1 ] && printf 'export GGML_RDMA_DEV=no-such-hca\n' + printf 'cd "$HOME/%s"\n' "$SPARK_REMOTE_DIR" + printf '%s\n' "$inner" + ) + + # Build before the unit exists, not inside it. `mix run` compiles on demand, + # and the readiness loop below cannot tell "still compiling" from "never going + # to listen": it only knows the unit is active and the port is silent. After a + # submodule bump that compile is a 2-3 minute llama.cpp rebuild, so the 120s + # budget expired, this reported a failure, and the worker then came up fine on + # its own -- an error message for a working worker. Building here also makes a + # compile error arrive as a compile error instead of as a readiness timeout. + # The upstream branch runs a prebuilt binary and has nothing to compile. + if [ "$upstream" -eq 0 ]; then + local prep + prep=$( + printf 'set -euo pipefail\n' + printf '%s\n' "$env_lines" + printf 'export LLAMA_RPC=1\n' + printf 'cd "$HOME/%s"\n' "$SPARK_REMOTE_DIR" + printf 'mix compile\n' + ) + spark_log "$node: building the NIF first (minutes, after a llama.cpp bump)" + remote "/bin/bash -c $(spark_shquote "$prep")" >/dev/null || + spark_die "$node: the worker's build failed; fix that before starting it" + fi + + spark_log "$node: starting worker on $endpoint (${threads} threads on cores $SPARK_BIG_CORES$([ "$debug" -eq 1 ] && echo ", GGML_RPC_DEBUG=1")$([ "$tcp" -eq 1 ] && echo ", TCP forced"))" + + remote "systemd-run --user --unit=$UNIT --collect --description='llama.cpp RPC worker' \ + /bin/bash -c $(spark_shquote "$script") >/dev/null" + + start_rss_sampler + + # systemd-run returns as soon as the unit is queued, which says nothing about + # whether anything is listening. Wait for the port. + local waited=0 + while [ "$waited" -lt 120 ]; do + if remote "exec 3<>/dev/tcp/$ip/$port" 2>/dev/null; then + spark_log "$node: listening on $endpoint" + status + return 0 + fi + if ! remote "systemctl --user is-active --quiet $UNIT"; then + spark_warn "$node: the worker unit exited. Last output:" + logs 40 + spark_die "worker failed to start" + fi + sleep 1 + waited=$((waited + 1)) + done + + logs 40 + spark_die "$node: nothing listening on $endpoint after 120s" +} + +# Samples the worker's RSS into journald and stops it before the node OOMs. +# A watchdog that only logs is not a watchdog: an unbounded leak on a box with +# no swap takes the whole machine, and losing the worker is the cheaper failure. +start_rss_sampler() { + local sampler + sampler=$(cat </dev/null) + [ -n "\$pid" ] && [ "\$pid" != 0 ] || { sleep 5; continue; } + rss_kb=\$(awk '/^VmRSS:/{print \$2}' /proc/\$pid/status 2>/dev/null || echo 0) + [ -n "\$rss_kb" ] && [ "\$rss_kb" -gt 0 ] || { sleep 5; continue; } + echo "rss \$((rss_kb / 1024)) MiB" + if [ "\$rss_kb" -gt "\$limit_kb" ]; then + echo "rss \$((rss_kb / 1024)) MiB exceeds \$((limit_kb / 1024)) MiB - stopping the worker" + systemctl --user stop $UNIT + exit 1 + fi + sleep 5 +done +SAMPLER + ) + + remote "systemd-run --user --unit=$RSS_UNIT --collect --description='llama.cpp RPC worker RSS sampler' \ + /bin/bash -c $(spark_shquote "$sampler") >/dev/null" +} + +stop() { + remote "systemctl --user stop $RSS_UNIT $UNIT 2>/dev/null || true; \ + systemctl --user reset-failed $RSS_UNIT $UNIT 2>/dev/null || true" + spark_log "$node: worker stopped (the port is released only because the VM exits; + the native accept loop has no shutdown hook)" +} + +status() { + remote "systemctl --user is-active $UNIT 2>/dev/null || echo inactive" | sed "s/^/ unit /" + rss +} + +rss() { + remote "pid=\$(systemctl --user show $UNIT --property=MainPID --value 2>/dev/null); \ + if [ -n \"\$pid\" ] && [ \"\$pid\" != 0 ]; then \ + awk '/^VmRSS:/{printf \" rss %d MiB\\n\", \$2/1024}' /proc/\$pid/status; \ + else echo ' rss n/a'; fi" +} + +logs() { + remote "journalctl --user -u $UNIT --no-pager -n ${1:-100}" +} + +case "$command" in + start) start ;; + stop) stop ;; + restart) stop; start ;; + status) status ;; + rss) rss ;; + logs) logs "$lines" ;; + *) usage >&2; exit 2 ;; +esac diff --git a/scripts/spark/rpc_check.exs b/scripts/spark/rpc_check.exs new file mode 100644 index 0000000..69ec8be --- /dev/null +++ b/scripts/spark/rpc_check.exs @@ -0,0 +1,199 @@ +# Two-node bring-up verification. Proves each link in the chain before any +# benchmark is allowed to mean anything. +# +# scripts/spark/rpc-worker.sh start spark-2 --debug +# scripts/spark/remote.sh --env LLAMA_RPC=1 --env MIX_ENV=test spark-1 \ +# mix run scripts/spark/rpc_check.exs 10.100.64.2:50052 +# scripts/spark/rpc-worker.sh logs spark-2 +# +# Set GGML_RPC_DEBUG=1 for the client side of the transport negotiation. The +# only way to know whether you got RDMA or TCP is that log: selection is silent +# auto-negotiation with no env var, no endpoint scheme and no return value. + +endpoint = + case System.argv() do + [e | _] -> e + [] -> raise "usage: mix run scripts/spark/rpc_check.exs " + end + +model_path = System.get_env("LLAMA_MODEL_PATH") || raise "LLAMA_MODEL_PATH is required" + +defmodule Check do + def start, do: Agent.start_link(fn -> [] end, name: __MODULE__) + + def step(label, fun) do + IO.write(" " <> String.pad_trailing(label, 46)) + + # Statement form, not `IO.puts(...) || :ok`. `IO.puts/1` returns `:ok`, which + # is truthy, so `||` short-circuited and BOTH branches evaluated to `:ok` -- + # `outcome` was always `:ok`, `results` never held an `:error`, and the + # `finish/0` gate below was dead code. This script printed + # "all N checks passed" and exited 0 even when the model failed to load + # across two nodes, which is the one thing it exists to prevent. + outcome = + case fun.() do + {:ok, detail} -> + IO.puts("PASS #{detail}") + :ok + + {:error, detail} -> + IO.puts("FAIL #{detail}") + :error + end + + Agent.update(__MODULE__, &[outcome | &1]) + outcome + end + + def finish do + results = Agent.get(__MODULE__, & &1) + + IO.puts("") + + if :error in results do + IO.puts("FAILED — do not benchmark until this is clean.") + System.halt(1) + else + IO.puts("all #{length(results)} checks passed") + end + end +end + +Check.start() + +# --- 1. Registration --------------------------------------------------------- +# +# An unreachable endpoint and a HELLO version mismatch both collapse to a null +# registration upstream, and a null registration is silently ignored by +# ggml_backend_register. Checking the device count is the only thing standing +# between us and a model that quietly loads entirely onto the local GPU while we +# benchmark "two nodes". + +before = length(LlamaCppEx.devices()) + +Check.step("register #{endpoint}", fn -> + case LlamaCppEx.RPC.add_server(endpoint) do + {:ok, n} when n >= 1 -> {:ok, "#{n} device(s) added"} + {:ok, 0} -> {:error, "already registered in this VM"} + {:error, reason} -> {:error, inspect(reason)} + end +end) + +devices = LlamaCppEx.devices() +remote = Enum.filter(devices, &(&1.backend == "RPC")) +local = Enum.find(devices, &(&1.type in [:gpu, :igpu] and &1.backend != "RPC")) + +Check.step("device registry grew", fn -> + if length(devices) > before, + do: {:ok, "#{before} -> #{length(devices)} devices"}, + else: {:error, "still #{before} — ggml_backend_register no-opped"} +end) + +gib = fn bytes -> Float.round(bytes / (1024 * 1024 * 1024), 1) end + +Check.step("remote device reports real memory", fn -> + case remote do + [d | _] when d.memory_total > 0 -> + {:ok, "#{d.name} #{gib.(d.memory_total)} GiB total, #{gib.(d.memory_free)} GiB free"} + + [d | _] -> {:error, "#{d.name} reports #{d.memory_total} bytes"} + [] -> {:error, ~s(no device with backend "RPC")} + end +end) + +Check.step("a local accelerator is still visible", fn -> + if local, do: {:ok, "#{local.name} (#{local.type})"}, else: {:error, "none"} +end) + +IO.puts("\ndevice registry order — what LlamaCppEx.devices/0 reports:") + +Enum.each(devices, fn d -> + IO.puts( + " [#{d.index}] #{String.pad_trailing(d.name, 8)}#{String.pad_trailing(d.backend, 7)}" <> + "#{String.pad_trailing(to_string(d.type), 7)}gpu_index=#{inspect(d.gpu_index)} #{d.description}" + ) +end) + +IO.puts(""" + + Placement order is NOT this order. llama.cpp rebuilds the list at load time + with RPC devices FIRST (src/llama.cpp:263-273), so tensor_split indexes a + different list. Naming :devices below removes the ambiguity entirely. +""") + +if remote == [] or local == nil do + Check.finish() + System.halt(0) +end + +[remote_dev | _] = remote + +# --- 2. A model loads and generates across the pair -------------------------- + +IO.puts("loading #{Path.basename(model_path)} across [#{local.name}, #{remote_dev.name}] 50/50\n") + +load_started = System.monotonic_time(:millisecond) + +load = + LlamaCppEx.Model.load(model_path, + n_gpu_layers: 99, + devices: [local.name, remote_dev.name], + split_mode: :layer, + tensor_split: [0.5, 0.5] + ) + +load_ms = System.monotonic_time(:millisecond) - load_started + +case load do + {:ok, model} -> + Check.step("model loads across two nodes", fn -> {:ok, "#{load_ms} ms"} end) + + Check.step("generates across the pair", fn -> + case LlamaCppEx.generate(model, "The capital of France is", max_tokens: 16, temp: 0.0) do + {:ok, text} when byte_size(text) > 0 -> {:ok, inspect(String.slice(text, 0, 48))} + {:ok, ""} -> {:error, "empty output"} + other -> {:error, inspect(other)} + end + end) + + # --- 3. The decode fast path --------------------------------------------- + # + # Decode across an RPC device is only affordable because a repeated graph + # collapses to a 4-byte GRAPH_RECOMPUTE. A cache miss re-serialises every + # tensor descriptor on every token, which is the difference between free and + # ruinous. The cache keys on the graph uid being unchanged since the last + # graph on that device, so anything that varies the batch shape per token + # reverts to the slow path. + # + # Two runs of the same shape: if the second is not materially faster per + # token than the first, the cache is not being hit. + warm = fn n -> + t0 = System.monotonic_time(:millisecond) + {:ok, _} = LlamaCppEx.generate(model, "Count from one:", max_tokens: n, temp: 0.0) + System.monotonic_time(:millisecond) - t0 + end + + first = warm.(32) + second = warm.(32) + + Check.step("steady-state decode", fn -> + tps = Float.round(32_000 / second, 1) + {:ok, "#{tps} tok/s (first pass #{first} ms, second #{second} ms)"} + end) + + IO.puts(""" + + Now check the worker journal for the transport and the graph cache: + + scripts/spark/rpc-worker.sh logs spark-2 200 | grep -E 'RDMA|GRAPH' + + Expect 'RDMA activated', and GRAPH_RECOMPUTE dominating GRAPH_COMPUTE in + the steady state. 'RDMA activate failed, staying on TCP' means you are + measuring the ~19 us TCP path, not the 1.39 us RDMA one. + """) + + {:error, reason} -> + Check.step("model loads across two nodes", fn -> {:error, inspect(reason)} end) +end + +Check.finish() diff --git a/scripts/spark/rpc_worker.exs b/scripts/spark/rpc_worker.exs new file mode 100644 index 0000000..c35391f --- /dev/null +++ b/scripts/spark/rpc_worker.exs @@ -0,0 +1,54 @@ +# The NIF-hosted RPC worker. Driven by scripts/spark/rpc-worker.sh, which sets +# the environment; run it directly only for debugging. +# +# SPARK_RPC_ENDPOINT=10.100.64.2:50052 mix run scripts/spark/rpc_worker.exs +# +# Blocks forever on purpose. `LlamaCppEx.RPC.Server` links to this process, and +# upstream's accept loop never returns, so there is nothing to wait on and +# nothing to shut down — the VM is the unit of restart. + + +# start_link/1 links, and a GenServer whose init/1 returns {:stop, reason} +# exits with that reason — which kills this script before it can say anything +# useful. Trapping turns the exit back into the {:error, reason} that +# start_link is documented to return. +Process.flag(:trap_exit, true) + +endpoint = + System.get_env("SPARK_RPC_ENDPOINT") || + raise "SPARK_RPC_ENDPOINT is required, e.g. 10.100.64.2:50052" + +cache_dir = System.get_env("SPARK_RPC_CACHE") +n_threads = String.to_integer(System.get_env("SPARK_RPC_THREADS") || "10") + +devices = + case System.get_env("SPARK_RPC_DEVICES") do + nil -> [] + "" -> [] + list -> String.split(list, ",", trim: true) + end + +case LlamaCppEx.RPC.Server.start_link( + endpoint: endpoint, + cache_dir: cache_dir, + n_threads: n_threads, + devices: devices + ) do + {:ok, pid} -> + %{devices: served} = LlamaCppEx.RPC.Server.info(pid) + IO.puts("worker ready: #{endpoint} serving #{Enum.join(served, ", ")}") + Process.sleep(:infinity) + + {:error, {:rpc, :rpc_unsupported}} -> + IO.puts(:stderr, """ + This build has no RPC backend. Rebuild the worker with: + + LLAMA_RPC=1 mix compile + """) + + System.halt(1) + + {:error, reason} -> + IO.puts(:stderr, "worker failed to start: #{inspect(reason)}") + System.halt(1) +end diff --git a/scripts/spark/sync.sh b/scripts/spark/sync.sh new file mode 100755 index 0000000..01a0ac2 --- /dev/null +++ b/scripts/spark/sync.sh @@ -0,0 +1,109 @@ +#!/usr/bin/env bash +# Push the working tree to one or both DGX Sparks and prove they match. +# +# scripts/spark/sync.sh # both nodes +# scripts/spark/sync.sh spark-1 # one node +# scripts/spark/sync.sh --check # compare only, sync nothing +# scripts/spark/sync.sh --dry-run # rsync -n +# +# Sync is rsync, not `git pull`: the remote needs the *working* tree including +# uncommitted changes, and rsync is one hop with no auth dance. +# +# After every sync both nodes are verified byte-identical to the control node +# over the synced file set. That check exists because the ggml RPC HELLO +# handshake compares only major/minor and ignores patch, so mismatched builds +# connect and then silently disagree about op codes. + +set -euo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +check_only=0 +dry_run=0 +target=all + +while [ $# -gt 0 ]; do + case "$1" in + --check) check_only=1; shift ;; + --dry-run | -n) dry_run=1; shift ;; + -h | --help) sed -n '2,16p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + -*) spark_die "unknown option '$1' (try --help)" ;; + *) target=$1; shift ;; + esac +done + +root=$(spark_repo_root) +cd "$root" + +# Read into an array without `mapfile`: /bin/bash on macOS is still 3.2. +nodes=() +while IFS= read -r _n; do nodes+=("$_n"); done < <(spark_resolve_nodes "$target") + +digest_cmd=$(spark_tree_digest_cmd) +local_digest=$(bash -c "$digest_cmd") +head_sha=$(git rev-parse HEAD 2>/dev/null || echo unknown) +dirty=$(git status --short 2>/dev/null || true) + +if [ "$check_only" -eq 0 ]; then + rsync_opts=(-az --delete --human-readable) + [ "$dry_run" -eq 1 ] && rsync_opts+=(-n --itemize-changes) + for excl in "${SPARK_SYNC_EXCLUDES[@]}"; do rsync_opts+=(--exclude "/$excl"); done + + for node in "${nodes[@]}"; do + spark_log "sync -> $node:~/$SPARK_REMOTE_DIR" + ssh "$(spark_host "$node")" "mkdir -p ~/$SPARK_REMOTE_DIR" + rsync "${rsync_opts[@]}" ./ "$(spark_host "$node"):$SPARK_REMOTE_DIR/" + done + + [ "$dry_run" -eq 1 ] && exit 0 + + # The stamp is written after the tree lands so a half-finished sync leaves the + # previous stamp in place rather than a lying new one. + stamp=$( + printf 'head=%s\n' "$head_sha" + printf 'digest=%s\n' "$local_digest" + printf 'synced=%s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" + printf 'from=%s\n' "$(hostname)" + printf 'dirty=%s\n' "$(printf '%s' "$dirty" | grep -c . || true)" + if [ -n "$dirty" ]; then printf -- '--- git status --short ---\n%s\n' "$dirty"; fi + ) + for node in "${nodes[@]}"; do + printf '%s\n' "$stamp" | ssh "$(spark_host "$node")" "cat > ~/$SPARK_REMOTE_DIR/.sync-stamp" + done +fi + +# --- Verification ------------------------------------------------------------ + +fail=0 + +for node in "${nodes[@]}"; do + remote_digest=$(ssh "$(spark_host "$node")" "cd ~/$SPARK_REMOTE_DIR 2>/dev/null && bash -c $(spark_shquote "$digest_cmd")" || echo MISSING) + if [ "$remote_digest" = "$local_digest" ]; then + spark_log "$node tree digest ${remote_digest:0:12} — matches control node" + else + spark_warn "$node tree digest $remote_digest != local $local_digest" + fail=1 + fi +done + +if [ "$fail" -ne 0 ]; then + spark_die "nodes are NOT byte-identical. Re-run without --check, or investigate before building: + a drifted tree still passes the RPC HELLO handshake and then corrupts silently." +fi + +# Makefile:29 resolves LLAMA_SHA from vendor/llama.cpp/.git when it is present +# and falls back to LLAMA_COMMIT when it is not. We exclude that .git, so this +# is the fallback path in production on every node — verified, not assumed. +local_sha=$(MIX_APP_PATH=/tmp/.spark-sha-probe make -n -p 2>/dev/null | sed -n 's/^LLAMA_SHA :*= *//p' | head -1) +for node in "${nodes[@]}"; do + remote_sha=$(ssh "$(spark_host "$node")" "cd ~/$SPARK_REMOTE_DIR && MIX_APP_PATH=/tmp/.spark-sha-probe make -n -p 2>/dev/null | sed -n 's/^LLAMA_SHA :*= *//p' | head -1" || true) + if [ -z "$remote_sha" ]; then + spark_die "$node: LLAMA_SHA did not resolve. The vendor/llama.cpp/.git exclusion broke Makefile:29." + elif [ "$remote_sha" != "$local_sha" ]; then + spark_die "$node: LLAMA_SHA is $remote_sha, control node says $local_sha." + fi + spark_log "$node LLAMA_SHA ${remote_sha:0:12} — resolves without vendor/llama.cpp/.git" +done + +spark_log "in sync at ${head_sha:0:12}${dirty:+ (+$(printf '%s' "$dirty" | grep -c .) local changes)}" diff --git a/scripts/spark/verify-build-flags.sh b/scripts/spark/verify-build-flags.sh new file mode 100755 index 0000000..3e99db4 --- /dev/null +++ b/scripts/spark/verify-build-flags.sh @@ -0,0 +1,122 @@ +#!/usr/bin/env bash +# Prove that a built tree actually got the architecture flags it was asked for. +# +# Run it ON a node, inside the synced repo, after a build: +# +# scripts/spark/remote.sh spark-1 scripts/spark/verify-build-flags.sh +# +# Three assertions, because each one catches a different silent failure: +# +# 1. ggml-cuda compiles for exactly ONE architecture. GGML_NATIVE=OFF is +# required to reach the ARM CPU flag and it drops ggml-cuda out of `native` +# into a seven-architecture fat binary. That is a ~6x build-time regression +# with no runtime benefit and nothing warns about it. +# Asserted from flags.make, NOT from CMakeCache.txt: CMAKE_CUDA_ARCHITECTURES +# is an ordinary variable and never appears in the cache. +# +# 2. ggml-cpu compiles at the named architecture rather than -mcpu=native. +# +# 3. The emitted archive actually contains the quantized matmul instructions. +# (2) can pass while (3) fails if the compiler accepts a flag and then +# declines to vectorize, and (3) is the thing anybody actually cares about. +# GCC 13.3 on GB10 fails (2) and (3) together, silently: it predates +# Cortex-X925, rejects -mcpu=cortex-x925, and degrades native to base +# ARMv8-A behind a soft CMake warning and exit 0. + +set -uo pipefail + +# shellcheck source=scripts/spark/lib.sh +. "$(dirname "${BASH_SOURCE[0]}")/lib.sh" + +build_dir="" +cpu_arch="$SPARK_CPU_ARM_ARCH" +cuda_arch="$SPARK_CUDA_ARCH" + +while [ $# -gt 0 ]; do + case "$1" in + --build-dir) build_dir="${2:?--build-dir needs a path}"; shift 2 ;; + --cpu-arch) cpu_arch="${2:?--cpu-arch needs a value}"; shift 2 ;; + --cuda-arch) cuda_arch="${2:?--cuda-arch needs a value}"; shift 2 ;; + -h | --help) sed -n '2,25p' "${BASH_SOURCE[0]}" | sed 's/^# \{0,1\}//'; exit 0 ;; + *) spark_die "unknown argument '$1' (try --help)" ;; + esac +done + +if [ -z "$build_dir" ]; then + # Newest first, so a fresh build wins over a stale one from another flag set. + build_dir=$(ls -dt _build/*/lib/llama_cpp_ex/obj/llama_build-* 2>/dev/null | head -1) +fi +[ -n "$build_dir" ] && [ -d "$build_dir" ] || + spark_die "no llama.cpp build tree found. Build first, or pass --build-dir." + +spark_log "verifying $build_dir" + +fails=0 +pass() { printf ' PASS %s\n' "$*"; } +fail() { printf ' FAIL %s\n' "$*"; fails=$((fails + 1)); } +skip() { printf ' SKIP %s\n' "$*"; } + +# --- 1. one CUDA architecture, and the right one ----------------------------- + +# 121a-real and 121a-virtual both compile for compute_121a; the suffix only +# selects which of cubin/PTX is embedded. +want_cc=${cuda_arch%-real} +want_cc=${want_cc%-virtual} + +cuda_flags=$(find "$build_dir" -path '*ggml-cuda.dir/flags.make' -print -quit 2>/dev/null) + +if [ -z "$cuda_flags" ]; then + skip "ggml-cuda: not a CUDA build" +else + archs=$(grep -oE 'arch=compute_[0-9a-z]+' "$cuda_flags" | sort -u | sed 's/arch=compute_//') + n=$(printf '%s\n' "$archs" | grep -c .) + if [ "$n" -ne 1 ]; then + fail "ggml-cuda compiles $n architectures ($(printf '%s' "$archs" | tr '\n' ' ')) — fat binary, LLAMA_CUDA_ARCH is not reaching cmake" + elif [ "$archs" != "$want_cc" ]; then + fail "ggml-cuda compiles compute_$archs, expected compute_$want_cc" + else + pass "ggml-cuda: one architecture, compute_$want_cc" + fi +fi + +# --- 2. the CPU architecture flag -------------------------------------------- + +cpu_flags=$(find "$build_dir" -path '*ggml-cpu.dir/flags.make' -print -quit 2>/dev/null) + +if [ -z "$cpu_flags" ]; then + fail "ggml-cpu: no flags.make under $build_dir" +else + if grep -q -- "-march=$cpu_arch" "$cpu_flags"; then + pass "ggml-cpu: -march=$cpu_arch" + else + got=$(grep -oE -- '-m(arch|cpu)=[^ ]+' "$cpu_flags" | sort -u | tr '\n' ' ') + fail "ggml-cpu: expected -march=$cpu_arch, got ${got:-nothing}" + fi +fi + +# --- 3. the instructions are actually in the archive ------------------------- + +archive=$(find "$build_dir" -name 'libggml-cpu.a' -print -quit 2>/dev/null) + +if [ -z "$archive" ]; then + fail "libggml-cpu.a not found under $build_dir" +elif ! command -v objdump >/dev/null 2>&1; then + skip "objdump unavailable; cannot inspect $archive" +else + disasm=$(objdump -d "$archive" 2>/dev/null) + sdot=$(printf '%s' "$disasm" | grep -cE '\bsdot\b') + smmla=$(printf '%s' "$disasm" | grep -cE '\bsmmla\b') + sve=$(printf '%s' "$disasm" | grep -cE '\bz[0-9]+\.[bhsd]\b') + + if [ "$sdot" -gt 0 ] && [ "$smmla" -gt 0 ]; then + pass "libggml-cpu.a: $sdot sdot, $smmla smmla, $sve SVE operands" + else + fail "libggml-cpu.a: $sdot sdot, $smmla smmla — the Q4/Q8 matmul kernels are + compiled at base ARMv8-A. The -march flag did not survive to the compiler." + fi +fi + +if [ "$fails" -ne 0 ]; then + spark_die "$fails check(s) failed" +fi +spark_log "all checks passed" diff --git a/test/makefile_arch_flags_test.exs b/test/makefile_arch_flags_test.exs new file mode 100644 index 0000000..979e74a --- /dev/null +++ b/test/makefile_arch_flags_test.exs @@ -0,0 +1,317 @@ +defmodule LlamaCppEx.MakefileArchFlagsTest do + use ExUnit.Case, async: true + + # LLAMA_CPU_ARM_ARCH and LLAMA_CUDA_ARCH are chained: reaching the ARM CPU flag + # requires GGML_NATIVE=OFF, and GGML_NATIVE=OFF drops ggml-cuda out of `native` + # into a seven-architecture fat binary. Setting one without the other is a + # silent ~6x build-time regression, so the Makefile refuses. That refusal, and + # the build-directory key that keeps a toggle from reusing a stale + # CMakeCache.txt, are the two things worth pinning. + # + # `make print-` echoes one fully expanded variable and runs no part of the + # build. Hermetic on macOS and Linux alike as long as three inputs are pinned + # rather than discovered: MIX_APP_PATH (the Makefile requires it), + # LLAMA_BACKEND (otherwise `auto` answers differently per host), and CUDA_HOME + # (the Linux link-flag block errors when it finds no toolkit libraries). The + # fake toolkit below supplies the last one; nothing is ever executed from it. + # + # scripts/spark/verify-build-flags.sh covers what this cannot: whether the + # flags survived cmake into the compiler command line and into the emitted + # machine code. This file covers whether the Makefile emits them at all. + + @cpu_arch "armv9.2-a+dotprod+i8mm+fp16+bf16+sve2" + @cuda_arch "121a-real" + + setup_all do + root = Path.expand("..", __DIR__) + + tmp = + Path.join( + System.tmp_dir!(), + "llama_cpp_ex_make_probe_#{System.unique_integer([:positive])}" + ) + + File.mkdir_p!(Path.join(tmp, "cuda/bin")) + File.mkdir_p!(Path.join(tmp, "cuda/lib64")) + File.write!(Path.join(tmp, "cuda/bin/nvcc"), "#!/bin/sh\nexit 0\n") + File.chmod!(Path.join(tmp, "cuda/bin/nvcc"), 0o755) + + on_exit(fn -> File.rm_rf!(tmp) end) + + %{root: root, cuda_home: Path.join(tmp, "cuda"), app_path: Path.join(tmp, "app")} + end + + # {:ok, %{flags: [...], build: "..."}}, or {:error, output} when make refused + # to parse — which is what $(error) does. + defp probe(ctx, env) do + env = + Map.merge( + %{ + "MIX_APP_PATH" => ctx.app_path, + "CUDA_HOME" => ctx.cuda_home, + "LLAMA_BACKEND" => "cuda", + # Every variable the Makefile reads has to be pinned, not just the + # ones under test: scripts/spark/remote.sh exports the whole Spark + # build contract, so an unpinned one leaks in and the build-directory + # assertions fail on a Spark while passing on a laptop. + "LLAMA_CPU_ARM_ARCH" => "", + "LLAMA_CUDA_ARCH" => "", + "LLAMA_PORTABLE" => "", + "LLAMA_CMAKE_ARGS" => "", + "LLAMA_CUDA_NCCL" => "0", + "LLAMA_RPC" => "0", + "LLAMA_RPC_RDMA" => "0" + }, + env + ) + + case System.cmd("make", ["print-CMAKE_FLAGS", "print-LLAMA_BUILD"], + cd: ctx.root, + env: env, + stderr_to_stdout: true + ) do + {out, 0} -> + [flags, build] = out |> String.trim() |> String.split("\n") + {:ok, %{flags: String.split(flags, ~r/\s+/, trim: true), build: build}} + + {out, _} -> + {:error, out} + end + end + + describe "the CPU/CUDA architecture pairing" do + test "both unset leaves today's behaviour untouched", ctx do + assert {:ok, %{flags: flags, build: build}} = probe(ctx, %{}) + + refute "-DGGML_NATIVE=OFF" in flags + refute Enum.any?(flags, &String.starts_with?(&1, "-DGGML_CPU_ARM_ARCH=")) + refute Enum.any?(flags, &String.starts_with?(&1, "-DCMAKE_CUDA_ARCHITECTURES=")) + + assert String.ends_with?(build, "llama_build-cuda"), + "an unflagged build must keep its existing build directory, got #{build}" + end + + test "both set emits the paired flags exactly once", ctx do + assert {:ok, %{flags: flags}} = + probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => @cpu_arch, "LLAMA_CUDA_ARCH" => @cuda_arch}) + + assert Enum.count(flags, &(&1 == "-DGGML_NATIVE=OFF")) == 1 + assert "-DGGML_CPU_ARM_ARCH=#{@cpu_arch}" in flags + assert "-DCMAKE_CUDA_ARCHITECTURES=#{@cuda_arch}" in flags + end + + # The whole reason the pairing is enforced rather than documented. + test "LLAMA_CPU_ARM_ARCH alone on a CUDA build is a hard error", ctx do + assert {:error, out} = probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => @cpu_arch}) + assert out =~ "LLAMA_CPU_ARM_ARCH requires GGML_NATIVE=OFF" + assert out =~ "LLAMA_CUDA_ARCH" + end + + # No CUDA, no fat-binary hazard, so no reason to demand a CUDA architecture. + test "LLAMA_CPU_ARM_ARCH alone on a non-CUDA build is allowed", ctx do + assert {:ok, %{flags: flags}} = + probe(ctx, %{"LLAMA_BACKEND" => "cpu", "LLAMA_CPU_ARM_ARCH" => @cpu_arch}) + + assert "-DGGML_NATIVE=OFF" in flags + assert "-DGGML_CPU_ARM_ARCH=#{@cpu_arch}" in flags + refute Enum.any?(flags, &String.starts_with?(&1, "-DCMAKE_CUDA_ARCHITECTURES=")) + end + + # LLAMA_PORTABLE sets GGML_NATIVE=OFF for a different reason — artifacts that + # leave the machine, built on release runners with no GPU to pin an arch for. + # The two sources must not double-emit, and portable alone must keep working + # with no CUDA architecture named. + test "LLAMA_PORTABLE and LLAMA_CPU_ARM_ARCH together emit GGML_NATIVE=OFF once", ctx do + assert {:ok, %{flags: flags}} = + probe(ctx, %{ + "LLAMA_PORTABLE" => "1", + "LLAMA_CPU_ARM_ARCH" => @cpu_arch, + "LLAMA_CUDA_ARCH" => @cuda_arch + }) + + assert Enum.count(flags, &(&1 == "-DGGML_NATIVE=OFF")) == 1 + end + + test "LLAMA_PORTABLE alone still needs no CUDA architecture", ctx do + assert {:ok, %{flags: flags}} = probe(ctx, %{"LLAMA_PORTABLE" => "1"}) + assert "-DGGML_NATIVE=OFF" in flags + end + end + + describe "the build-directory key" do + # Without this, toggling the flags reuses the previous CMakeCache.txt and the + # new configuration silently no-ops — the exact failure the key exists for. + test "differs between flag sets and is stable within one", ctx do + {:ok, a} = probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => @cpu_arch, "LLAMA_CUDA_ARCH" => @cuda_arch}) + + {:ok, a_again} = + probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => @cpu_arch, "LLAMA_CUDA_ARCH" => @cuda_arch}) + + {:ok, b} = + probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => "armv8.2-a+dotprod", "LLAMA_CUDA_ARCH" => @cuda_arch}) + + {:ok, c} = probe(ctx, %{"LLAMA_CPU_ARM_ARCH" => @cpu_arch, "LLAMA_CUDA_ARCH" => "90-real"}) + + assert a.build == a_again.build + assert a.build != b.build + assert a.build != c.build + + # Portability is a separate axis and must stay one. + {:ok, portable} = + probe(ctx, %{ + "LLAMA_PORTABLE" => "1", + "LLAMA_CPU_ARM_ARCH" => @cpu_arch, + "LLAMA_CUDA_ARCH" => @cuda_arch + }) + + assert portable.build != a.build + end + + # LLAMA_RPC adds a whole backend library and a public define, and the RDMA + # toggle changes the code inside it. Both have to key the directory or a + # toggle silently reuses the previous cmake tree. + test "RPC and its RDMA toggle are part of the key", ctx do + {:ok, off} = probe(ctx, %{}) + {:ok, on} = probe(ctx, %{"LLAMA_RPC" => "1", "LLAMA_RPC_RDMA" => "1"}) + {:ok, tcp} = probe(ctx, %{"LLAMA_RPC" => "1", "LLAMA_RPC_RDMA" => "0"}) + + assert off.build != on.build + assert String.ends_with?(tcp.build, "-rpc-tcp") + + case :os.type() do + {:unix, :linux} -> + # RDMA is a real option here, so the two RPC builds must not collide. + assert String.ends_with?(on.build, "-rpc") + assert on.build != tcp.build + + _ -> + # ggml forces GGML_RPC_RDMA off anywhere but Linux, so asking for it is + # a configure-time lie and the Makefile does not pass it on. + assert String.ends_with?(on.build, "-rpc-tcp") + end + end + end + + describe "the RPC flags" do + # Stated in both directions, never left to ggml's default, so the cmake + # configuration and the hand-assembled link line cannot disagree — the same + # discipline the NCCL comment in the Makefile exists to enforce. + test "GGML_RPC is always stated explicitly", ctx do + assert {:ok, %{flags: off}} = probe(ctx, %{}) + assert "-DGGML_RPC=OFF" in off + + assert {:ok, %{flags: on}} = probe(ctx, %{"LLAMA_RPC" => "1"}) + assert "-DGGML_RPC=ON" in on + refute "-DGGML_RPC=OFF" in on + end + + test "GGML_RPC_RDMA is declared, never auto-detected", ctx do + # ggml/src/ggml-rpc/CMakeLists.txt:11-22 turns RDMA on whenever libibverbs + # happens to exist on the build host. Same source, different artifact per + # machine — exactly what this pins shut. + assert {:ok, %{flags: on}} = probe(ctx, %{"LLAMA_RPC" => "1", "LLAMA_RPC_RDMA" => "1"}) + assert {:ok, %{flags: off}} = probe(ctx, %{"LLAMA_RPC" => "1", "LLAMA_RPC_RDMA" => "0"}) + + assert "-DGGML_RPC_RDMA=OFF" in off + + case :os.type() do + {:unix, :linux} -> assert "-DGGML_RPC_RDMA=ON" in on + _ -> assert "-DGGML_RPC_RDMA=OFF" in on + end + end + end + + describe "the configuration stamp" do + # File timestamps do not capture a flag change: toggling LLAMA_RPC alters + # CXXFLAGS and the archive set while touching no source, so make kept a + # previously linked .so and shipped a NIF missing functions it was built to + # export. Silent, and it looks like an Elixir bug. The stamp's name carries + # a hash of the configuration so the prerequisite disappears when it changes. + defp config_stamp(ctx, env) do + base = %{ + "MIX_APP_PATH" => ctx.app_path, + "CUDA_HOME" => ctx.cuda_home, + "LLAMA_BACKEND" => "cuda", + "LLAMA_CPU_ARM_ARCH" => "", + "LLAMA_CUDA_ARCH" => "", + "LLAMA_PORTABLE" => "", + "LLAMA_CMAKE_ARGS" => "", + "LLAMA_CUDA_NCCL" => "0", + "LLAMA_RPC" => "0", + "LLAMA_RPC_RDMA" => "0" + } + + {out, 0} = + System.cmd("make", ["print-LLAMA_CONFIG_STAMP", "print-NIF_LINK_STAMP"], + cd: ctx.root, + env: Map.merge(base, env), + stderr_to_stdout: true + ) + + [config, link] = out |> String.trim() |> String.split("\n") + %{config: config, link: link} + end + + test "changes with the compiler flags", ctx do + # -DGGML_USE_RPC lands in CXXFLAGS and nowhere else observable. + assert config_stamp(ctx, %{}).config != config_stamp(ctx, %{"LLAMA_RPC" => "1"}).config + end + + test "changes with the linker and cmake flags", ctx do + # NCCL changes LDFLAGS on Linux and CMAKE_FLAGS everywhere, while leaving + # the build directory alone — exactly what a timestamp rule cannot see. + assert config_stamp(ctx, %{}).config != + config_stamp(ctx, %{"LLAMA_CUDA_NCCL" => "1"}).config + end + + test "is stable for an unchanged configuration", ctx do + assert config_stamp(ctx, %{}) == config_stamp(ctx, %{}) + end + + # The .so lives in priv/, which Mix symlinks into every MIX_ENV's build tree, + # so dev, test and bench share ONE artifact while keeping separate objects + # and separate llama.cpp trees. Two ways that goes wrong, both observed: + # another MIX_ENV linking a different configuration over it, and Mix copying + # a downloaded precompiled artifact over it with a fresh mtime. Neither is + # visible to a timestamp rule, so the marker records what the artifact IS — + # config hash plus a digest of the linked bytes — and lives beside it. + test "the link marker sits beside the shared artifact", ctx do + %{link: link} = config_stamp(ctx, %{}) + + assert String.ends_with?(link, "/priv/.llama_cpp_ex_nif.built"), + "the marker must live beside the artifact it describes, got #{link}" + end + + test "the marker is shared, not per-configuration", ctx do + # Deliberately one filename: a per-config *name* would let two + # environments' markers coexist beside a single artifact, which is exactly + # the ambiguity being removed. Discrimination is by content. + assert config_stamp(ctx, %{}).link == + config_stamp(ctx, %{"LLAMA_RPC" => "1"}).link + end + + test "`all` runs the artifact check before deciding to link", ctx do + # Without this wiring the digest is recorded and never consulted. + {out, 0} = + System.cmd("make", ["-n", "all"], + cd: ctx.root, + env: %{ + "MIX_APP_PATH" => ctx.app_path, + "CUDA_HOME" => ctx.cuda_home, + "LLAMA_BACKEND" => "cuda", + "LLAMA_CPU_ARM_ARCH" => "", + "LLAMA_CUDA_ARCH" => "", + "LLAMA_PORTABLE" => "", + "LLAMA_CMAKE_ARGS" => "", + "LLAMA_CUDA_NCCL" => "0", + "LLAMA_RPC" => "0", + "LLAMA_RPC_RDMA" => "0" + }, + stderr_to_stdout: true + ) + + assert out =~ ".llama_cpp_ex_nif.built", + "`make all` must consult the link marker" + end + end +end diff --git a/test/rpc_test.exs b/test/rpc_test.exs new file mode 100644 index 0000000..7e7eba3 --- /dev/null +++ b/test/rpc_test.exs @@ -0,0 +1,243 @@ +defmodule LlamaCppEx.RPCTest do + use ExUnit.Case, async: true + + alias LlamaCppEx.{Model, RPC, Server} + + # The RPC backend is opt-in at build time. The contract worth pinning on a + # default build is that it degrades *cleanly*: every entry point reports an + # error tuple, nothing raises, nothing aborts, and the option plumbing accepts + # the same keys either way — so a build flag is the only difference between the + # two configurations. + # + # These tests run on BOTH configurations and assert the *exact* refusal for the + # build they are on. The earlier version accepted either atom + # (`reason in [:rpc_unsupported, :unreachable]`), which meant an RPC build + # answering `:rpc_unsupported` was indistinguishable from a correct non-RPC + # build. `RPC.supported?/0` makes each assertion exact. + # + # Be precise about what that does and does not buy, because it is tempting to + # overclaim: `supported?/0` reads the *same* artifact as `add_server/1`, so a + # stale or cross-environment `.so` makes the two agree and these tests pass. + # Detecting *that* needs a source of truth outside the artifact — the requested + # build flag — which is the last test in this block. + # + # Tests that need a reachable worker are tagged `:rpc_live`; see + # test/test_helper.exs. + + # The refusal this build must give for an endpoint that cannot serve devices. + # On an RPC build the port is genuinely probed and comes back :unreachable; on a + # non-RPC build the call never leaves the NIF. + defp expected_refusal do + if RPC.supported?(), do: :unreachable, else: :rpc_unsupported + end + + describe "supported?/0" do + test "is a boolean and cannot disagree with the error path" do + supported = RPC.supported?() + assert is_boolean(supported) + + assert {:error, reason} = RPC.add_server("127.0.0.1:1") + assert reason == :rpc_unsupported == not supported + end + + # The one assertion here that can catch a stale or cross-environment + # artifact, because `LLAMA_RPC` is evidence from outside the `.so`. Every + # MIX_ENV shares one `llama_cpp_ex_nif.so` (Mix symlinks `priv/`), so + # building test with `LLAMA_RPC=1` and bench without it used to leave + # whichever ran last in place — twice during development. The Makefile's link + # marker is the fix; this is the tripwire that says the fix stopped working. + # + # A no-op when the variable is unset, which is the common case and cannot be + # helped: nothing else in the running VM knows what was asked for. + test "the loaded NIF matches the build that was requested" do + case System.get_env("LLAMA_RPC") do + flag when flag in ["1", "true", "yes"] -> + assert RPC.supported?(), + "LLAMA_RPC=#{flag} is set, but the loaded NIF reports no RPC backend. " <> + "The artifact is stale or came from another MIX_ENV — check " <> + "priv/.llama_cpp_ex_nif.built against `make print-LLAMA_CONFIG_HASH`." + + _ -> + assert is_boolean(RPC.supported?()) + end + end + end + + describe "add_server/1" do + test "an endpoint that cannot serve devices is an error, never a crash" do + # Port 1 on loopback: nothing listens. Neither build may raise, and neither + # may succeed: upstream's ggml_backend_register silently no-ops on a null + # registration, so a vanished endpoint would leave the model loading onto + # local devices while the caller believed otherwise. + assert RPC.add_server("127.0.0.1:1") == {:error, expected_refusal()} + end + + test "ping/1 reports the same refusal" do + assert RPC.ping("127.0.0.1:1") == {:error, expected_refusal()} + end + end + + describe "add_servers/1" do + test "stops at the first failure and names the endpoint" do + # A partially registered set would place tensors somewhere nobody intended, + # so the whole call fails and says which endpoint did it. + assert RPC.add_servers(["127.0.0.1:1", "127.0.0.1:2"]) == + {:error, {"127.0.0.1:1", expected_refusal()}} + end + + test "an empty list is a no-op" do + assert RPC.add_servers([]) == {:ok, 0} + end + end + + describe "devices/0" do + test "reports only RPC-backed devices" do + # Nothing registered in this VM, so this is empty — and on a machine where + # something *is* registered it must still never include the local backend. + assert Enum.all?(RPC.devices(), &(&1.backend == "RPC")) + end + end + + describe "Model.load/2 with :rpc_servers" do + test "surfaces the registration failure instead of loading" do + assert {:error, message} = + Model.load("/nonexistent/model.gguf", rpc_servers: ["127.0.0.1:1"]) + + assert message =~ "127.0.0.1:1" + + # Registration happens before the load, so this must fail at the endpoint + # and never reach the (also missing) file. Placement is computed from the + # devices that exist at load time; registering afterwards would be useless. + refute message =~ "/nonexistent/model.gguf" + end + + test "an empty list skips registration entirely" do + assert {:error, message} = Model.load("/nonexistent/model.gguf") + assert message =~ "/nonexistent/model.gguf" + end + end + + # Needs a reachable worker, not just an RPC build — so it is tagged separately + # and test_helper.exs excludes it unless LLAMA_RPC_ENDPOINT is set. The + # "a closed port is :unreachable" case that used to live here is now covered + # unconditionally by add_server/1 above, via RPC.supported?/0. + describe "against a live worker" do + @describetag :rpc_live + + # `--include rpc_live` beats the exclusion in test_helper.exs, so the tag + # alone cannot stop this running on a machine with no worker. A compile-time + # `skip:` can: module attributes are evaluated when the file is compiled, + # which for a test file is during the run, so this sees the real environment. + # The result is that asking for the tag without a worker skips with a reason + # instead of failing on a missing variable. + if System.get_env("LLAMA_RPC_ENDPOINT") in [nil, ""] do + @describetag skip: "set LLAMA_RPC_ENDPOINT to a reachable RPC worker" + end + + test "a live worker registers, is idempotent, and appears in devices/0" do + endpoint = System.fetch_env!("LLAMA_RPC_ENDPOINT") + + assert {:ok, n} = RPC.add_server(endpoint) + assert n >= 1 + + # Upstream memoizes per endpoint, so a repeat adds nothing. + assert {:ok, 0} = RPC.add_server(endpoint) + + devices = RPC.devices() + assert Enum.any?(devices, &(&1.description == endpoint)) + + # Hardcoded upstream even when the worker serves only a CPU device. + assert Enum.all?(devices, &(&1.type == :gpu)) + + # There are TWO device orderings and they disagree. This one — the ggml + # registry, which is what devices/0 enumerates — is *registration* order, + # so a locally-detected backend comes first and RPC endpoints are appended + # as they are registered. + all = LlamaCppEx.devices() + refute hd(all).backend == "RPC" + assert List.last(Enum.filter(all, &(&1.backend == "RPC"))).description == endpoint + + # llama.cpp builds a *different* list for placement and inserts RPC + # devices at the FRONT of it (src/llama.cpp:263-273, "to minimize network + # transfers"), so `tensor_split[0]` and `main_gpu: 0` address the remote + # node even though devices/0 shows it last. `gpu_index` is derived from + # registry order and therefore does NOT index tensor_split once an RPC + # device exists. Nothing in this VM can observe the placement list, so the + # invariant pinned here is the one that misleads: the two differ. + rpc_gpu_index = Enum.find(all, &(&1.backend == "RPC")).gpu_index + assert rpc_gpu_index > 0, "registry order put RPC first; docs/multi-gpu.md needs revisiting" + end + end + + describe "split mode encoding" do + # Upstream's llama_split_mode enum (llama.h). A silent drift here places + # tensors somewhere nobody asked for and nothing downstream would notice. + test "maps every mode to its upstream value" do + assert Model.encode_split_mode(:none) == 0 + assert Model.encode_split_mode(:layer) == 1 + assert Model.encode_split_mode(:row) == 2 + assert Model.encode_split_mode(:tensor) == 3 + end + + # There was no fallback clause, so `split_mode: :tensor` raised + # FunctionClauseError from inside load/2 with no indication of what was + # wrong. A typo should say what the accepted set is. + test "an unknown mode raises with the accepted set" do + assert_raise ArgumentError, ~r/unknown split_mode :diagonal/, fn -> + Model.encode_split_mode(:diagonal) + end + + assert_raise ArgumentError, ~r/:none, :layer, :row or :tensor/, fn -> + Model.encode_split_mode("layer") + end + end + end + + describe ":rpc_servers option plumbing" do + test "is a forwardable model tuning option" do + assert :rpc_servers in Model.tuning_option_keys() + refute :rpc_servers in Model.structural_option_keys() + end + + test "Server.start_link/1 accepts it" do + assert :rpc_servers in Server.start_option_keys() + end + + # Registration mutates a process-global device registry and must precede a + # load, so it cannot be a per-request option: a request cannot move layers + # onto another machine mid-flight. + test "request-level calls reject it" do + refute :rpc_servers in Server.request_option_keys() + end + end + + describe ":devices option" do + test "is a forwardable model tuning option" do + assert :devices in Model.tuning_option_keys() + assert :devices in Server.start_option_keys() + refute :devices in Server.request_option_keys() + end + + test "an unknown device name is refused, and says what exists" do + # llama.cpp uses params.devices verbatim with no validation of its own, so + # a typo would otherwise become a null entry in a NULL-terminated array — + # a silently truncated device list. + assert {:error, message} = Model.load("/nonexistent/model.gguf", devices: ["GPU42"]) + + assert message =~ "unknown device: GPU42" + assert message =~ "available:" + refute message =~ "failed to load model" + end + + test "an empty list leaves llama.cpp to build the placement list" do + assert {:error, message} = Model.load("/nonexistent/model.gguf", devices: []) + assert message =~ "failed to load model" + end + + test "a real device name gets past placement and fails on the file" do + name = hd(LlamaCppEx.devices()).name + assert {:error, message} = Model.load("/nonexistent/model.gguf", devices: [name]) + assert message =~ "failed to load model" + end + end +end diff --git a/test/test_helper.exs b/test/test_helper.exs index 57b21c1..1279d69 100644 --- a/test/test_helper.exs +++ b/test/test_helper.exs @@ -11,6 +11,16 @@ # loop over shared contexts. See test/mtp_model_test.exs. # :slow — long-running comparison matrices (F16 vs Q8_0 KV cache); # needs LLAMA_SMOKE_GEN_MODEL +# :rpc_live — needs a *reachable RPC worker*, not just a model: set +# LLAMA_RPC_ENDPOINT to "host:port". Excluded automatically when +# that variable is unset, so `--include rpc_live` without a +# worker is a no-op rather than a failure. +# +# There is deliberately no tag for "needs an RPC build". Those +# tests run on every build and assert the exact behaviour of the +# build they are on, via LlamaCppEx.RPC.supported?/0 — accepting +# either refusal used to hide the stale-artifact bug the +# Makefile's link marker exists to catch. # # `--include` beats `--exclude` in ExUnit, so the tags are independent: opt into # exactly the ones whose model you have. The helper `LlamaCppEx.TestModels` @@ -29,6 +39,9 @@ # LLAMA_SMOKE_MTP_MODEL=/path/to/mtp-model.gguf \ # mix test --include mtp # +# LLAMA_RPC=1 mix compile +# LLAMA_RPC_ENDPOINT=10.100.64.2:50052 mix test --include rpc_live +# # `GGML_METAL_NO_RESIDENCY=1` is only needed on Metal, and only to keep the VM # from aborting *after* the suite has passed: # @@ -50,4 +63,8 @@ Code.require_file("support/test_models.exs", __DIR__) Code.require_file("support/test_slots.exs", __DIR__) -ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp, :mtp_cancel]) +# `:rpc_live` needs a reachable worker, so it is excluded here to keep the +# default run quiet, and rpc_test.exs additionally carries a compile-time `skip:` +# so an explicit `--include rpc_live` without a worker skips rather than fails +# (`--include` beats `--exclude`, so the exclusion alone cannot do that). +ExUnit.start(exclude: [:smoke, :embeddings, :slow, :mtp, :mtp_cancel, :rpc_live]) diff --git a/vendor/llama.cpp b/vendor/llama.cpp index 4801e3c..a94d563 160000 --- a/vendor/llama.cpp +++ b/vendor/llama.cpp @@ -1 +1 @@ -Subproject commit 4801e3c567d5131dd41b387df5f2d4b1370d92be +Subproject commit a94d563ed801d1da1b8c2432946de07d0231bb3d