ds4-server Bus error (SIGBUS) in vocab_token_is_literal_special during generation after successful 34K-token distributed prefill

[dongzhang777-source]

Title: ds4-server Bus error (SIGBUS) in vocab_token_is_literal_special during generation after successful 34K-token distributed prefill

Summary

ds4-server crashes with a Bus error (SIGBUS / KERN_MEMORY_ERROR) at the start of generation, immediately after a successful 34,384-token distributed prefill. The prefill itself completes without errors at ~152 t/s (225 seconds). The crash happens when the server tries to convert the first generated token ID back to text via the vocabulary table.

The same long prompt works fine via the CLI (./ds4 --prompt-file), so this appears specific to the ds4-server binary's generation/vocab path.

Crash Details

Type: EXC_BAD_ACCESS | KERN_MEMORY_ERROR at 0x0000007000233f35
Termination: Bus error: 10

Faulting stack:
  ds4-server                vocab_token_is_literal_special
  ds4-server                ds4_token_text
  ds4-server                generate_job_inner
  ds4-server                generate_job
  ds4-server                worker_main
  libsystem_pthread.dylib   _pthread_start

Bus error (not SIGSEGV) suggests a truncated/corrupted mmap region or misaligned access — possibly the vocab mmap became invalid during the 225-second prefill wait.

Environment

• ds4 main branch (commit ~8db89fe / cc231cb era), macOS 26.6.2, arm64, Metal backend
• 2× MacBook Pro M1 Max 64GB, Thunderbolt 4 bridge, static IPs
• Model: DeepSeek-V4-Flash-0731 DS4-Quality128 (95.76 GiB, SHA verified)
• Split: coordinator layers 0:22 (A) / worker layers 23:output (B)
• ctx=262144, prefill-chunk=1024, dist-activation-bits=8
• Prompt: 34,384 tokens (ZCode agent system prompt + tools)

Reproduction

1. Start worker on machine B: ./ds4 --metal -m model.gguf --role worker --layers 23:output --coordinator <A_IP> 5555 -c 262144 --prefill-chunk 1024
2. Start API server on machine A: ./ds4-server --metal -m model.gguf --role coordinator --layers 0:22 --listen <A_IP> 5555 -c 262144 --prefill-chunk 1024 --dist-activation-bits 8 --host 0.0.0.0 --port 8002
3. Send a POST to /v1/chat/completions with a ~34K token message
4. Prefill completes at 100% (~225s), then ds4-server crashes with Bus error

Key Observations

• The prefill completes successfully (100%, 225s, 152 t/s avg) — the distributed pipeline works
• The crash is in generation/vocab lookup, not in compute
• CLI mode (./ds4 --prompt-file) with 51K tokens works fine on the same setup
• Short prompts (<20K) via API work fine
• The Bus error (not SIGSEGV) points to an mmap/truncation issue rather than a wild pointer

Guess

The vocabulary/token table mmap may become invalid during the long prefill wait — possibly USB SSD page eviction, mmap truncation under memory pressure, or a ds4-server-specific mmap handling difference from the CLI path.

Happy to test a fix or provide the full .ips crash report.

[dongzhang777-source]

ROOT CAUSE FOUND: model file on USB SSD — mmap pages become unreadable during long prefill. Fixed by moving to internal NVMe.

Root cause

The model file (96 GiB DeepSeek Quality128 GGUF) was on an external USB SSD. ds4-server mmaps the vocabulary/token table from this file. During a long prefill (~225 seconds), the vocab pages are never touched (vocab is only needed during generation, not prefill). The USB SSD idles during this time, and when generation finally tries to read the vocab pages via mmap, the pages cannot be paged back in from the (now unresponsive/sleeping) USB SSD → SIGBUS.

Fix

Moved the model file from USB SSD to internal NVMe. The Bus error is completely eliminated — 34K-token prefill + generation now completes successfully (257s, no crash).

Additional context

• Short prompts (<20K) worked even on USB SSD because the prefill was fast enough that the SSD didn't idle long enough to cause page eviction
• The issue only manifested with prompts >~25K tokens (prefill >30s)
• TSO=0 on the Thunderbolt bridge and freeing memory from an auto-starting service (unsloth studio, 46GB) were also needed for overall stability, but the Bus error specifically was the USB SSD mmap issue

Recommendation for ds4-server

Consider warning users (or detecting) when the model file is on an external/removable storage device — long prefills may cause mmap page faults on the vocab table when the storage goes to sleep. A fsync or periodic page touch during prefill could also prevent this.