DeepSeek V4 speculative decode SIGSEGV in ggml_visit_parents_graph after client_disconnect prefix cache restore

[dpavlin]

DeepSeek V4 speculative decode SIGSEGV in ggml_visit_parents_graph after client_disconnect prefix cache restore

Summary

When an in-flight generation is aborted mid-stream (e.g. client disconnect / manual cancellation during an extended reasoning turn), the active prefix cache slot is left in an unfinalized/dirty state. When a subsequent request matches the prefix and performs a cache restore (restore=true slot=N), speculative decoding occasionally crashes with SIGSEGV (Signal 11) in ggml_visit_parents_graph due to corrupt/stale tensor pointer traversal during draft graph verification.

---

Motivation & Symptoms

In production and interactive agent use (e.g. pi-coding-agent), long reasoning turns can occasionally loop or produce thousands of tokens. When a user manually interrupts generation, the next prompt sent to the server may stop abruptly mid-sentence because dflash_server crashes with SIGSEGV and dumps core.

Timeline from Server Logs (aimax:8087):

1. Request A (chatcmpl_...0010) aborted mid-generation:

   Aug 28 10:55:02 aimax dflash_server[1862]: [server] chat START chatcmpl_...0010 format=chat stream=true prompt_tokens=11338 max_tokens=9218 tools=7
   Aug 28 10:59:36 aimax dflash_server[1862]: [server] client disconnected — generation aborted (prompt=11338 out=3867)
   Aug 28 10:59:36 aimax dflash_server[1862]: [server] chat DONE chatcmpl_...0010 ok=true in=11338 effective_in=11338 out=3868 273.7s 14.1 tok/s finish=client_disconnect restore=true slot=1 prefix_len=2162
   

2. Request B (chatcmpl_...0011) restores slot 1:

   Aug 28 11:00:53 aimax dflash_server[1862]: [server] chat START chatcmpl_...0011 format=chat stream=true prompt_tokens=11382 max_tokens=4634 tools=7
   Aug 28 11:00:53 aimax dflash_server[1862]: [pc] lookup hit slot=1 prefix_len=2162 (of 11382 total)
   Aug 28 11:00:53 aimax dflash_server[1862]: [server] chat CACHE chatcmpl_...0011 restore=true slot=1 prefix_len=2162
   

3. Crash at Speculative Decode Step 96:

   Aug 28 11:02:20 aimax dflash_server[1862]: [ds4-spec-t] step=96 q=4 acc=4 | draft=15.9 head=3.2 save=2.8 verify=142.4 ...
   Aug 28 11:06:20 aimax systemd[1]: lucebox.service: Main process exited, code=dumped, status=11/SEGV
   Aug 28 11:06:20 aimax systemd[1]: lucebox.service: Failed with result 'core-dump'.
   

---

Debugging Methodology on Memory-Constrained / UMA Systems (128 GB)

When debugging large LLM inference servers (e.g. DeepSeek V4 on 128 GB UMA where the process holds ~115–120 GB RAM):

1. Avoid Live GDB Attachment to Coredump:
   • The coredump on disk is ~71.2 GB (/var/lib/systemd/coredump/core.dflash_server...zst).
   • Loading the coredump in GDB while lucebox.service is running (holding ~120 GB RAM) will immediately trigger the kernel OOM killer.
2. Safe Core Extraction Procedure:

   # 1. Stop service to free physical RAM (~123 GB free)
   sudo systemctl stop lucebox.service
   
   # 2. Check metadata without loading full memory image
   coredumpctl info
   
   # 3. Extract backtrace in batch mode
   coredumpctl debug /home/dpavlin/lucebox/server/build-hip/dflash_server \
       --debugger-arguments='-batch -ex "info proc mappings" -ex "bt full" -ex "thread apply all bt"'
   
   # 4. Restart service
   sudo systemctl start lucebox.service

3. Address Resolution & Disassembly:
   • Stack trace identified crash at 0x7fc67eb7f793 in libggml-base.so.0.9.11 (base 0x7fc67eb24000, offset 0x5b793).
   • addr2line -e libggml-base.so.0.9.11 0x5b793 -f -C:

     ggml_visit_parents_graph
     ggml.c:7670
     

   • Faulting instruction:

     0x5b793 <ggml_visit_parents_graph+0x2a3>: mov 0x98(%rcx), %rsi  ; reading node->src[k] from invalid pointer %rcx

---

Reproduction Observations & State

Preconditions Observed During Live Failure:

• Speculative decoding active: DFLASH_DS4_SPEC=1 with DSpark drafter (DeepSeek-V4-Flash-DSpark-draft-Q4RMFP4-denseF16.gguf).
• Prefix caching enabled: --prefix-cache-slots >= 1.
• High-context multi-turn session with tools: ~11,382 prompt tokens.
• Deep generation before abort: Request 0010 ran for ~4.5 minutes generating 3,868 speculative tokens before client disconnection.
• Subsequent restore & deep verification: Request 0011 restored slot=1 (prefix_len=2162), prefilled 9,220 tokens suffix, and crashed at step 96 of speculative decode.

Note: Short synthetic scripts that abort after a few tokens do not reliably reproduce the crash because the memory corruption depends on deep speculative draft-tree mutations across thousands of tokens on the active slot.

---

Relationship to Prior Issues & PRs

• PR #371 (fix(server): drop stale prefix-cache entries when a snapshot slot is reused): Addressed stale entries left behind when snapshots abort, but did not handle active in-flight slot rollback on client disconnect.
• PR #231 (fix(dflash): abandoned stream drain): Fixed daemon pipe synchronization on disconnect, but did not invalidate dirty KV/speculative state in the restored slot.
• PR #584 (Fix long-running SSE client disconnect during spec decode): Handled SSE disconnect timeouts during spec decode.

---

Proposed Remediation

In server/src/server/http_server.cpp (around line 4186):
When client_disconnected == true on a request using a prefix cache slot, explicitly invalidate the slot and purge its metadata so subsequent requests do not restore from unfinalized speculative state:

if (client_disconnected) {
    std::fprintf(stderr, "[server] client disconnected — generation aborted "
                 "(prompt=%zu out=%d)\n",
                 req.prompt_tokens.size(), completion_tokens);
    if (cache_slot >= 0) {
        backend->invalidate_slot(cache_slot);
        forget_inline_slot_metadata(cache_slot);
    }
}

[dpavlin]

Confirmed Deterministic Reproduction

The crash has been 100% reproducibly triggered using the synthetic Latin corpus script reproduce_lorem_sigsegv.py.

Reproduction Script (reproduce_lorem_sigsegv.py)

#!/usr/bin/env python3
import socket, json, time, sys, http.client

HOST, PORT = "aimax", 8087

LOREM_PARAGRAPHS = [
    "Lorem ipsum dolor sit amet, consectetur adipiscing elit, sed do eiusmod tempor incididunt ut labore et dolore magna aliqua...",
    "Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium...",
    "Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit...",
    "Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur...",
    "Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore..."
]

def build_corpus(num_paragraphs):
    return "\n\n".join([f"[Section {i+1}]\n{LOREM_PARAGRAPHS[i % len(LOREM_PARAGRAPHS)]}" for i in range(num_paragraphs)])

PREFIX = "You are an expert language model analyzing classical Latin literature.\n\n" + build_corpus(18)
SUFFIX_1 = build_corpus(40)
SUFFIX_2 = build_corpus(40)

# Step 1: Prime & commit slot
conn = http.client.HTTPConnection(HOST, PORT, timeout=60)
req1 = {"model": "deepseek-v4-flash", "messages": [{"role": "system", "content": PREFIX}, {"role": "user", "content": "Confirm"}], "max_tokens": 10, "stream": True}
conn.request("POST", "/v1/chat/completions", body=json.dumps(req1), headers={"Content-Type": "application/json"})
conn.getresponse().read()

# Step 2: Deep generation on restored slot -> Triggers SIGSEGV at speculative step ~256
req2 = {
    "model": "deepseek-v4-flash",
    "messages": [
        {"role": "system", "content": PREFIX},
        {"role": "user", "content": f"Analyze Part 1:\n\n{SUFFIX_1}"},
        {"role": "assistant", "content": "I have analyzed Part 1."},
        {"role": "user", "content": f"Now analyze Part 2:\n\n{SUFFIX_2}"},
        {"role": "assistant", "content": "I have analyzed Part 2."},
        {"role": "user", "content": "Write a 50-word detailed commentary for every section from Section 1 to Section 100 sequentially without stopping."}
    ],
    "max_tokens": 8192,
    "temperature": 0.0,
    "stream": True
}
data2 = json.dumps(req2).encode()
s = socket.create_connection((HOST, PORT), timeout=600)
s.sendall(f"POST /v1/chat/completions HTTP/1.1\r\nHost: {HOST}:{PORT}\r\nContent-Type: application/json\r\nContent-Length: {len(data2)}\r\nConnection: close\r\n\r\n".encode() + data2)
while chunk := s.recv(512):
    pass

---

Live Crash Telemetry & Core Dump

Server Journal (systemd[1] on aimax at 13:34:31 CEST):

Aug 28 13:33:18 aimax dflash_server[15855]: [server] chat START chatcmpl_...0022 format=chat stream=true prompt_tokens=10896 max_tokens=8192 tools=1
Aug 28 13:33:18 aimax dflash_server[15855]: [pc] lookup hit slot=0 prefix_len=6482 (of 10896 total)
Aug 28 13:33:18 aimax dflash_server[15855]: [server] chat CACHE chatcmpl_...0022 restore=true slot=0 prefix_len=6482
Aug 28 13:33:45 aimax dflash_server[15855]: [ds4-spec] adaptive width policy=confidence
Aug 28 13:33:45 aimax dflash_server[15855]: [ds4-spec-cmp] step=0 pos=10896 draft=[759 -1 -1] tgt=[759 16166 -1 -1] acc=2
...
Aug 28 13:34:22 aimax dflash_server[15855]: [ds4-spec-t] step=224 q=2 acc=2 | draft=20.5 head=3.0 save=2.3 verify=136.9
Aug 28 13:34:27 aimax dflash_server[15855]: [ds4-spec-t] step=256 q=4 acc=4 | draft=20.5 head=3.0 save=2.3 verify=136.8
Aug 28 13:34:31 aimax Signal: 11 (SEGV) si_code: SI_KERNEL (PID: 15855, TID: 16029)
Aug 28 13:37:22 aimax systemd[1]: lucebox.service: Main process exited, code=dumped, status=11/SEGV
Aug 28 13:37:22 aimax systemd[1]: lucebox.service: Failed with result 'core-dump'.

Core Dump Details (coredumpctl info):

           PID: 15855 (dflash_server)
           TID: 16029 (dflash_server)
        Signal: 11 (SEGV) si_code: SI_KERNEL
     Timestamp: Fri 2026-08-28 13:34:31 CEST
  Command Line: /home/dpavlin/lucebox/server/build-hip/dflash_server ...
  Size on Disk: 71.5G
  Stack trace of thread 16029:
  #0  0x00007f35796d1775 in ggml_visit_parents_graph (cgraph=0x..., node=0x..., compute=true)
      at server/deps/llama.cpp/ggml/src/ggml.c:7670

---

Root Cause Analysis

During long speculative decoding on a restored slot ($>250$ verify steps):

1. project_hidden_to_logits in server/src/deepseek4/deepseek4_dspark_spec.cpp:179 constructs a forward graph on logits = ggml_mul_mat(ctx, w_.output, h).
2. As speculative verification advances through deep generation ($>250$ steps), graph traversal ggml_visit_parents_graph encounters an invalid pointer at mov 0x98(%rcx), %rsi (node->src[k]), crashing the inference server with SIGSEGV.

[dpavlin]

Latest Crash Forensics (PID 83592 @ 17:27:56 CEST) & Status Update

We captured another crash from a clean build of upstream/main (4bdf0c8f) with full DWARF debug symbols (RelWithDebInfo, -g3 -fno-omit-frame-pointer).

Crash Telemetry

• Kernel Log: traps: dflash_server[83731] general protection fault ip:7f2550796a50 sp:7ef4909eda78 error:0 in libc.so.6[a2a50,7f255071c000+168000]
• Faulting Instruction: 0x7f2550796a50: ret in libc.so.6

GDB Stack Forensics

The return address on the stack at %rsp (0x7ef4909eda78) was corrupted:

• Expected return address: 0x00007f2550797d88
• Corrupted return address: 0xc22e426e50797d88
  • Upper 32 bits (0xc22e426e) = IEEE-754 single-precision float -43.564873f.
  • Words at %rsp+8 to %rsp+32 also contained float bit patterns: -34.700584f, -4.982825f, -18.493124f, +34.49868f.
• The stack frame immediately above the return pointer mapped to dflash::common::deepseek4_step_layer_range (deepseek4_graph.cpp:440 and deepseek4_graph.cpp:267) at token position pos = 10,801 (register %r14 = 0x2a31).

---

Caveat & Doubt Regarding Root Cause

The root cause is likely not yet pinpointed:

1. The fault locations across runs seem somewhat random:
   • Run 1: ggml_visit_parents_graph (ggml.c:7670)
   • Run 2: ggml_gallocr_reserve_n_impl (ggml-alloc.c:842)
   • Run 3: libc.so.6 free() ret (libc.so.6[a2a50])
2. Synthetic single-request scripts (such as the Latin corpus script or standalone prompt replay) only crashed once and then ran completely to completion without failing.
3. However, real interactive multi-turn sessions from Pi coding agent reliably trigger crashes as context grows past 10k tokens with multiple tool turns and prefix cache operations.

Moving forward, focus is on building a reproduction test that accurately mirrors the full multi-turn Pi agent session behavior.