Bump to SGLang 0.5.17-cu129, add Blackwell support, and fix five defects

[justinwlin]

Upgrades the worker to SGLang 0.5.17, adds Blackwell support, and fixes five defects found while validating the current 2.0.2 release on live endpoints.

Contributes to DR-1441.

Why

The worker was pinned to lmsysorg/sglang:v0.5.2-cu126, released ~9 months ago. That build has no sm_120 kernels, so on RTX PRO 6000 Blackwell hardware model load dies with CUDA error: no kernel image is available for execution on the device. Blackwell MIG slices sit in the AMPERE_24 and ADA_48_PRO pools, so this is reachable from ordinary GPU selections.

Validating that turned up four more problems that were already broken on main.

What changed

Base image	v0.5.2-cu126 → v0.5.17-cu129. Blackwell now works.
allowedCudaVersions	[12.9, 12.8, 12.7, 12.6] → [13.0, 12.9]
gpuIds	ADA_24 → seven pools, 24GB–80GB
OpenAI SDK	Was impossible to use; now works
Upstream errors	Were returned as successful completions; now fail the job
HF_TOKEN	Restored to hub.json (dropped by #26)
Parser options	Every REASONING_PARSER option was invalid; corrected
README	Adds a Deploy section and hardware requirements
CI	Drops the blacksmith layer cache, which corrupts the larger base image
.gitignore	Ignores __pycache__/ and *.py[cod], which had no entries at all
method override	openai_route callers can now force the HTTP verb

Breaking change

Non-streaming /run and /runsync output changes from a list of JSON strings to a list of objects. This is what makes the documented OpenAI SDK usage possible at all, but anyone parsing json.loads(output[0]) today needs to drop the inner parse. Suggest cutting this as a major release.

The defects, in detail

1. The OpenAI SDK never worked

The README documents driving the worker with the official openai SDK against /openai/v1. The platform's passthrough returns whatever the handler yields, and the handler yielded response text, so the client received a JSON string:

>>> client.chat.completions.create(...)
AttributeError: 'str' object has no attribute 'choices'

Non-streaming responses now yield response.json(). This matches worker-vllm, which yields parsed JSON for non-streaming and raw SSE only while streaming.

2. openai_route without a body crashed the handler

The README's own "List Models" example:

{ "input": { "openai_route": "/v1/models" } }

had no openai_input, so openai_input.get("stream") raised AttributeError: 'NoneType'. Body-less routes now issue a GET.

3. Streaming emitted unparseable SSE

format_chunk re-serialized each chunk with indent=4 and prefixed every resulting line with data: , turning one chat.completion.chunk into ~25 data: lines. No OpenAI-compatible client can read that. Chunks are now emitted compactly, one data: line each.

4. A dead SGLang server silently ate jobs

When sglang.launch_server died during startup, wait_for_server kept polling for the full 900s. The platform marks the worker ready in the meantime, so it accepts jobs it can never serve and the caller is billed for a request that sits IN_QUEUE. PYTHONUNBUFFERED was also unset, so the container produced no logs at all while this happened.

wait_for_server now reaps the child and fails immediately, and logging is unbuffered.

5. hub.json offered values SGLang rejects

Selecting any of these in the Hub UI made launch_server exit at startup:

• All five REASONING_PARSER options (llama3, llama4, mistral, qwen25, deepseekv3) — none are in SGLang's DetectorMap.
• TOOL_CALL_PARSER: llama4 — not in ToolCallParserEnum.
• QUANTIZATION: squeezellm — never accepted, in 0.5.2 or 0.5.17.

Lists are now generated from 0.5.17's source: 33 tool-call parsers, 25 reasoning parsers. HF_TOKEN is also restored — it was added in 771d3b6 and dropped again by #26, so gated models could not be deployed from the UI.

Why allowedCudaVersions narrows to [13.0, 12.9]

The cu129 base declares NVIDIA_REQUIRE_CUDA=cuda>=12.9, so 12.6–12.8 hosts cannot start the container — they fail in the nvidia runtime hook before the handler runs.

allowedCudaVersions is an exact-match list, not a floor. Under the SERVERLESS product, 12.9 exists only on 3090/3090Ti, while 13.0 covers 4090, A100, H100/H200, L40S and every Blackwell MIG slice — so 13.0 is what actually buys breadth. worker-vllm pins ["13.0"] for the same reason.

minCudaVersion would express this properly but is dropped on the Hub deploy path — tracked in CON-1157.

CI

The blacksmith layer cache cannot handle this base image. Every build failed on the same 5.7GB layer:

failed commit on ref "layer-sha256:045d2658...": unexpected commit digest
sha256:1e8cedfc..., expected sha256:045d2658...: failed precondition

Three runs produced three different wrong digests for that one layer, so the blob is mangled in transit rather than a bad entry being cached. Isolated against the same pipeline:

Config	Result
blacksmith cache + cu126 base (unmodified main)	passes, 5m
blacksmith cache + cu129 base	fails 3/3
no blacksmith cache + cu129 base	passes, 6m

Only the cache is implicated. It buys about a minute on a build dominated by pulling a ~19GB base, so dropping it costs effectively nothing — the docker/setup-buildx-action builder already in the job is used instead. Applied to release.yml too, which builds the same image on tag push, and both workflows carry a comment where the step used to sit so it does not get silently re-added.

This is a bug in their cache, not something this repo can fix properly — tracked in DR-1446.

Verification

Built and pushed a test image from this branch, then deployed it on two real endpoints pinned to ["13.0","12.9"] — one on BLACKWELL_96, one on ADA_24 — with MODEL_NAME=HuggingFaceTB/SmolLM2-1.7B-Instruct.

The image is public, so you can reproduce any of the results below without building:

justinrunpod/worker-sglang:dr1441-cu129-v2
sha256:77f65623082b5a77beac3fafddeadf8977723f1fce585850dbd9239178054581

(CI on this branch also publishes runpod/worker-sglang:chore-bump-sglang-0.5.17-cu129, which is the same build from the official pipeline.)

Reproduce it

Create the endpoint. allowedCudaVersions has to be set at creation — the API rejects a PATCH that swaps it for minCudaVersion, and the console/MCP create path sets minCudaVersion by default:

curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" -d '{
  "name": "sglang-review",
  "type": "QUEUE",
  "image": "justinrunpod/worker-sglang:dr1441-cu129-v2",
  "gpu": {"pools": ["BLACKWELL_96"], "count": 1, "allowedCudaVersions": ["13.0", "12.9"]},
  "scaling": {"type": "QUEUE_DELAY", "queueDelay": 4},
  "workers": {"min": 0, "max": 1, "idleTimeout": 5},
  "disk": 60,
  "timeout": 900000,
  "flashboot": "FLASHBOOT",
  "env": {"MODEL_NAME": "HuggingFaceTB/SmolLM2-1.7B-Instruct", "TRUST_REMOTE_CODE": "true"}
}' https://api.runpod.io/v2/serverless

First request cold-starts in ~3-4 min (18.8 GB pull). Swap BLACKWELL_96 for ADA_24 to check the Ada path.

# chat completions
curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"input":{"messages":[{"role":"user","content":"What is the capital of France?"}],"max_tokens":20}}' \
  "https://api.runpod.ai/v2/$ENDPOINT/runsync?wait=90000"

# body-less openai_route - crashes on main
curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"input":{"openai_route":"/v1/models"}}' "https://api.runpod.ai/v2/$ENDPOINT/runsync?wait=90000"

# upstream error - returns as a successful completion on main
curl -X POST -H "Authorization: Bearer $RUNPOD_API_KEY" -H "Content-Type: application/json" \
  -d '{"input":{"messages":[{"role":"user","content":"hi"}],"temperature":-99}}' \
  "https://api.runpod.ai/v2/$ENDPOINT/runsync?wait=90000"

The OpenAI SDK check, which fails on main with 'str' object has no attribute 'choices':

from openai import OpenAI
c = OpenAI(api_key=os.environ["RUNPOD_API_KEY"],
           base_url=f"https://api.runpod.ai/v2/{ENDPOINT}/openai/v1")
r = c.chat.completions.create(model="HuggingFaceTB/SmolLM2-1.7B-Instruct",
                              messages=[{"role": "user", "content": "Say hi in 2 words"}],
                              max_tokens=16)
print(r.choices[0].message.content)

Build it yourself with docker buildx build --platform linux/amd64 -t <your-tag> .

Test	Blackwell	4090
Chat completions	✅	✅
Native /generate	✅	—
openai_route: /v1/models	✅	—
Streaming SSE	✅	—
OpenAI SDK, non-streaming	✅	—
OpenAI SDK, streaming	✅	—
Upstream 400/404 → FAILED	✅	—

Evidence

Blackwell, chat completions — the hardware that fails on main:

{"delayTime":256124,"executionTime":854,"status":"COMPLETED",
 "output":[{"choices":[{"message":{"role":"assistant","content":"Paris"},
 "finish_reason":"stop"}],"model":"HuggingFaceTB/SmolLM2-1.7B-Instruct"}]}

OpenAI SDK, both modes, against /openai/v1:

non-stream SDK OK: 'Sure, Hi!'
stream SDK OK: '1. One.\n2. Two.\n3. Three.'

openai_route with no body — crashes on main:

{"status":"COMPLETED","output":[{"object":"list","data":[
 {"id":"HuggingFaceTB/SmolLM2-1.7B-Instruct","max_model_len":8192}]}]}

Streaming — 11 chunks, one data: line each (main produces ~25 lines per chunk).

Upstream error handling:

{"status":"FAILED","error":"SGLang returned HTTP 400: temperature must be a non-negative finite number, got -99.0."}

What the failure looks like on main, from a worker on cu126 Blackwell:

torch.AcceleratorError: CUDA error: no kernel image is available for execution on the device
[2026-08-19 16:15:56] Received sigquit from a child process. It usually means the child failed.

Docker build. uv pip install --system fails on the new base — cu129 adds a PEP 668 EXTERNALLY-MANAGED marker that cu126 lacked:

error: The interpreter at /usr is externally managed
hint: Virtual environments were not considered due to the `--system` flag

Fixed with --break-system-packages, and the install target confirmed importable by the interpreter that runs the handler:

python3: /usr/bin/python3
runpod 1.12.0 /usr/local/lib/python3.12/dist-packages/runpod/__init__.py
deps ok
launch_server importable
PYTHONUNBUFFERED=1

All 34 flags engine.py passes were checked against 0.5.17's server_args dataclass. One is gone — --enable-flashinfer-mla, removed upstream, and passing it fails startup — so ENABLE_FLASHINFER_MLA is dropped from hub.json and engine.py. Every other flag still exists.

Both test endpoints have been deleted.

Review feedback

• Committed bytecode removed. __pycache__/utils.cpython-314.pyc slipped in via a broad git add while testing locally. Deleted, and .gitignore now covers __pycache__/ and *.py[cod] — it previously had no Python entries. The Dockerfile COPYs source by name, so it never reached the image.
• HTTP verb on openai_route is now explicit. The default is unchanged and correct: bodies POST, body-less requests GET, because SGLang registers /v1/models and /health as GET-only (@app.get("/v1/models"), http_server.py:1823) and POSTing to them was the crash this PR fixes. But the rule was implicit, so callers can now pass method to override it, and the README documents both. This also covers the one edge the heuristic arguably gets wrong — an explicitly empty openai_input is falsy and would otherwise route to GET.

Notes for the reviewer

• Ships the fix that open PR fix(handler): return error response when SGLang requests fail with non-200 status #32 asks for (structured error on non-200), so that PR can likely close.
• BLACKWELL_96 is deliberately not in the default gpuIds — $3.49/hr serverless. Blackwell is still reachable via the MIG slices in AMPERE_24 at $0.69.

What was and wasn't exercised

Covered on both Blackwell and Ada: chat completions via the queue API. Covered on Blackwell only (shared code path, but worth knowing the table's dashes are gaps, not passes): /v1/models, native /generate, streaming, both OpenAI SDK modes, and error handling.

Not exercised at all:

• Tool-calling and reasoning parsers. Both option lists were rewritten, but validated by reading ToolCallParserEnum and DetectorMap in 0.5.17's source — no model was deployed with a parser actually set.
• HF_TOKEN against a gated model. Restored to hub.json, never used against a real gated repo.
• The Hub build path. hub.json and tests.json changed substantially, but they're consumed by the Hub builder and this listing isn't published, so nothing validated them end to end.
• Multi-GPU (TENSOR_PARALLEL_SIZE > 1), quantization, and large models. All verification used a single GPU and a 1.7B model, chosen to keep cold starts short — which means nothing here says anything about memory pressure or parallelism.

Testing was serverless only; no pod deploys.

[JessicaGarson]

Found two small items here.

[JessicaGarson]

Shouldn't this be a POST request instead of a GET request

[justinwlin]

TLDR: This is just ternary if it has a body use post, otherwise use GET. But I change it to be more explicit if user wanted to drive the method themselves rather than implicit derivation.

method = (job_input.get("method") or ("POST" if body else "GET")).upper()

Ex.
Someone will do /v1/models with no body should be get. But if they give a body, they probably trying to do a POST with their route. But modified it so user has option to override.