feat: one command to prepare and serve - #20
Merged
Merged
Conversation
`servekit prepare` is gone as a subcommand. `servekit launch` now takes
`--servekit-artifact-path <dir>`, prepares the checkpoint there if it is
not prepared yet, and serves it. `--model-path` always names the real
model, and `--load-format sharded_state` is added automatically.
servekit launch --servekit-artifact-path /scratch/llama-tp4 \
-- python -m sglang.launch_server --model-path /models/llama --tp 4
`--only-prepare` does the prepare step alone, for building the artifact
in its own job.
An artifact that does not fit the command no longer fails the run: it
names the setting that disagrees and falls back to the engine's own
loader. Same for an artifact path that cannot be written. A manifest
prepared from a different model now counts as a mismatch too.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Both jobs now run the same command; the only difference is whether the artifact directory already holds a prepared checkpoint. So correctness-apertus8b.sbatch takes a FRESH switch that points the artifact at an empty scratch dir, and prepare-serve-apertus8b.sbatch goes away. sbatch_wait takes an env overlay, so FRESH reaches the job without being set on os.environ where it would leak into the other e2e module's jobs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
youssef62
changed the title
Preparing Qwen3-Coder-30B never finished: the MoE loader faults pages out of an mmap'd safetensors file, which on Lustre becomes random reads. It sat in _load_w13 for 30 minutes against ~2 minutes of streaming reads, so the sharding run now sets weight_loader_disable_mmap. The slow-path fallback in launch gets the same flag, since that is the other place servekit hands the original checkpoint to the engine. The fast path keeps mmap, where it reads the staged copy out of tmpfs for free. Also raise the sharding run's distributed timeout. One prepare died when ranks that finished writing waited past torch's 600s default for a rank that had not. That run turned out to be an unusually slow window rather than a real limit, so this is a guard: crossing that boundary throws away a load that already succeeded. Drop the "JIT caches will be built in /tmp/..." line, which read as though the caches were not kept with the checkpoint. Verified on Clariden: Apertus-8B TP4 and TP2/PP2, Llama-70B TP4, and Qwen3-Coder-30B TP4 all prepare from empty and pass servekit verify. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GLM-4.7 gets OOM-killed with the flag on, reproducibly: OUT_OF_MEMORY at 1:39 and 1:37 across two runs, one shard into the load, where the same model on the same nodes loads fine with mmap. Disabling mmap reads each shard into anonymous memory the kernel cannot evict, and the multi-thread loader holds several at once. Size does not predict it, so a threshold would not be safe either: Qwen3-235B carries 219G per node with the flag on and loads in 87s, while GLM-4.7 died carrying 167G per node. Preparing a checkpoint that hangs on mmap is still fixable by passing --weight-loader-disable-mmap in the engine command, which the sharding run already forwards. Servekit no longer chooses it for you. The distributed timeout stays: it guards a real cliff and costs nothing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
servekit prepareis gone.servekit launchdoes both steps now:--servekit-artifact-pathis where servekit keeps the presharded copy. Ifthere is nothing prepared there, launch prepares it first and then serves.
--model-pathalways names the real model, and--load-format sharded_stateis added for you.
Use
--only-prepareto run just the prepare step, for building the artifactin its own job.
Mismatches are warnings now
Before, a command that did not fit the checkpoint made
launchexit 2. Now itsays which setting disagrees and runs the plain engine command instead:
A slower start beats a failed job. Same when the artifact path cannot be
written. With
--only-preparea mismatch rebuilds the artifact instead.One new check: an artifact prepared from a different model is a mismatch. That
is easy to hit now that the artifact dir is a separate argument.