Recover PD engine crashes and NATS JetStream filestore failures - #162
Merged
jiejingzhangamd merged 3 commits intoSep 9, 2026
Merged
Conversation
weilei0120
requested review from
JohnQinAMD,
jiejingzhangamd,
limou102 and
xiaobochen-amd
as code owners
September 9, 2026 09:53
A SIGKILL on the inference subprocess used to finish the worker with exit 0, and a 1P1D outage was reported as a missing mixed worker. KV bucket put failures now back off geometrically instead of flooding logs. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: leiwei12 <lei.wei@amd.com>
An empty -sd / corrupt filestore cannot open msg blocks, so KV view and event streams retry in MEMORY. The operator pins nats 2.11.1 and passes store_dir as a literal path. Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: leiwei12 <lei.wei@amd.com>
Signed-off-by: leiwei12 <lei.wei@amd.com> Co-authored-by: Cursor <cursoragent@cursor.com> Signed-off-by: leiwei12 <lei.wei@amd.com>
weilei0120
force-pushed
the
feature/weilei/engine-crash-and-nats-filestore
branch
from
September 9, 2026 09:57
6b731fa to
fc7f524
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
The new relay filestore-recovery flags can permanently disable recovery after a failed remediation attempt, which can prevent later self-healing under repeated 10077 conditions.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR improves resiliency and diagnostics across routing, KV/NATS JetStream persistence, and engine subprocess lifecycle handling, so the system fails with clearer 503s and can keep operating when JetStream FILE storage becomes unusable.
Changes:
- Router now distinguishes “half PD pool missing” vs “no active worker” (Python + Rust) with clearer 503 error messages.
- KV/NATS components detect JetStream filestore error 10077 / msg-block failures and recreate streams/KV buckets in MEMORY to keep publishing and bootstrapping.
- Engine entrypoints propagate observed engine subprocess death as a non-zero launcher exit code; operator NATS defaults updated (pinned image + literal store dir).
File summaries
| File | Description |
|---|---|
| tests/unit/router/test_failover.py | Adds coverage for empty fleet error messaging in Python auto-router. |
| tests/unit/kv/test_nats_relay.py | Adds validation + tests for geometric logging and filestore rebuild behavior in relay. |
| tests/unit/kv/test_nats_bus.py | Adds unit tests for filestore-failure detection helper. |
| tests/unit/engine/test_engine_death_exit.py | Adds tests for subprocess returncode→exit status mapping and watcher behavior. |
| rust/router/tests/functional.rs | Adds functional coverage for PD-missing-pool vs empty-fleet error messages. |
| rust/router/src/proxy.rs | Improves routed dispatch error reporting for PD pool absence and empty fleets. |
| rust/router/src/kv_event_nats.rs | Adds filestore-failure detection and MEMORY fallback for JetStream stream/KV operations. |
| infera/router/auto.py | Improves 503 error selection for empty fleet vs PD half-pool deployments. |
| infera/kv/nats_relay.py | Adds worker_id validation and recovery behavior for JetStream filestore failures + geometric logging. |
| infera/kv/nats_bus.py | Adds filestore-failure detection and FILE→MEMORY fallback/rebuild for stream + KV bucket creation. |
| infera/engine/vllm/main.py | Propagates observed engine death as non-zero exit status. |
| infera/engine/sglang/main.py | Propagates observed engine death as non-zero exit status. |
| infera/engine/atom/main.py | Propagates observed engine death as non-zero exit status. |
| infera/engine/base.py | Introduces EngineDeath + extends watcher to record returncode and map to exit status. |
| deploy/operator/internal/controller/nats.go | Pins NATS image and uses literal JetStream store dir in args. |
Review details
- Files reviewed: 15/15 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Comment on lines
+160
to
+168
| if js_store_failed(exc) and not self._stream_healed: | ||
| self._stream_healed = True | ||
| await self._bus.ensure_event_stream() | ||
| try: | ||
| await self._bus.js_publish(subject, payload) | ||
| except Exception as retry_exc: | ||
| logger.warning("KV relay NATS publish failed: %s", retry_exc) | ||
| else: | ||
| logger.warning("KV relay NATS publish failed: %s", exc) |
Comment on lines
+274
to
+293
| if js_store_failed(exc) and not self._kv_healed: | ||
| self._kv_healed = True | ||
| store = await self._bus.rebuild_kv_view_store() | ||
| if store is not None: | ||
| self._kv = store | ||
| try: | ||
| await self._kv.put( | ||
| kv_key_for_worker(self._worker_id, rank), | ||
| self._encoder.encode(view), | ||
| ) | ||
| self._dirty[rank] = False | ||
| self._bucket_failures = 0 | ||
| self._next_bucket_warn = 1 | ||
| logger.warning( | ||
| "KV view bucket rebuilt after filestore failure (r%d)", rank | ||
| ) | ||
| return | ||
| except Exception as retry_exc: | ||
| self._note_bucket_failure(rank, retry_exc) | ||
| return |
Comment on lines
85
to
88
| Image: natsImage(idep), | ||
| // K8s expands $(NATS_STORE_DIR) in args from the env below. | ||
| Args: []string{"-js", "-sd", "$(NATS_STORE_DIR)", "-m", "8222"}, | ||
| Env: []corev1.EnvVar{{Name: "NATS_STORE_DIR", Value: "/data/jetstream"}}, | ||
| Args: []string{"-js", "-sd", "/data/jetstream", "-m", "8222"}, | ||
| Env: []corev1.EnvVar{{Name: "NATS_STORE_DIR", Value: "/data/jetstream"}}, | ||
| Ports: []corev1.ContainerPort{ |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nats:2.11.1and a literal/data/jetstreamstore dir instead of CRI-dependent$(NATS_STORE_DIR)expansion.Test plan
pytest -n auto tests/unit/kv/test_nats_bus.py tests/unit/kv/test_nats_relay.py tests/unit/engine/test_engine_death_exit.py-js -sd /data/jetstream -m 8222Made with Cursor