[faresobeid]

Stacked on #4022 (which is on #3971). Review only the top commit (git diff feat/slurm-engine-steps...feat/engine-groups).

With #3971 the unit of eviction is one admin URL. A multi-node expert-parallel replica or a P/D role replica runs one engine per DP rank, and those engines fail together: one dead node stalls every rank of its group. So the ranks were evicted one by one (up to 300 s each), and a single relaunched rank could rejoin a group that cannot serve. A rejoining engine was also routed as soon as /update_weights returned, with no check of what it holds.

What

• Engine groups. orchestrator.model.client.admin_engine_groups gives the group of each admin_base_url entry. The default is one group per URL.
  • AdminPlane evicts a whole group when any of its engines fails a liveness probe, stalls or fails an admin op.
  • A group recovers only when all its engines answer again. It rejoins the weight updates, and a regroup on NCCL, as a block.
• Version gate. /update_weights and /load_lora_adapter now carry the policy version. The vLLM server stores it and serves it on GET /weight_version. A rejoining group goes back into the router only if every engine in it reports the version the update just sent. Otherwise it stays out and retries at the next update.
• SLURM. multi_node_rl.sbatch.j2 builds ADMIN_GROUPS next to ADMIN_URLS and passes them to the orchestrator. The groups match feat: evict inference engine groups as a block and run each as its own SLURM step, so an engine exit does not end the job #4022's engine-group steps:
  • one group per P/D role replica;
  • one per node with expert parallelism;
  • one per engine otherwise.

Integrity triggers:

• feat: opt-in cross-engine weight checksums after weight updates #4000 checksum mismatch → eviction is a separate draft PR stacked on this one: feat(orchestrator): evict the engine group whose weight checksums lose the majority vote #4034.
• feat(trainer): measure how often replayed routes agree with the trainer's routing #3989 router-replay disagreement → eviction is not included. feat(trainer): measure how often replayed routes agree with the trainer's routing #3989's agreement is a trainer-wide metric, and samples carry no engine identity, so a disagreement cannot be traced to a group. That needs a per-sample engine tag and per-engine stats in the trainer.

P/D: vllm-router cannot re-add a prefill or decode worker through its API (PdRouterBase::add_worker). So a P/D group still takes traffic again only after a router restart. Removal works.

Checks

• pytest tests/unit/orchestrator tests/unit/test_configs.py -m "not gpu": 338 passed. ruff is clean. New test: test_engine_group_is_evicted_and_readmitted_as_one, which covers group eviction, partial recovery, and the version gate.

• Rendered sbatch:

  • GLM-5.3 P/D (4 prefill replicas × 2 nodes, 1 decode replica × 8 nodes, DP 8 per node) gives 128 URLs in groups 0–3 (16 each) and 4 (64).
  • GLM-4.5-Air multi-node EP (4 nodes, TP 8) gives groups 0–3.
  • bash -n passes. --model.client.admin-engine-groups 0 0 parses.

• GPU, 2 H200 nodes, job 3827. Setup:

  • one inference node with 4 TP=1 Qwen3-0.6B-Reverse-Text-SFT engines behind vllm-router, admin_engine_groups = [0, 0, 1, 1];
  • one node with the trainer and orchestrator, NCCL broadcast, reverse-text;
  • the run also had feat(orchestrator): evict the engine group whose weight checksums lose the majority vote #4034's evict_mismatched on.

  Results:

  • SIGKILL of engine 8101 at v4: Evicted inference engines …:8100, …:8101 (/pause failed … on …:8101); 2 engines left. The router listed only 8102/8103, and training continued.
  • 8101 relaunched: both engines logged answers again; its group rejoins at the next weight update, then generation 3: 4 engines and Inference engines …:8100, …:8101 rejoined at policy v51. The router listed all 4 again, and /weight_version returned 51 on all 4.
  • While the group was out, 8100 kept reporting {"version": 5} and stayed out of the router.
  • 60 steps, no traceback.

• That run showed a bug, fixed in the second commit: when one engine of a group failed /pause, the engines of the group that had paused fine were still passed to the NCCL regroup (generation 1: 3 engines, then generation 2: 2 engines at the next update). AdminPlane._each now returns only engines that are still members.