[faresobeid]

Stacked on #4033 (engine groups), with #4000 (weight checksums) merged in. Review only the top commit (git diff 537db777b...feat/checksum-evict).

#4000 detects engines whose weights differ after an update, but it can only warn or fail the run. With engine eviction (#3971) the bad engine can instead leave and reload.

What

• [weight_broadcast.checksum] evict_mismatched = true (off by default):
  • after an update, and still paused, an engine whose checksums for a shard differ from a strict majority of the engines holding that shard is evicted with its engine group;
  • it rejoins at a later update, and is evicted again before it resumes if its checksums still differ;
  • a shard needs at least three holders. Without a majority, a mismatch warns, or raises with fail_on_mismatch, as before.
• The checksum runs over the engines that took part in the update, not the full client list.
• find_checksum_outliers holds the vote logic and has a unit test.

Checks

• pytest tests/unit/orchestrator tests/unit/test_configs.py tests/unit/inference -m "not gpu": 358 passed. ruff is clean.
• GPU, 2 H200 nodes, job 3827, in the same run as feat(orchestrator): evict and readmit inference engine groups, route them only at the current weight version #4033: 4 TP=1 engines that all hold shard tp0-pp0. A validation-only patch, not in this PR, adds 1.0 to one element of embed_tokens on engine 8103 when its checksums are computed.
  • At v52: Weight checksums of policy v52 mismatch for 1 tensors: engine1/tp0-pp0/model.embed_tokens.weight != engine0, then Evicted inference engines …:8102, …:8103 (weight checksums of policy v52 differ from the majority …).
  • At v54, after rejoining, the corrupted element was still there, and the group was evicted again.
  • From v57 the checksums matched: rejoined at policy v57, all 4 engines in the router at v57.
  • Training never stopped.