Skip to content

feat(moe): add bounded expert warm-set controller - #352

Open
PhilipJohnBasile wants to merge 1 commit into
youssofal:mainfrom
PhilipJohnBasile:feat/expert-residency
Open

feat(moe): add bounded expert warm-set controller#352
PhilipJohnBasile wants to merge 1 commit into
youssofal:mainfrom
PhilipJohnBasile:feat/expert-residency

Conversation

@PhilipJohnBasile

Copy link
Copy Markdown
Contributor

Splits the expert warm-set controller out of #336.

This PR contains only the expert-residency system:

  • byte-bounded expert selection from weighted locality evidence
  • hysteresis and decayed scores
  • bounded prefetch and eviction per tick
  • explicit safe-point application
  • backend capability protocol
  • generic MLX materialization backend labelled materialize_only
  • a standalone no-model planner measurement

It does not include router instrumentation, server wiring, unified-memory coordination, OTLP, policy hooks, replay, dashboard code, or workflows.

Measurement on commit dc4d939, macOS 27.0 arm64, Python 3.14.6, 40 layers, 256 experts per layer, 10,240 tracked experts, 1 GiB budget, and 100 plans:

Metric Result
Planner mean 5.668 ms
Planner p50 5.535 ms
Planner p95 7.622 ms
Planner p99 7.875 ms
Target warm set 512 experts
Target bytes 1,073,741,824
Apply time 0.019 ms
Prefetches in bounded apply 8

Reproduce:

python scripts/bench_expert_residency.py \
  --layers 40 \
  --experts 256 \
  --events 1000000 \
  --plans 100 \
  --output /tmp/expert-residency.json

Validation:

python -m pytest tests/test_expert_residency.py
8 passed in 0.02s

python -m compileall -q mtplx/expert_residency.py tests/test_expert_residency.py scripts/bench_expert_residency.py
PASS

ruff check mtplx/expert_residency.py tests/test_expert_residency.py scripts/bench_expert_residency.py
PASS

git diff --check
PASS

This is a planner and control-path measurement against a synthetic backend. It does not claim physical page residency, reduced page faults, or serving throughput improvement. Live locality wiring and real-model A/B evidence remain separate work.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant