Skip to content

feat(runtime): add opt-in expert locality telemetry - #355

Open
PhilipJohnBasile wants to merge 1 commit into
youssofal:mainfrom
PhilipJohnBasile:feat/expert-locality-telemetry
Open

feat(runtime): add opt-in expert locality telemetry#355
PhilipJohnBasile wants to merge 1 commit into
youssofal:mainfrom
PhilipJohnBasile:feat/expert-locality-telemetry

Conversation

@PhilipJohnBasile

Copy link
Copy Markdown
Contributor

This extracts only expert-routing locality telemetry from #336.

Change

  • Adds an opt-in, bounded tracker for per-layer and per-lane expert counts, working-set coverage, reuse distance, consecutive overlap, and simulated LRU hit rates.
  • Installs read-only switch_mlp taps on the model owner thread at server startup when MTPLX_EXPERT_LOCALITY=1.
  • Leaves the disabled path untouched because no wrapper is installed unless explicitly enabled.
  • Defaults enabled collection to one sample per 16 router calls. The interval and event bound remain configurable.
  • Adds focused tests, operator documentation, a reproducible no-model measurement script, and the raw receipt.

Measurement

Command:

python3 scripts/bench_expert_locality.py --iterations 50000 --repeats 7 --sample-every 16

Apple M5 Max, 128 GB, macOS 27.0, Python 3.14.7. Seven repeats of 50,000 calls each, using a rotating eight-expert top-2 route pattern:

Arm Median per call Added time per call Accepted samples WS90
Unwrapped baseline 0.068 us 0 0 n/a
Sample every 16 1.199 us 1.130 us 21,875 8
Sample every call 3.474 us 3.405 us 350,000 8

Output parity was true in both instrumented arms. The committed receipt is docs/perf/receipts/expert-locality-no-model-20260826.json.

This is a CPU-only measurement of the installed Python router tap using list inputs. It does not claim model throughput, Metal timing, or MLX array materialization cost. A model-backed throughput measurement should be a separate follow-up because this PR makes no routing or residency change.

Validation

  • tests/test_expert_locality.py: 18 passed
  • Python compilation: passed for the module, server integration, tests, and measurement script
  • Ruff: passed for all new Python files
  • Server integration lint: passed with upstream pre-existing F841 excluded
  • Ruff format check: passed for all new Python files
  • git diff --check: passed
  • U+2013 and U+2014 scan across every changed file: no matches

Exclusions

No semantic anchors, memory governor, request capture, deterministic replay, expert residency, unified-memory coordination, OTLP export, policy hooks, dashboard UI, workflow changes, or generated dashboard assets are included.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant