Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,12 @@ reservations. Policy-specific token/KV accounting stays inside each subclass.
Aggregated and P/D request paths share this boundary; see
[Scheduling Policies](scheduling.md#common-policy-contract).

Policies do not hold or import the instance registry. Capacity-aware policies
declare their data requirement; the proxy composition layer queries node model
lengths and injects a mapping into their factory. Strategy constructors perform
no backend I/O. Runtime filtering remains the only scheduling-layer consumer
of registry health and active-request state.

## Request Flow

```
Expand Down
35 changes: 34 additions & 1 deletion docs/scheduling.md
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,12 @@ minimizes active requests, breaking busy-node ties using in-flight prompt tokens
P/D candidates must also satisfy the model-length limit. Reservations release
these counts when a request completes, fails, or is cancelled.

The policy does not query backends. At startup the proxy composition layer
queries P/D node capacities and passes an address-to-`max_model_len` mapping
into the policy. Unreachable nodes retain the existing logged fallback of
131072, so proxy-first startup remains supported. Dynamically added nodes
receive their validated capacity through `on_instance_added()`.

### Characteristics

- **Load-aware** — naturally adapts to heterogeneous instance performance.
Expand Down Expand Up @@ -215,6 +221,32 @@ policy and context, so changing a model's tokenizer fallback cannot release
load against the wrong policy. Releasing a reservation does not record success
or failure; the request executor handles outcomes separately.

Only the runtime holds an `InstanceRegistry`; policies and their base class
neither accept nor store one. `from_config()` receives topology data,
`model_lengths`, tokenizer input and strategy options, not live registry state.
Policies declare `requires_model_lengths = True` when they need startup
capacities. The composition layer inspects that declaration via
`PolicyRegistry.get_policy_class()`, queries capacities only for those policies,
then uses the common factory. Aliases and custom subclasses follow the same
path without a branch on a concrete policy name or type.

For direct Python construction, capacity-aware P/D policies require supplied
capacities; missing entries are errors rather than implicit network queries:

```python
from xpyd.scheduler import LoadBalancedScheduler

policy = LoadBalancedScheduler(
["prefill:8100"],
["decode:8200"],
model_lengths={"prefill:8100": 8192, "decode:8200": 16384},
)
```

The mapping is read by address, not by iteration order. Node membership lists
retain their shared lifecycle semantics; capacity values are copied into policy
state. Aggregated-only load balancing needs no P/D capacity data.

The proxy uses this same contract for aggregated and P/D requests. It does not
branch on concrete strategy types. Request code calls `Proxy.reserve(context)`
and retains the returned reservation until it can call `release()`.
Expand All @@ -232,7 +264,8 @@ power-of-two counters.

Implement `SchedulingPolicy.select_node()` and any required lifecycle hooks.
Override `from_config()` if the constructor needs special arguments; otherwise
the default factory accepts `workers`, `registry`, and the configured options.
the default factory passes `workers` and the configured options to the constructor.
Override it to consume additional common inputs such as `model_lengths`.
Register the class before loading YAML:

```python
Expand Down
16 changes: 6 additions & 10 deletions tests/unit/routes/test_instance_removal.py
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ def _proxy(registry: InstanceRegistry) -> Proxy:
decode_instances=["127.0.0.1:8200", "127.0.0.1:8201"],
model="test/model",
registry=registry,
scheduling_policy=RoundRobinSchedulingPolicy(registry=registry),
scheduling_policy=RoundRobinSchedulingPolicy(),
)
proxy.health_monitor = MagicMock()
proxy.discovery = MagicMock()
Expand Down Expand Up @@ -170,15 +170,11 @@ async def test_load_balanced_scheduler_can_select_runtime_instance() -> None:
registry.add("decode", "127.0.0.1:8200", model="test/model")
registry.mark_healthy("127.0.0.1:8100")
registry.mark_healthy("127.0.0.1:8200")
with patch(
"xpyd.scheduler.load_balanced.query_instance_model_len",
side_effect=lambda instances: [4096] * len(instances),
):
policy = LoadBalancedScheduler(
["127.0.0.1:8100"],
["127.0.0.1:8200"],
registry=registry,
)
policy = LoadBalancedScheduler(
["127.0.0.1:8100"],
["127.0.0.1:8200"],
model_lengths={"127.0.0.1:8100": 4096, "127.0.0.1:8200": 4096},
)
proxy = _proxy(registry)
proxy.scheduling_policy = policy
proxy.prefill_instances = policy.prefill_instances
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/scheduler/test_cache_aware.py
Original file line number Diff line number Diff line change
Expand Up @@ -260,7 +260,7 @@ def test_schedule_does_not_fall_back_to_a_draining_worker(self):
registry.add("decode", "w1")
registry.mark_healthy("w1")
registry.begin_draining("decode", "w1")
policy = CacheAwarePolicy(workers=["w1"], registry=registry)
policy = CacheAwarePolicy(workers=["w1"])

selected = Scheduler(registry).reserve(
policy,
Expand Down
2 changes: 1 addition & 1 deletion tests/unit/scheduler/test_consistent_hash.py
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ def test_schedule_does_not_fall_back_to_a_draining_worker(self):
registry.add("decode", "w1")
registry.mark_healthy("w1")
registry.begin_draining("decode", "w1")
policy = ConsistentHashPolicy(workers=["w1"], registry=registry)
policy = ConsistentHashPolicy(workers=["w1"])

selected = Scheduler(registry).reserve(
policy,
Expand Down
41 changes: 37 additions & 4 deletions tests/unit/scheduler/test_load_balanced_deep.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,11 +11,44 @@

@pytest.fixture
def policy():
return LoadBalancedScheduler(
["p1", "p2"],
["d1", "d2"],
model_lengths={address: 4096 for address in ("p1", "p2", "d1", "d2")},
)


@pytest.mark.parametrize("role", ["prefill", "decode"])
def test_injected_capacities_are_matched_by_address_without_backend_io(role):
capacities = {"large": 8192, "small": 1024}
nodes = ["small", "large"]
with patch(
"xpyd.scheduler.load_balanced.query_instance_model_len",
side_effect=lambda nodes: [4096] * len(nodes),
):
return LoadBalancedScheduler(["p1", "p2"], ["d1", "d2"])
"requests.get", side_effect=AssertionError("Policy performed HTTP I/O")
) as query:
policy = LoadBalancedScheduler(
nodes if role == "prefill" else [],
nodes if role == "decode" else [],
model_lengths=capacities,
)
capacities["small"] = 8192
lease = Scheduler().reserve(
policy,
SchedulingContext(role=role, request_len=1024, max_tokens=1),
nodes,
)
query.assert_not_called()
assert lease.address == "large"
lease.release()


@pytest.mark.parametrize("role", ["prefill", "decode"])
def test_missing_capacity_is_rejected(role):
with pytest.raises(ValueError, match="Missing model lengths.*missing"):
LoadBalancedScheduler(
["missing"] if role == "prefill" else [],
["missing"] if role == "decode" else [],
model_lengths={},
)


@pytest.mark.parametrize("role", ["prefill", "decode"])
Expand Down
21 changes: 8 additions & 13 deletions tests/unit/scheduler/test_policy_registry.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,8 +10,8 @@
class _DummyPolicy(SchedulingPolicy):
"""Minimal concrete policy for testing."""

def __init__(self, workers=None, registry=None, **kwargs):
super().__init__(registry=registry, workers=workers)
def __init__(self, workers=None, **kwargs):
super().__init__(workers=workers)
self.kwargs = kwargs

def select_node(self, context, candidates):
Expand Down Expand Up @@ -78,18 +78,13 @@ def test_duplicate_register_warns(self, caplog):

def test_build_returns_correct_type(self):
"""build() returns an instance of the registered class."""
from unittest.mock import patch

rr = default_registry.build("roundrobin")
assert isinstance(rr, SchedulingPolicy)

with patch(
"xpyd.scheduler.load_balanced.query_instance_model_len",
return_value=[131072],
):
lb = default_registry.build(
"loadbalanced",
prefill_instances=["p1"],
decode_instances=["d1"],
)
lb = default_registry.build(
"loadbalanced",
prefill_instances=["p1"],
decode_instances=["d1"],
model_lengths={"p1": 131072, "d1": 131072},
)
assert isinstance(lb, SchedulingPolicy)
32 changes: 12 additions & 20 deletions tests/unit/scheduler/test_scheduler.py
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
"""Policy selection and runtime eligibility are separate contracts."""

from unittest.mock import patch

import pytest

from xpyd.registry import InstanceRegistry
Expand Down Expand Up @@ -76,15 +74,12 @@ def test_runtime_filters_unhealthy(strategy, role, healthy):
registry.add(role, node)
for node in healthy:
registry.mark_healthy(node)
with patch(
"xpyd.scheduler.load_balanced.query_instance_model_len",
side_effect=lambda instances: [4096] * len(instances),
):
policy = default_registry.build(
strategy,
prefill_instances=nodes if role == "prefill" else [],
decode_instances=nodes if role == "decode" else [],
)
policy = default_registry.build(
strategy,
prefill_instances=nodes if role == "prefill" else [],
decode_instances=nodes if role == "decode" else [],
model_lengths={address: 4096 for address in nodes},
)
runtime = Scheduler(registry)
context = SchedulingContext(role=role, request_len=100, max_tokens=10)
leases = [runtime.reserve(policy, context, nodes) for _ in range(6)]
Expand All @@ -101,15 +96,12 @@ def test_runtime_filters_unhealthy(strategy, role, healthy):
@pytest.mark.parametrize("role", ["prefill", "decode"])
def test_runtime_without_registry_uses_explicit_candidates(strategy, role):
nodes = ["a", "b", "c"]
with patch(
"xpyd.scheduler.load_balanced.query_instance_model_len",
side_effect=lambda instances: [4096] * len(instances),
):
policy = default_registry.build(
strategy,
prefill_instances=nodes if role == "prefill" else [],
decode_instances=nodes if role == "decode" else [],
)
policy = default_registry.build(
strategy,
prefill_instances=nodes if role == "prefill" else [],
decode_instances=nodes if role == "decode" else [],
model_lengths={address: 4096 for address in nodes},
)
runtime = Scheduler()
context = SchedulingContext(role=role, request_len=100, max_tokens=1)
leases = [runtime.reserve(policy, context, nodes) for _ in nodes]
Expand Down
Loading
Loading