Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 40 additions & 15 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -55,29 +55,47 @@ The existing `xpyd.proxy:main` CLI entry point remains supported.
configuration-only commands. `load_config()` returns a `ProxyConfig` when a
server should start, or `None` when a configuration command completes without
starting one. It does not import or construct the proxy runtime.
- `xpyd/app.py` exposes `create_app(config, proxy, health_monitor=None)`. It
assembles FastAPI around an existing proxy, mounts its router, and owns CORS,
readiness checks, and the discovery/health-monitor/ZMQ-listener lifecycle.
Constructing the app does not start a network listener or background services.
- `xpyd/app.py` exposes
`create_app(router, *, discovery, health_monitor=None, zmq_notifications=None)`.
It mounts the supplied router and owns CORS, readiness checks, and the
background-service lifecycle. It does not receive a `Proxy`, create runtime
dependencies, or write state back to a coordinator.
- `ProxyServer` constructs discovery, the optional health monitor and ZMQ
listener, and shares the same listener with completion execution and app
lifecycle management. `ProxyServer.create_app()` wires the application without
starting a network listener or background services.

Backends may be offline at startup. Discovery starts in the background, inference
returns 503 until ready, and health/status/metrics/admin endpoints remain
accessible.

### Endpoint Modules (`xpyd/routes/`)

Each logical endpoint has a module with `register(router, server)`.
Each logical endpoint has a module with `register(router, *, ...)`, taking only
the callbacks or data that endpoint needs.
`routes/__init__.py` registers them all. URL aliases share their logical module:
`score.py` handles `/score` and `/v1/score`, `rerank.py` handles all three rerank
paths, and `health.py` handles `/health` and `/ping`. `status.py` serves both
`/status` and `/status/instances`; `instances.py` handles addition and removal.
Existing OPTIONS handlers stay beside their endpoints.

`completions.py` and `chat_completions.py` are separate HTTP entry points. Both
delegate to `xpyd/completion_handler.py`, which contains shared validation,
request preparation, aggregated/P-D execution, response conversion, first-token
generators, metrics, and reservation cleanup. Passthrough endpoints reuse
`Proxy.post_to_instance()`.
`completions.py` and `chat_completions.py` are separate HTTP entry points. Each
receives an async `execute(request)` callback and an optional custom handler.
Passthrough endpoints receive a `forward(request, path, template)` callback;
health/version endpoints receive a fetch callback. Model/status endpoints use
query callbacks and the relevant live membership lists. Instance administration
receives only add/remove callbacks; metrics needs no injected runtime object.

`xpyd/completion_handler.py` takes explicit keyword-only operations for scheduling,
forwarding, tokenization, topology lookup, model lookup and outcome recording,
plus `CompletionOptions` and an optional notification interface. Options contain
only transfer configuration, not service objects. The aggregated sub-handler
receives only scheduling, forwarding, model lookup and outcome callbacks; pure
prompt helpers receive only a token-counting function or tokenizer.

`xpyd/interfaces.py` defines narrow callable/tokenizer/notification contracts.
There is no `ProxyProtocol`, generic service container, or all-purpose context
passed through request handling.

Runtime imports are one-way:

Expand All @@ -86,13 +104,20 @@ proxy -> cli -> config
proxy -> app -> discovery / health_monitor
proxy -> routes -> completion_handler
proxy -> completion_handler
routes / completion_handler -> interfaces
```

App assembly, routes, and completion helpers receive the proxy object rather
than importing its implementation at runtime. References to `Proxy` for type
annotations are guarded by `TYPE_CHECKING`. Completion helpers do not import
routes or app assembly. Resource ownership during streaming is unchanged:
request completion and reservation release remain tied to response consumption.
Only the composition code in `proxy.py` selects concrete implementations and
binds endpoint-specific completion callbacks. It currently supplies bound
runtime methods, but the receiving modules depend only on the declared callable
contracts: independent functions or alternative services can be supplied instead.
Routes, app assembly, and completion helpers neither receive a `Proxy`/server
object nor import that implementation, including for type annotations.

Completion helpers do not import routes or app assembly. Resource ownership
during streaming is unchanged: request completion and reservation release remain
tied to response consumption. Membership lists remain live so runtime
addition/removal is reflected in status responses.

### Scheduling Policies

Expand Down
12 changes: 10 additions & 2 deletions tests/unit/routes/test_admin_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,8 +16,16 @@


def register(router, server):
instances.register(router, server)
status.register(router, server)
instances.register(
router,
add_instance=server.add_instance,
remove_instance=server.drain_and_remove_instance,
)
status.register(
router,
prefill_instances=server.prefill_instances,
decode_instances=server.decode_instances,
)


API_KEY = "s3cr3t"
Expand Down
55 changes: 29 additions & 26 deletions tests/unit/routes/test_completion_routes.py
Original file line number Diff line number Diff line change
@@ -1,27 +1,23 @@
# SPDX-License-Identifier: Apache-2.0
"""Completion entry points share execution without coupling route modules."""

from types import SimpleNamespace
from unittest.mock import AsyncMock, patch
from unittest.mock import AsyncMock

import pytest
from fastapi import Request
from fastapi import APIRouter, FastAPI, Request
from fastapi.responses import JSONResponse
from fastapi.testclient import TestClient

from xpyd.routes import chat_completions, completions

from ._helpers import build_app


@pytest.fixture(
params=[
(completions, "/v1/completions", False, "custom_create_completion"),
(completions, "/v1/completions", False),
(
chat_completions,
"/v1/chat/completions",
True,
"custom_create_chat_completion",
),
],
ids=["completions", "chat-completions"],
Expand All @@ -31,42 +27,48 @@ def endpoint(request):


def test_default_entry_delegates_to_shared_handler(endpoint):
module, path, is_chat, _ = endpoint
server = SimpleNamespace(
custom_create_completion=None,
custom_create_chat_completion=None,
)
module, path, is_chat = endpoint
handler = AsyncMock(return_value=JSONResponse({"shared": True}, status_code=202))
with patch.object(module, "handle_completion", handler):
client = TestClient(build_app(module.register, server))
response = client.post(path, json={"model": "example"})
router = APIRouter()
module.register(router, execute=handler)
app = FastAPI()
app.include_router(router)
client = TestClient(app)
response = client.post(path, json={"model": "example"})

assert response.status_code == 202
assert response.json() == {"shared": True}
handler.assert_awaited_once()
assert handler.call_args.args[0] == path
assert isinstance(handler.call_args.args[1], Request)
assert handler.call_args.args[2] is server
assert handler.call_args.kwargs == {"is_chat": is_chat}
assert len(handler.call_args.args) == 1
assert isinstance(handler.call_args.args[0], Request)
assert handler.call_args.args[0].url.path == path
assert handler.call_args.kwargs == {}
assert client.options(path).status_code == 200
other_path = "/v1/completions" if is_chat else "/v1/chat/completions"
assert client.post(other_path, json={}).status_code == 404


def test_custom_handler_keeps_content_type_validation(endpoint):
module, path, _, custom_attribute = endpoint
@pytest.mark.parametrize("synchronous", [False, True])
def test_custom_handler_keeps_content_type_validation(endpoint, synchronous):
module, path, _ = endpoint
calls = []

async def custom(request: Request):
calls.append(await request.json())
return JSONResponse({"custom": True})

server = SimpleNamespace(
custom_create_completion=None,
custom_create_chat_completion=None,
def sync_custom(request: Request):
calls.append({"model": "example"})
return JSONResponse({"custom": True})

execute = AsyncMock()
router = APIRouter()
module.register(
router, execute=execute, custom_handler=sync_custom if synchronous else custom
)
setattr(server, custom_attribute, custom)
client = TestClient(build_app(module.register, server))
app = FastAPI()
app.include_router(router)
client = TestClient(app)

assert (
client.post(
Expand All @@ -79,3 +81,4 @@ async def custom(request: Request):
assert response.status_code == 200
assert response.json() == {"custom": True}
assert calls == [{"model": "example"}]
execute.assert_not_called()
2 changes: 1 addition & 1 deletion tests/unit/routes/test_forward_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ def register(router, server):
rerank,
invocations,
):
module.register(router, server)
module.register(router, forward=server.post_to_instance)


# (route, forwarded path, expected json template, a valid request body)
Expand Down
14 changes: 12 additions & 2 deletions tests/unit/routes/test_health_routes.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,18 @@


def register(router, server):
for module in (health, models, version, metrics):
module.register(router, server)
health.register(router, fetch=server.get_from_instance)
version.register(router, fetch=server.get_from_instance)
models.register(
router,
fetch=server.get_from_instance,
get_registered_models=(
server.registry.get_registered_models
if server.registry is not None
else None
),
)
metrics.register(router)


class _HealthServer:
Expand Down
38 changes: 34 additions & 4 deletions tests/unit/test_aggregated_completion.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@

import pytest

from xpyd.completion_handler import handle_completion
from xpyd.completion_handler import CompletionOptions, handle_completion
from xpyd.proxy import Proxy
from xpyd.registry import InstanceRegistry
from xpyd.scheduler import SchedulingContext, default_registry
Expand All @@ -32,6 +32,21 @@ def make_proxy(strategy="roundrobin", override=None, registry=True):
)


def completion_arguments(proxy):
return {
"options": CompletionOptions(),
"reserve": proxy.reserve,
"forward_request": proxy.forward_request,
"count_tokens": proxy.get_total_token_length,
"get_tokenizer": proxy.get_tokenizer,
"tokenize_on_backend": proxy.tokenize_on_backend,
"is_aggregated_model": proxy._is_aggregated_model,
"get_registered_models": proxy.registry.get_registered_models,
"record_success": proxy.registry.record_success,
"record_failure": proxy.registry.record_failure,
}


def test_model_topology_detection():
proxy = make_proxy()
assert proxy._is_aggregated_model(MODEL)
Expand Down Expand Up @@ -152,7 +167,12 @@ async def forward(*args, **kwargs):
patch.object(proxy.registry, "record_success") as success,
patch.object(proxy.registry, "record_failure") as failed,
):
response = await handle_completion("/v1/completions", request, proxy, False)
response = await handle_completion(
request,
endpoint="/v1/completions",
is_chat=False,
**completion_arguments(proxy),
)
if failure is RuntimeError:
with pytest.raises(RuntimeError):
_ = [chunk async for chunk in response.body_iterator]
Expand All @@ -176,7 +196,12 @@ async def forward(*args, **kwargs):
proxy.forward_request = forward
with patch("xpyd.completion_handler.track_request_end") as ended:
with pytest.raises(asyncio.CancelledError):
await handle_completion("/v1/completions", request, proxy, False)
await handle_completion(
request,
endpoint="/v1/completions",
is_chat=False,
**completion_arguments(proxy),
)
ended.assert_called_once()
assert all(proxy.registry.get_active_requests(address) == 0 for address in NODES)

Expand All @@ -195,7 +220,12 @@ async def forward(*args, **kwargs):

proxy.forward_request = forward
with patch.object(proxy.registry, "record_success") as success:
response = await handle_completion("/v1/completions", request, proxy, False)
response = await handle_completion(
request,
endpoint="/v1/completions",
is_chat=False,
**completion_arguments(proxy),
)
await response.body_iterator.__anext__()
await response.body_iterator.aclose()
success.assert_not_called()
Expand Down
Loading
Loading