[1/3 entrypoint refactor] [feat(mesh)]: unify multi-API ingress for HTTP and ext-proc

[Yuechguo]

Motivation

Mesh is the request frontend in front of ATOM workers, but before this PR it only served OpenAI Chat/Completions well:

• Anthropic Messages and OpenAI Responses were not first-class. They had no shared routing with HTTP and ext-proc, Responses resource operations (GET/DELETE /v1/responses/{id}, input items, cancel) were not forwarded, and Messages/Responses SSE streams were not accounted correctly.
• Request preparation ran on the async runtime. JSON parsing and tokenization (needed for cache-aware / prefix routing) ran on Tokio workers. A few long prompts could stall I/O for every other connection, and nothing bounded or measured preparation work.
• HTTP and ext-proc deployments diverged. The standalone HTTP router and the Envoy ext-proc path each had their own parsing, routing and error handling. The same request could be routed or rejected differently depending on how Mesh was deployed, and every fix had to be made twice.

This PR unifies the Chat, Completions, Messages and Responses APIs behind one ingress shared by HTTP, PD and ext-proc, and moves preparation onto a bounded, observable thread pool. The goal is a frontend that costs less CPU and adds less tail latency per request than a Python (FastAPI) frontend, in either deployment mode.

Technical Details

1. One ingress for every API and deployment mode

• Single endpoint registry. Each inference API is declared once, with its path and sub-resources:

  • /v1/chat/completions
  • /v1/messages
  • /v1/responses, plus /{id}, /{id}/cancel and /{id}/input_items
  • /v1/completions
  • /generate

  The standalone HTTP router, the PD router and the Envoy ext-proc path all resolve requests through this registry, so they parse, validate and reject requests identically.

• Capability-aware placement. Each request is checked against the deployment once (API, topology, model).

  • Only workers that serve the API (the mesh.apis label) and match the topology are offered to the load-balancing policy.
  • When no configured backend can serve the request, Mesh answers 501 unsupported_api instead of forwarding to a backend that would fail.

• Responses state safety. Some requests reference state stored on a backend: previous_response_id, conversation, or item_reference input items (including items with no type). They are routed only when every candidate shares one backend, or all candidates are labeled mesh.responses_state=shared. Otherwise Mesh returns 501 stateful_routing_unsupported. Unhealthy workers count in this check, so failover cannot silently move a conversation to a backend that does not hold its state.

• Responses resource operations. Retrieve, delete, cancel and input-item listing are forwarded over HTTP with the original query string and the worker's credentials. DP ranks that share one backend address are queried once. Resource IDs are limited to [A-Za-z0-9_-], at most 256 bytes, so a decoded ID cannot change the forwarded path or query.

• Payload pass-through. Mesh parses a request only to build a routing view, and forwards the client's original bytes.

  • For DP-aware workers, a body without data_parallel_rank is built once during preparation, preserving key order. Each attempt only appends the selected worker's rank, so retries never reparse or reserialize the body.
  • Messages and Responses receive the rank only when the worker declares support for them in mesh.apis.

• Credential isolation. When the selected worker has an API key, the client's Authorization and x-api-key headers are dropped and the worker key is sent instead, in regular and PD dispatch alike. Messages requests carry the worker key in both headers, because Anthropic clients authenticate with x-api-key.

• Response accounting.

  • The worker's load count is held until the response body finishes.
  • Non-streaming responses with a known length are now recorded as completed instead of canceled.
  • For Messages and Responses SSE:
    • refusal deltas count as the first output token for TTFT;
    • error events end the stream;
    • Anthropic cache tokens are included in input usage.

• ext-proc parity. ext-proc shares the registry, placement, DP rank handling and error mapping with HTTP. Responses resource operations, which need a direct backend call, return 501 unsupported_in_ext_proc.

HTTP ingress and ext-proc share endpoint definitions, request validation, routing metadata preparation, and backend eligibility checks, while retaining their respective execution and response handling paths. Standalone currently uses the common HTTP parsing path with typed dispatch; broader reuse by the ATOM layer is planned after the standalone refactoring and is outside the scope of this PR.

2. Bounded preparation pool (core/prepare_pool/)

• Off the I/O threads. These steps run on a dedicated pool of OS threads, so a long prompt no longer stalls I/O for other connections:

  • JSON parsing
  • chat-template rendering
  • extracting routing text
  • tokenization, including /v1/tokenize and /v1/detokenize

  When the routing policy needs text or tokens (for example cache-aware or prefix-hash routing), they are produced in the same job as the parse, so each request enters the pool once.

• Admission control.

  • Requests wait asynchronously for a free worker, and at most --prepare-queue-capacity may wait. Beyond that, Mesh rejects immediately with 503 prepare_queue_full.
  • Waiting and running share one deadline (--prepare-queue-timeout-ms, --prepare-timeout-ms), and exceeding it returns 504. The deadline applies only to preparation: a request whose preparation finished is always forwarded.
  • Inputs larger than --prepare-max-tokenize-bytes are rejected with 413.

• Cancellation. If the client disconnects or the deadline passes before a worker picks up a job, the job is cancelled and never runs. Every job outcome (success, timeout, cancellation, failure) is counted exactly once.

• Shutdown and health.

  • On shutdown the pool stops accepting work after connections drain. It then waits a bounded time for running jobs, so one stuck job cannot block exit.
  • If the pool fails, new requests are rejected and /liveness and /health report unhealthy.

• Observability. New Prometheus metrics:

  • Queue depth and worker gauges: mesh_prepare_queued_jobs, mesh_prepare_dispatched_jobs, mesh_prepare_execution_reserved, mesh_prepare_busy_workers, mesh_prepare_live_workers
  • Latency histograms: mesh_prepare_queue_seconds, mesh_prepare_run_seconds, mesh_prepare_stage_seconds
  • Errors by reason: mesh_prepare_errors_total

• CPU budget. By default, half of the available CPUs run preparation (--prepare-workers, minimum 1) and the other half serve I/O, unless TOKIO_WORKER_THREADS is set. The older ext-proc parser settings map onto the same pool.

3. Faster tokenization

• Tokenizer copies per worker group. A HuggingFace BPE tokenizer guards its word cache with a single lock. Shared by 16 threads, one instance loses about a third of its throughput. Every 4 preparation workers now share their own copy.
  • This matches the throughput of one copy per thread at a quarter of the memory (about 60 MiB per copy for a 150k vocabulary).
  • Copies follow tokenizer re-registration and removal.
• Log filtering. Tokenizer-internal trace logs were dispatched even when the log level was higher. Filtering them at the configured level cuts a 16 KiB encode from ~4.7 ms to ~3.3 ms.
• Build. The tokenizer dependency is built with full optimization (opt-level = 3); the rest of the binary keeps the size-optimized release profile.

Benchmark: Mesh vs. a Python frontend

Question. How much CPU and latency does Mesh spend per request compared with a typical Python serving frontend? Both are measured on the same cores with the same requests and the same backends.

Setup

	Mesh (this PR)	Python baseline
Server	atomesh at 237714bd2	FastAPI 0.115.6 + uvicorn 0.34 (uvloop, httptools)
Processes / threads	1 process. Proxy: 8 Tokio + 4 prepare. Tokenize: 4 Tokio + 8 prepare	8 uvicorn worker processes
Request handling	Parse and tokenize in the preparation pool; forward the original bytes	pydantic validation; tokenize via run_in_executor; aiohttp forwards the original bytes
Tokenizer	Qwen2.5-7B-Instruct, tokenizers 0.22.2, one copy per 4 prepare workers	Same file, tokenizers 0.21.0 (Python binding), one per process
Routing	round robin (proxy), prefix hash over 256 tokens (tokenize)	same
Build	opt-level=3, fat LTO, target-cpu=native via env overrides; repo profile unchanged	generic x86-64 wheels

Method

• Hardware. Intel Xeon Platinum 8480C (2 × 56 cores, SMT).
  • Each frontend is pinned with taskset to the same 8 physical cores (SMT siblings excluded).
  • Eight mock backends (atomesh-mocker virtual workers returning a fixed JSON body) use 18 other logical CPUs.
  • The load generator uses another 32.
• Workloads.
  • Parse + proxy: no tokenization; measures pure frontend overhead.
  • Parse + tokenize + proxy: prefix-hash routing on the first 256 tokens, so every request is tokenized.
• Requests. Deterministic /v1/completions prompts of 1 KiB, 16 KiB, 64 KiB, 128 KiB, 512 KiB and 1 MiB.
• Load. Closed loop with 1, 4, 16, 64 and 256 concurrent clients, 8 s per point, after a warm-up. Mesh and FastAPI alternate at each point, in alternating order, for 2 rounds; the median is reported.
• Limits. Mesh queues are effectively unbounded (--queue-size 100000, --prepare-queue-capacity 100000, 120 s preparation deadlines) to match uvicorn/asyncio. --prepare-max-tokenize-bytes is raised to 8 MiB so 1 MiB prompts are accepted.
• Metrics.
  • Throughput is successful req/s.
  • CPU time per request is utime + stime of the frontend process tree divided by successful requests. In the detailed tables, "CPU per request, FastAPI / Mesh" = 2.00× means FastAPI spends twice the CPU per request.
  • Latency uses fine buckets up to 10 ms and coarse buckets above (20, 50, 100, 200, 500 ms, 1 s, …), so p99 values above 10 ms are bucket edges.

Summary. Mesh is faster at all 60 measured points.

Workload	Throughput (higher is better)	Mean latency (lower is better)	CPU time per request (lower is better)
Parse + proxy	Mesh is 1.46–1.93× FastAPI (median 1.59×)	Mesh is 33–50% lower (median 39%)	Mesh uses 38–63% less (median 50%)
Parse + tokenize + proxy	Mesh is 1.36–2.51× FastAPI (median 1.57×)	Mesh is 22–62% lower (median 36%)	Mesh uses 29–57% less (median 36%)

Ranges exclude points marked † or ‡ in the detailed tables.

Workload	Size	Clients	Mesh req/s	FastAPI req/s	Mesh p99	FastAPI p99
Proxy	1 KiB	256	57,588	38,046	5.0 ms	35 ms
Proxy	16 KiB	256	53,810	34,492	5.8 ms	50 ms
Proxy	128 KiB	16	22,641	12,355	2.0 ms	2.7 ms
Tokenize	1 KiB	16	21,511	8,561	1.7 ms	4.0 ms
Tokenize	128 KiB	256	269	160	1.0 s	7.5 s

Throughput ratio

Each cell is Mesh throughput divided by FastAPI throughput; blue means Mesh is faster.

Latency vs. throughput (Pareto)

Both axes are logarithmic and lower-right is better. Each line runs from 1 to 256 concurrent clients.

Parse + proxy, p99 latency

Parse + tokenize + proxy, p99 latency

Mean latency vs. throughput

Detailed results

All 60 points (median of 2 rounds)

Parse + proxy (Mesh: 8 Tokio + 4 prepare threads)

Size	Clients	Mesh req/s	FastAPI req/s	Mesh / FastAPI	Mesh p99 ms	FastAPI p99 ms	Mesh mean ms	FastAPI mean ms	Req per CPU-s, Mesh / FastAPI
1 KiB	1	6,900	3,891	1.77×	0.16	0.29	0.14	0.25	2.35×
	4	22,898	13,987	1.64×	0.22	0.38	0.17	0.28	1.88×
	16	53,608	35,335	1.52×	0.59	0.76	0.29	0.45	1.68×
	64	60,009	38,164	1.57×	1.64	2.25	1.06	1.68	1.60×
	256	57,588	38,046	1.51×	4.98	35.0	4.42	6.70	1.63×
16 KiB	1	5,851	3,359	1.74×	0.19	0.33	0.16	0.29	2.40×
	4	20,120	13,268	1.52×	0.25	0.36	0.19	0.29	1.98×
	16	50,697	30,849	1.64×	0.58	0.83	0.30	0.51	1.73×
	64	54,646	34,793	1.57×	1.80	2.98	1.16	1.83	1.68×
	256	53,810	34,492	1.56×	5.75	50.0	4.73	7.39	1.73×
64 KiB	1	3,612	2,155	1.68×	0.36	0.60	0.27	0.45	2.46×
	4	12,509	7,895	1.58×	0.42	0.69	0.31	0.50	2.05×
	16	33,090	18,798	1.76×	1.18	1.48	0.47	0.87	1.89×
	64	44,809	23,212	1.93×	2.17	5.47	1.36	2.74	1.97×
	256	42,873	24,584	1.74×	13.8	50.0	5.66	10.3	1.87×
128 KiB	1	2,409	1,569	1.54×	0.54	0.96	0.40	0.63	2.32×
	4	8,475	5,590	1.52×	0.66	1.04	0.46	0.71	1.94×
	16	22,641	12,355	1.83×	1.98	2.68	0.69	1.28	2.09×
	64	28,951	17,047	≥1.70× †	5.49	5.02	0.93	3.78	2.04×
	256	29,298	18,532	≥1.58× †	20.0	50.0	1.57	13.7	2.06×
512 KiB	1	833	536	1.55×	1.73	2.37	1.18	1.90	2.67×
	4	3,177	1,888	1.68×	1.96	3.54	1.24	2.13	2.43×
	16	8,698	5,452	1.60×	4.76	5.85	1.81	3.02	2.42×
	64	7,895	7,046	≥1.12× †	6.79	20.0	1.60	8.67	2.19×
	256	8,003	6,874	≥1.16× †	20.0	100	1.84	28.5	2.29×
1 MiB	1	412	281	1.46×	5.43	4.41	2.38	3.56	2.44×
	4	1,630	1,043	1.56×	4.08	5.58	2.40	3.79	2.42×
	16	4,340	2,346	1.85×	7.34	20.0	3.60	6.69	2.29×
	64	3,865	3,451	≥1.12× †	13.1	50.0	3.11	15.0	2.27×
	256	3,957	3,553	≥1.11× †	12.7	110	2.83	29.1	2.15×

Parse + tokenize + proxy (Mesh: 4 Tokio + 8 prepare threads)

Size	Clients	Mesh req/s	FastAPI req/s	Mesh / FastAPI	Mesh p99 ms	FastAPI p99 ms	Mesh mean ms	FastAPI mean ms	Req per CPU-s, Mesh / FastAPI
1 KiB	1	2,822	1,583	1.78×	0.40	0.74	0.35	0.62	2.01×
	4	9,658	4,266	2.26×	0.53	1.46	0.41	0.94	2.31×
	16	21,511	8,561	2.51×	1.69	3.96	0.74	1.88	2.28×
	64	22,929	12,222	1.88×	3.98	20.0	2.78	5.23	1.79×
	256	22,772	13,559	1.68×	20.0	50.0	11.2	18.8	1.61×
16 KiB	1	319	212	1.50×	3.25	5.01	3.13	4.70	1.53×
	4	1,225	696	1.76×	4.44	12.6	3.25	5.84	1.57×
	16	2,062	1,398	1.47×	20.0	50.0	7.75	11.4	1.50×
	64	2,032	1,451	1.40×	50.0	200	31.4	43.9	1.42×
	256	2,043	1,499	1.36×	200	500	124	169	1.41×
64 KiB	1	78.1	50.2	1.56×	35.0	50.0	12.8	19.9	1.57×
	4	311	125	2.49×	35.0	100	12.8	33.8	1.59×
	16	522	350	1.49×	75.0	200	30.5	45.5	1.48×
	64	527	356	1.48×	200	750	120	177	1.51×
	256	512	353	1.45×	1,000	5,000	483	674	1.53×
128 KiB	1	37.2	25.3	1.47×	50.0	50.0	26.7	39.4	1.49×
	4	147	95.7	1.54×	50.0	75.0	27.0	41.6	1.55×
	16	266	168	1.58×	150	500	59.8	94.0	1.49×
	64	269	174	1.54×	500	3,500	234	351	1.55×
	256	269	160	1.68×	1,000	7,500	892	1,180	1.72×
512 KiB	1	8.65	5.60	1.54×	350	200	114	174	1.62×
	4	32.8	20.9	1.57×	200	500	120	189	1.60×
	16	59.4	37.4	1.59×	500	2,000	263	409	1.57×
	64	59.7	34.2	1.74×	2,000	5,000	991	1,590	1.79×
	256	52.3	25.6	2.04× ‡	7,500	10,000	3,662	4,047	2.44×
1 MiB	1	3.40	2.50	1.36×	750	500	298	381	2.17×
	4	14.6	8.20	1.79×	500	1,250	264	476	1.57×
	16	27.3	17.0	1.61×	1,000	3,500	554	882	1.64×
	64	27.2	13.7	1.99× ‡	5,000	10,000	1,974	3,091	2.18×
	256	19.1	9.95	1.92× ‡	10,000	10,000	4,120	4,915	2.65×

† Load-generator bound for Mesh: the ratio is a lower bound and latencies are not comparable.
‡ Finite 8 s window: mean latency undercounts in-flight multi-second requests for one or both frontends.

[github-actions]

🏷️ CI Guide

Runs automatically on every eligible PR before approval:

• ✅ Pre Checkin: Black, Ruff, catalog schema validation, non-GPU unit tests

Heavy model tests:

• ✅ Run after the PR is approved and Pre Checkin passes
• ✅ Run immediately when an approval review is submitted
• ✅ Can be requested before approval with labels

Label	Tests
ci:full	Run all heavy PR model tests: native ATOM, vLLM, and SGLang
ci:atom	Run native ATOM model accuracy tests
ci:vllm	Run ATOM vLLM OOT model accuracy tests
ci:sglang	Run ATOM SGLang model accuracy tests

Heavy jobs are skipped when the PR is not approved and no matching ci:* label is present.
Add labels via the sidebar or gh pr edit 2453 --add-label <label>

[Jasen2201]

Thanks for the refactor. Sharing EndpointSpec / IngressRouting between HTTP and ext-proc is a clear improvement, and several fixes here are valuable: exactly-once outcome accounting in ProxyBody, no longer stripping content-encoding (reqwest is built without the gzip feature, so compressed bodies were previously forwarded without the header), preserving repeated response headers, and HTTP-mode prefix_hash now actually receiving tokens (previously always tokens: None -> PolicyReturnedNone). The new HTTP/ext-proc contract tests are good too.

Main items (details inline):

1. The PD path was not updated for the new x-api-key forwarding, so the client key now reaches P/D alongside the worker Bearer.
2. Messages/Responses routing text is serialized in alphabetical key order (BTreeMap), so cache_aware prefix matching sees messages/input before system/tools/instructions, the opposite of prompt order.
3. HTTP ingress now tokenizes synchronously on a tokio worker thread (ext-proc uses spawn_blocking with bounded slots).
4. Both READMEs link docs/mesh_multi_api_guide.md, which is not in this PR or on main; the ext-proc README also overstates Responses support (resource operations are rejected in ext-proc mode).

Questions / notes inline: the typed route_* path that is no longer reachable from the server, PD now forwarding the raw body, item_reference state detection, resource fan-out filtering, DP data_parallel_rank injection into Messages/Responses, and ingress-side schema strictness.

Note: reviewed by reading the diff and cross-checking against the PR head (9a7d7f8); I did not build or run the tests locally.

[Jasen2201]

Thanks for the quick turnaround. I checked the replies against the five new commits (78aa57d..a5c01ac; the first commit is unchanged from 9a7d7f8). These look good: the PD x-api-key fix plus its tests, the prompt-order routing view, item_reference detection, the 500 classification for prepare_request failures, and --prepare-max-tokenize-bytes.

Blocking

1. Prepare pool admission (inline on ingress.rs). Every inference request now goes through a fail-fast pool. With the defaults, more than 14 overlapping parse/prepare jobs get a 503. The total size of in-flight request bodies is also capped at 32 MiB, and that budget is held through the upstream call (in PD, through the whole prefill). Both are new failure modes on the HTTP path for long-context, high-concurrency traffic. Could the pool move to a separate PR with load data? For this PR, offloading only when needs_tokens() is true would address the original comment.
2. 503 ambiguous_response_credentials (inline on http_router.rs). This blocks Responses resource operations for the standard OpenAI and Anthropic SDKs whenever there are multiple backends without worker keys.
3. Ingress schema narrowing (reply: "Agreed, I'll narrow"). The code at a5c01ac is unchanged: max_tokens is still required (inference.rs:212), Messages temperature is still capped at 1, top_k: -1 is still rejected, and roles are still limited to user/assistant. You already confirmed these don't match ATOM, so please include the change in this PR.
4. Resource fan-out has no tests. 17f0ea7 adds mod resource_fanout_test; and a README row for it, but the file was never committed, and 2526ca5 removes both as "stale". Please add tests for the filtering, credential handling, origin dedup, and error selection.

Still open from the first round

• The ext-proc README (scripts/ext-proc/README.md:64) still says Responses works through "both ordinary HTTP and ext-proc". ext-proc still returns 405 for non-POST requests (ext_proc/request.rs:106), so GET/DELETE/cancel only work over HTTP. One sentence would fix this.
• The PR description doesn't yet have the notes promised in the replies: that the legacy typed path is removed in a follow-up, and that PD also forwards the raw body. For PD raw-body forwarding, is there a PD chat smoke or GSM8K run? The GPU ATOMesh jobs were skipped in CI.
• ENDPOINT_MESSAGES hasn't been added (metrics_utils.rs:13 still uses "messages").
• The placement-failure retry for Responses isn't implemented yet. That's fine as a follow-up if the description says so.

Nits

• There's no test that locks in the shared-prefix property of the new routing view order (same system+tools, different user turn).
• item_reference: in the OpenAI schema type is optional on item references, so an item that is only {"id": "..."} isn't detected.

Note: reviewed by reading the PR head (a5c01ac); I did not build or run the tests.

[JiaoliangYu]

Manual re-review of the latest head (c73b493f). The previous missing-test-module and Responses ID path-validation issues are fixed. I found one remaining blocking runtime issue and one documentation cleanup item. I reviewed the code manually; I could not run Rust tests locally because this machine does not have the Rust toolchain. git diff --check passes, and the GitHub non-GPU unit tests are currently green.