Skip to content

perf: restore asynchronous route publication for Owner writes - #64

Merged
Kaimary merged 1 commit into
mainfrom
perf/owner-async-route-publication
Oct 29, 2026
Merged

Kaimary merged 1 commit into
mainfrom
perf/owner-async-route-publication

Conversation

@ActivePeter

Copy link
Copy Markdown
Collaborator

External Owner-local PUTs currently wait for Master route publication after the values are already locally readable. Return success after local commit and handoff to the existing Owner publication task, removing Master publication latency from the PUT completion path used by SGLang KV cache writes.

The Owner task continues to hold pending contexts and per-key fences through publication, retry, and terminal cleanup. Successful completion confirms local commit; global route visibility remains eventual, and cross-owner readers must use actual GET results. Background publication failures are logged and handled by the existing publication lifecycle.

Update the regression coverage for both transfer-end and commit entrypoints to check acknowledgement while publication is paused, retained fences and readable holders after the caller leaves, eventual publication, and safe cleanup of independently rejected items. Tests that require a global route now wait for publication explicitly.

Validation:

  • The release build containing the production change completed successfully.
  • Llama-3.1-8B-Instruct ShareGPT replay: four TP1 workers, eight offered QPS levels from 5 to 40, two repetitions each; all 23,392 requests succeeded. Actual multi-turn histories, output lengths, phase ordering, cache accounting, and per-run cleanup were checked.
  • Mean TTFT across the two repetitions was 6.3–25.1% below the historical reference across the eight QPS levels. The historical and current runs used different hosts, so these measurements do not isolate the effect of this change.
  • The updated Rust regressions await CI execution; the benchmark results do not establish that the full CI suite passes.

@Kaimary
Kaimary merged commit 62c7f17 into main Oct 29, 2026
4 of 10 checks passed

This branch had an error being deployed

1 failed deployment
OPENAI_API_KEY — 99faf039 Deployed Oct 27, 2026 by ActivePeter via Codex failure analysis #519
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants