perf(runtime): reduce database-bound turn latency

[gouhongshen]

Summary

Ports the database-bound turn-latency work from #656 onto current main without restoring implementation that main has superseded.

High MatrixOne latency magnifies serialized reads and writes around a warm agent turn. This change preserves durable, authority, reconnect, and recovery boundaries while reducing avoidable round trips and overlapping causally independent work:

• atomically acquires the canonical writer, reserves the next turn, and loads the session admission snapshot once;
• runs distributed admission, immutable history materialization, and canonical authority admission concurrently, with explicit release on partial success;
• captures Tool/Skill discovery once per request and reuses the frozen snapshot for runtime capability projections;
• batches messaging claims/releases and trace-event/edge inserts instead of opening one transaction per item;
• preserves exact event_count and last_event_id under mixed trace replays by resolving existing IDs once under the session write fence and batch-inserting only new IDs;
• reuses already-loaded run, quota, provider-attempt, invocation, and task-board facts within the same lifecycle boundary;
• atomically settles the successful provider attempt, request context, and logical invocation;
• keeps the terminal response critical path limited to canonical commit, provider settlement, and the fenced terminal transition, then runs derived projections after terminal SSE closes and only for the terminal owner;
• keeps lightweight reasoning_done / thinking_done markers durable for reconnect while raw reasoning content and deltas remain live-only;
• scopes compiled prompt static-section caching to PipelineSession;
• separates canonical manifest-chain and materialized-conversation hash domains during compaction;
• reactivates fully verified legacy reachable = 0 staged manifests inside the canonical commit transaction, preserving generation and aggregate fences without adding a read to the new-manifest hot path.

The branch is rebuilt directly on current main. It contains no #663 Edge registry commit or implementation.

Main-specific integration is deliberate:

• perf(runtime): make canonical transition recovery linear #698 ProviderCanonical V2 remains authoritative: replacement authorization is bound to the exact durable predecessor, is consumed once, and retains linear recovery. The compaction proof separately fences the manifest root and verifies the materialized conversation root; neither invariant replaces the other.
• fix(runtime): preserve prompt cache and durable provider context #693 prompt cache diagnostics and durable provider-context recovery remain intact while static prompt sections are cached only within one PipelineSession.
• feat: prepare Astra for open source release #639 removed the legacy MatrixOne task-board owner; this PR does not reintroduce it.
• main owns atomic composite checkpoint persistence; this PR keeps that write-through contract and does not restore heavy-checkpoint deferral state.
• main performs authoritative logical-attempt recovery before prompt capture. Prompt capture remains after durable admission so recovery cannot persist a prompt under the preceding attempt identity.

No heuristic fast path, Tool/Skill suppression, authorization relaxation, schema migration, public configuration field, compatibility fallback, or parallel state owner is introduced.

Related issue

Follow-up/main port of #656. The original latency investigation and paired MOI integration are tracked in matrixorigin/matrixflow#16389.

Change type

• Feature
• Bug fix
• Documentation
• Refactor or performance improvement
• Test
• Build, CI, or maintenance

User and compatibility impact

No intended API or user-visible behavior change. Warm Server turns perform fewer serialized database operations. Terminal SSE closes immediately after the durable correctness boundary, so slow derived projections do not extend the visible tail. Reconnect retains reasoning/thinking completion markers. Trace replay updates session counts and tails only for rows actually inserted. No schema or migration is required.

Architecture and complexity delta

• Canonical owners extended: SessionContextCoordinator owns combined admission and writer/turn reservation; inference execution owns atomic successful settlement; runtime lifecycle owns request-scoped capability discovery and owner-fenced post-terminal projection; PipelineSession owns compiled static prompt sections; the trace writer owns batched idempotent insertion and exact session deltas.
• Existing owners/callers inspected: Server run admission/lifecycle, canonical coordinator/commit and ProviderCanonical WAL recovery, inference invocation/provider-attempt settlement, runtime capability discovery, messaging transport, trace persistence/session summary projection, resource governor, storage readiness, prompt construction, finalization, compaction, reconnect replay, and CLI state construction.
• Superseded paths removed or kept absent: repeated preflight reads and serialized independent writes are replaced in place. The legacy task-board MatrixOne owner, checkpoint-deferral state, and fix(edge): make registry claim settlement idempotent #663 Edge registry changes remain absent. No compatibility shim or parallel store is added.
• Remaining parallel implementation: none. The frozen discovery snapshot is request-scoped evidence consumed by the existing capability owner, not another authority.

Net diff against current main: 26 files, +2782/-876; zero database tables, migrations, public statuses, or configuration fields.

Verification

• cargo fmt --all and git diff --check — passed after rebasing onto perf(runtime): make canonical transition recovery linear #698.
• cargo test -p astra-runtime --lib --no-run — passed; all runtime tests, including the MatrixOne-gated trace regression, compile.
• cargo test -p astra-runtime trace_batch_identifies_the_last_new_event_across_mixed_replays --lib -- --nocapture — passed; existing/new and new/existing orders plus repeated new IDs retain the actual inserted tail.
• cargo test -p astra-runtime turn::canonical_commit::tests --lib -- --nocapture — 2 passed; covers exact predecessor binding, one-shot authorization, recovery, and distinct manifest/conversation roots.
• cargo test -p astra-runtime provider_wal_planner_binds_rewrite_authority_to_the_exact_durable_snapshot --lib -- --nocapture — passed; V2 durable predecessor binding and tampered authorization rejection are preserved after integration.
• cargo clippy -p astra-runtime --all-targets -- -D warnings — passed after the perf(runtime): make canonical transition recovery linear #698 rebase and review fixes.
• Added ignored live MatrixOne regression trace_batch_tail_tracks_the_last_new_event_in_both_replay_orders; it compiles in the runtime test target but was not executed in this review iteration because no live MatrixOne test environment was configured.
• cargo check -p astra-services --tests, cargo clippy -p astra-services --tests -- -D warnings, and cargo test -p astra-services session_context --lib — passed for staged-manifest recovery.
• cargo test -p astra-runtime superseded_terminal_skips_all_owner_derived_projections -- --nocapture — passed; a superseded terminal owner invokes no derived projection sink.
• cargo test -p astra-runtime terminal_events_for_persistence_keeps_only_terminal_lifecycle_events -- --nocapture — passed; completion markers persist while raw reasoning remains excluded.
• cargo test -p astra-runtime turn::llm::durable::tests --lib -- --nocapture — 48 passed.
• cargo test -p astra-runtime typed_objective_relations_survive_real_tiered_compaction --lib -- --nocapture — passed with from_materialized_admission.
• cargo test -p astra-services resource_governor --lib — 17 passed.
• make test-contract — 10 admin, 4 HTTP, and 2 settings contract tests passed.
• Live MatrixOne completion_http_boundary_persists_success_failure_and_caller_cancellation passed twice on this port: 2.29s including first schema readiness and 0.42s warm.
• The cancellation regression against matrixone:latest passed 20/20 after removing the unsafe INSERT IGNORE affected-row early return and restoring authoritative settlement-debt identity recovery.
• make test-offline previously progressed through Server runtime and Web profile tests, then stopped in an unchanged astra-edge test because this macOS host rejected its temporary workspace-lock namespace. This PR changes neither crates/astra-edge nor crates/astra-tools.

Public entrypoint exercised on the original #656 tree: local MOI /chat/stream with Astra Server and EPH Sandbox, including warm follow-up turns under injected SQL latency. This exact main port was not redeployed.

Unhappy paths covered include authority/cursor conflict, partial admission cleanup, cancellation and lease loss, provider delivery uncertainty, terminal CAS supersession, settlement failure/recovery, missing discovery snapshots, compaction hash mismatch, ProviderCanonical predecessor mismatch/repeated replacement, mixed trace replay ordering, and quota boundaries.

Final checklist

• I added or updated tests at the layer that owns the behavior, or explained why no test is needed.
• I updated public or design documentation for contract changes, or the change needs no documentation update.
• I checked the diff for credentials, private URLs, customer data, generated files, and other sensitive information.
• The PR title follows the repository's Conventional Commit format.

[XuPeng-SH]

结论：当前不建议合并，请先修复以下问题。

1. [P1] 流式执行失去 terminal ownership 后仍会写派生状态

crates/runtime/src/server/run/lifecycle/mod.rs 的 terminal projection block（约 L16629-L16684）没有受 owner_terminal_committed 约束，并调用了无 generation fence 的 persist_usage、transcript materialization 和 run_after_core。当 terminal CAS 返回 Superseded、authority 不确定或 generation 过期时，旧 executor 仍可能覆盖 usage、写入 CSL/hook/observer/promotion/projection，污染企业审计、计费和 ContextPipeline 数据。

请让所有派生写入重新受 terminal owner fence 保护；删除额外的 unfenced usage 写入，只保留 atomic settlement 或 persist_usage_if_current_owner。需要补 terminal CAS=Superseded 的测试，断言这些派生 sink 不会被调用。

2. [P1] best-effort projection 被放入 terminal SSE 关键路径

同一段代码在发布 terminal SSE 前 tokio::join! 等待 usage、transcript、hook DB、Memoria observer、SessionEnd hooks、promotion 和 projection。任一 sink 变慢都会阻塞已经 durable committed 的回答，导致客户端超时并触发重复请求；这也与本 PR 的降低 turn latency 目标相悖。

canonical commit、provider settlement、terminal CAS 才是响应正确性的关键路径。请恢复“先完成 terminal SSE，再执行派生 projection”的边界；如果必须提前执行，也必须有明确超时、降级和 owner fence。

3. [P2] 删除 reasoning_done / thinking_done 的 durable replay 会破坏断线恢复

crates/runtime/src/server/run/lifecycle/run_state.rs 的 terminal persistence 和 live persistence 过滤掉了 reasoning/thinking completion marker。客户端已经收到 reasoning delta 后断线重连时，没有 completion marker 可能无法关闭 thinking UI，直到整个 run 结束。

请保留轻量 completion marker，或先定义并实现明确的 reconnect state snapshot，再删除 marker，并补充断线恢复测试。

4. [P1/process] PR 混入了尚未批准且已过时的 #663

PR #670 的 commits 包含 #663 的旧版 edge registry commit（9728d51），但 #663 仍为 Open / CHANGES_REQUESTED，当前 head 已经不同；当前 main 也不包含该 commit。PR 描述中“#663 changes are already part of the base”不成立。请基于最新 main 重新整理纯 latency 分支，避免把未批准的 edge claim/heartbeat/unregister 实现带入本 PR。

正向评价：atomic writer+turn admission、successful inference settlement、PipelineSession-scoped prompt cache、request-scoped discovery snapshot，以及 hash-domain separation 的方向是正确的，也符合 Astra 的企业 durable-agent 定位。但性能优化必须服从 authority fencing、Trace/审计完整性、Runner/Edge side-effect recovery 和断线恢复语义。

[gouhongshen]

已按 review 完成自检和修复，提交为 6abe5e99：

1. 删除了流式终态前无 generation fence 的 persist_usage，保留 atomic settlement / persist_usage_if_current_owner；journal、CSL/transcript、hooks/observer/promotion、Work/Memory/workspace 等派生副作用统一受 owner_terminal_committed gate 约束。新增 Superseded 回归测试，确认 gate 内的派生 sinks 均不执行。
2. 派生 projection 已移到 drop(event_tx) 之后；terminal SSE 不再等待这些 best-effort 写入。
3. 恢复 reasoning_done / thinking_done 的轻量 durable marker；raw reasoning content/delta 仍保持 live-only。
4. 分支已从最新 origin/main 重建并 force-with-lease 更新；9728d51、相关 merge commits，以及 edge_registry.rs / edge_dispatch_db_it.rs diff 均已移除。PR 描述中的错误 base 声明和净 diff 统计也已更正。

本地定向验证通过：两项新增/恢复的 runtime tests、cargo fmt --check、git diff --check。新的 CI 已由重写后的纯 latency 分支触发。

[gouhongshen]

The current head 6abe5e99 addresses all four findings from the review of 9c48ca57:

1. All journal, transcript, hook, observer, promotion, projection, and workspace-cleanup writes are now inside run_owner_fenced_terminal_projections(owner_terminal_committed, ...). A Superseded terminal owner invokes none of those sinks; the new deterministic test verifies zero calls.
2. The terminal events and turn_complete are sent and event_tx is dropped before derived projections run. The only non-atomic usage repair left in the durable boundary is persist_usage_if_current_owner; the unfenced usage write is gone.
3. reasoning_done and thinking_done are again retained as lightweight durable terminal/reconnect markers, while raw reasoning/thinking deltas remain excluded. The persistence test covers both markers and raw-content exclusion.
4. The branch was rebuilt directly on current main; git diff origin/main...HEAD contains no edge_registry.rs or edge_dispatch_db_it.rs, so no fix(edge): make registry claim settlement idempotent #663 implementation remains in perf(runtime): reduce database-bound turn latency #670.

The PR description has also been updated to reflect these ownership, SSE ordering, reconnect, and branch-scope corrections. Please re-review the current head rather than the superseded reviewed commit.

[XuPeng-SH]

当前 head 已修复上轮四个问题，但仍有一个需要阻断合并的持久化兼容性问题：

[P1] 复用历史 staged manifest 时没有把它重新激活

persist_database_immutables_in_tx 现在只在新插入时把 reachable 写成 1；当 INSERT IGNORE 命中已存在的相同 manifest 时，代码仅校验 manifest_json，不会把已有行从 reachable = 0 更新为 1。见：

Astra/crates/services/src/session_context_coordinator.rs

Lines 2597 to 2660 in 6abe5e9

	let manifest_insert_sql = matrixone_statement_with_null_shape(
	"INSERT IGNORE INTO conversation_manifest_nodes
	(isolation_domain, owner_user_id, session_id, branch_id, manifest_root,
	parent_manifest_root, completed_turn, conversation_seq,
	compaction_generation, canonical_segment_bytes, total_canonical_bytes,
	total_message_count, manifest_json, reachable)
	VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, 1)",
	[node.parent_manifest_root.is_some()],
	);
	let result = sqlx::query(&manifest_insert_sql)
	.bind(&key.isolation_domain)
	.bind(&key.owner_user_id)
	.bind(&key.session_id)
	.bind(&key.branch_id)
	.bind(&node.manifest_root)
	.bind(&node.parent_manifest_root)
	.bind(i64::from(node.completed_turn))
	.bind(i64_from_u64(
	"conversation sequence",
	node.conversation_seq,
	)?)
	.bind(i64_from_u64(
	"manifest compaction generation",
	node.compaction_generation,
	)?)
	.bind(i64_from_u64(
	"manifest segment bytes",
	canonical_segment_bytes,
	)?)
	.bind(i64_from_u64(
	"manifest total canonical bytes",
	total_canonical_bytes,
	)?)
	.bind(i64_from_u64(
	"manifest total message count",
	total_message_count,
	)?)
	.bind(database_to_json("manifest", node)?)
	.execute(&mut **tx)
	.await
	.map_err(|source| database_error("persist_manifest", source))?;
	if result.rows_affected() == 0 {
	let stored = sqlx::query(
	"SELECT manifest_json FROM conversation_manifest_nodes
	WHERE isolation_domain = ? AND owner_user_id = ?
	AND session_id = ? AND branch_id = ? AND manifest_root = ?",
	)
	.bind(&key.isolation_domain)
	.bind(&key.owner_user_id)
	.bind(&key.session_id)
	.bind(&key.branch_id)
	.bind(&node.manifest_root)
	.fetch_one(&mut **tx)
	.await
	.map_err(|source| database_error("verify_existing_manifest", source))?
	.try_get::<String, _>("manifest_json")
	.map_err(|source| database_error("decode_existing_manifest", source))?;
	let stored: ContextManifestNodeV1 = database_json("existing_manifest", &stored)?;
	if stored != *node {
	return Err(SessionContextCoordinatorError::NeedsRepair(
	"existing immutable manifest does not match its content-addressed key".into(),
	));
	}
	}

这是升级兼容场景，不是理论分支：当前 main 的旧流程会先在独立事务里写入默认 reachable = 0 的 manifest，再在 canonical commit 事务里执行 UPDATE ... SET reachable = 1。进程中断、cursor conflict 或 commit 失败都可能留下内容完整但不可达的 staged manifest。升级到本 PR 后重试相同确定性 delta 会命中这行；随后 session head 被提交为该 manifest root，但 materialize 仍按 reachable = 1 过滤，canonical head 因而指向不可物化的历史。

请在确认 existing manifest 及其 references 完全匹配后，于同一事务显式激活该行（保留 generation/totals 等 fence，并校验恰好一行），并增加回归测试：预置相同 root、reachable = 0 的旧版 staged manifest，重试 commit 后应变为 reachable 且可 materialize。

[gouhongshen]

已修复这个 P1，并将分支 rebase 到最新 main。修复在 7ca196d：新 manifest 仍走一次直接 INSERT 的热路径；只有命中历史重复主键时才进入兼容慢路径，在同一事务中 FOR UPDATE 校验完整 manifest、独立 totals/generation/segment-byte 字段和有序 references。仅当已验证旧行的 reachable = 0 时，才以 generation/totals 等 fence 更新为 1，并要求恰好更新一行；已可达的同内容行保持幂等。新增 MatrixOne 回归用例预置旧版完整 staged manifest（reachable = 0），验证 retry 后可达且能够 materialize。cargo check -p astra-services --tests、cargo clippy -p astra-services --tests -- -D warnings 和格式检查已通过。PR 描述也已同步。

[XuPeng-SH]

重新审查了最新 head 7ca196d3。上轮指出的 legacy staged manifest 问题已经正确修复：existing row 与 references 先被完整校验，reachable = 0 再在同一事务内带 fence 激活，并有真实 MatrixOne 的 materialize 回归测试。

当前仍有一个合并阻断项：

[P1] rebase 后 runtime 测试无法编译

最新 main/#690 增加的测试仍调用已被本 PR 重命名移除的 CanonicalRewriteProof::new：

Astra/crates/runtime/src/server/run/lifecycle/tests.rs

Lines 833 to 837 in 7ca196d

	// Admit an actual prefix, then prove the real compaction rewrites it.
	let prior = messages[..7].to_vec();
	let root = astra_turn_types::canonical_conversation_root(&prior);
	let mut proof = CanonicalRewriteProof::new(&prior, &root, 0);
	let permit = proof.begin(&messages);

本 PR 只保留 CanonicalRewriteProof::from_materialized_admission，因此 clippy、astra-runtime、core crates 和 online integration 都在同一处报 E0599；这不是 CI/flaky 或网络问题，而是确定性的 source integration failure。

请把该测试迁移到 from_materialized_admission，并继续保持两个 hash domain 分离：第二个参数应表达 synthetic/admitted manifest root，不要重新把 canonical_conversation_root(&prior) 当成 manifest root。修复后至少确保 runtime test compile、clippy 和相关测试通过。

[gouhongshen]

已修复最新 P1，并同时处理了冲突和同源编译问题。分支已 rebase 到 23ee6747（#693），冲突合并保留了 #693 的 final wire-prompt cache diagnostics / durable provider-context recovery，也保留本 PR 的 session-scoped static-section cache。a906bc44 将 #690 lifecycle 测试迁移到 CanonicalRewriteProof::from_materialized_admission，传入独立的 synthetic manifest root，并显式断言它与 materialized conversation root 不同。自检还发现 canonical commit 自身有 4 处旧构造器和 2 处旧字段引用，因此一并迁移，并新增独立 base_prefix_root 保存 durable conversation hash；base_manifest_root 继续只承担 commit fence，两个 hash domain 没有重新混合。验证通过：runtime 全测试目标编译、2 个 canonical proof tests、typed-objective real-compaction regression、cargo clippy -p astra-runtime --all-targets -- -D warnings、格式和 diff check。PR 描述已同步。

[XuPeng-SH]

Two blockers remain against latest main.

First, insert_trace_events returns the aggregate INSERT IGNORE affected-row count together with events.last() as the last inserted event. A mixed idempotent batch can insert an earlier item while ignoring the final duplicate; the caller then increments event_count but advances last_event_id to an event that was not inserted by this delta, potentially moving the session trace tail backward. Current main updates the tail only for each row that actually inserted. Please preserve the exact inserted tail identity and add mixed existing/new regression cases in both orders.

Second, #698 is now in main, and this PR has a real conflict in canonical_commit.rs. The conflict is semantic: current main uses ProviderCanonical V2, binds replacement authorization to the exact durable predecessor, and makes it one-shot, while this head still carries the older V1 authorization shape while separating manifest and conversation roots. Please rebase and compose both invariants; choosing either conflict side wholesale would regress one of them. Exercise recovery, predecessor mismatch, repeat replacement, and distinct manifest-root/conversation-root cases after integration.

[gouhongshen]

已基于最新 main（含 #698）完成 rebase，并在 head 7b434981 处理本轮两个 blocker：

1. Trace mixed replay：意见成立。批量 INSERT IGNORE 的聚合 affected rows 无法证明最后一个输入事件就是最后一个实际新增事件。现在在已有的 session write fence 内一次批量读取已存在 event IDs，去重后仅对新 IDs 做一次批量 INSERT；返回的 delta count 和 tail 都来自实际新增集合。没有退回逐条 SQL。补了 existing→new、new→existing 和批内重复 ID 的单元测试，以及两种顺序的 live MatrixOne 回归测试。
2. perf(runtime): make canonical transition recovery linear #698 integration：意见成立。冲突没有选择任一侧覆盖，而是组合两套约束：ProviderCanonical V2 的 exact durable predecessor、one-shot authorization 和 linear recovery 全部保留；本 PR 的 manifest-chain root 与 materialized conversation root 继续分域，commit fence 使用前者，rewrite/WAL proof 使用后者。同步迁移了 perf(runtime): make canonical transition recovery linear #698 新增测试。

本地验证：runtime 全量 lib test target 编译通过；trace mixed replay、canonical recovery/one-shot/distinct roots、provider WAL exact predecessor/tamper rejection 均通过；cargo clippy -p astra-runtime --all-targets -- -D warnings 通过。PR 描述和准确 diff 已同步更新。请重新 review。

[XuPeng-SH]

Re-reviewed the current head against main, including the full integrated diff. Both prior blockers are resolved: mixed trace replay derives event_count and last_event_id from the actual new-ID set under the session write fence, and the #698 integration preserves exact durable-predecessor binding, one-shot replacement authorization, linear recovery, and separate manifest/conversation hash domains. I also rechecked admission cleanup, successful provider/logical settlement, cancellation recovery, terminal-owner fencing, mailbox wake/poll behavior, and quota snapshot changes. I found no remaining correctness blocker; required CI is green and the merge tree is clean.