feat: treat OpenClaw as a first-class agent source - #5929
Conversation
🦋 Changeset detectedLatest commit: ed4fd87 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
|
Running ultrareview automatically — Hand-edited ClickHouse MV migrations change usage aggregation predicates across all agents — a missed bug could corrupt token/cost/tool-call summaries and billing data, so this warrants a deeper multi-pass review.. I'll post findings when complete. |
|
|
||||||||||||||||
|
|
||||||||||||||||
There was a problem hiding this comment.
Ultrareview completed in 18m 43s
1 issue found across 15 files
Prompt for AI agents (unresolved issues)
Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.
<file name="server/internal/hooks/ingest_hooks.go">
<violation number="1" location="server/internal/hooks/ingest_hooks.go:1635">
P1: When an embedded OpenClaw session is also observed through LiteLLM, this new correlation path leaves both assistant responses in the chat transcript. Mark correlated OpenClaw prompts as native and classify `openclaw` in `nativeAssistantTurnSource` so `proxiedTurnDuplicatesNativeStream` suppresses the proxy row.</violation>
</file>
Linked issue analysis
Linked issue: DNO-959: First-class source: alias, display label, turn correlation, usage rows
| Status | Acceptance criteria | Notes |
|---|---|---|
| ✅ | OpenClaw appears once in the dashboard agent-type filter with the “OpenClaw” label | The PR adds the OpenClaw source alias, and the PR notes the display label already existed. |
| ✅ | OpenClaw is included in canonical turn correlation after stable turn IDs are proven | The whitelist accepts OpenClaw and tests cover both native and proxied turn IDs. |
| ✅ | OpenClaw usage rows populate usage dashboards when the generic usage path does not already cover them | Both materialized views and the mirrored session predicates admit OpenClaw per-turn usage and completed tool-call rows, with a migration applying the changes. |
| ❌ | Existing goldens are updated | The supplied diff shows new correlation tests and schema-sync coverage, but no golden-file updates or evidence of existing goldens being regenerated. |
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
| if adapter != "codex" && adapter != "opencode" && adapter != "litellm" { | ||
| // openclaw carries OpenClaw's own ctx.runId, which the spike proved stable | ||
| // across before_agent_run, before_tool_call, llm_output and agent_end. | ||
| if adapter != "codex" && adapter != "opencode" && adapter != "openclaw" && adapter != "litellm" { |
There was a problem hiding this comment.
P1: When an embedded OpenClaw session is also observed through LiteLLM, this new correlation path leaves both assistant responses in the chat transcript. Mark correlated OpenClaw prompts as native and classify openclaw in nativeAssistantTurnSource so proxiedTurnDuplicatesNativeStream suppresses the proxy row.
Prompt for AI agents
Check if this issue is valid — if so, understand the root cause and fix it. At server/internal/hooks/ingest_hooks.go, line 1635:
<comment>When an embedded OpenClaw session is also observed through LiteLLM, this new correlation path leaves both assistant responses in the chat transcript. Mark correlated OpenClaw prompts as native and classify `openclaw` in `nativeAssistantTurnSource` so `proxiedTurnDuplicatesNativeStream` suppresses the proxy row.</comment>
<file context>
@@ -1630,15 +1630,17 @@ func canonicalAgentTurnID(payload *gen.IngestPayload) string {
- if adapter != "codex" && adapter != "opencode" && adapter != "litellm" {
+ // openclaw carries OpenClaw's own ctx.runId, which the spike proved stable
+ // across before_agent_run, before_tool_call, llm_output and agent_end.
+ if adapter != "codex" && adapter != "opencode" && adapter != "openclaw" && adapter != "litellm" {
return ""
}
</file context>
- sourceAliases: openclaw entry so the agent-type filter lists it once rather than leaking raw casing variants as separate surfaces - canonicalAgentTurnID: accept the openclaw adapter and the proxied agent-turn prefix. OpenClaw's ctx.runId is stable across before_agent_run / before_tool_call / llm_output / agent_end, which is the condition this was gated on; verified against the DNO-950 spike fixtures The dashboard display label already existed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TiJy9DrFpsmnD2vEkDHKNv
4302391 to
3bad2ad
Compare
Adding openclaw to canonicalAgentTurnID extended prompt correlation to OpenClaw, but the correlated upsert still only promoted codex and opencode rows. When LiteLLM recorded an OpenClaw prompt before the native hook arrived, the native row could not promote the proxy row and the session stayed labelled litellm. Also pins both OpenClaw spellings in the source-alias tests so the agent-type filter cannot regress into two entries. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TiJy9DrFpsmnD2vEkDHKNv
hook_source keeps its casing into ClickHouse and the summary MVs match the lowercase literal, so an alias folding "OpenClaw" into "openclaw" would unify the agent-type filter while those rows silently undercounted in usage. A visible split is a better failure than a silent undercount. The alias bought nothing regardless: the generated plugin always passes --provider=openclaw, so there is one raw spelling, pass-through already gives the filter a single entry, and there is no historical OpenClaw data to reconcile. This matches opencode, which carries no alias either. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TiJy9DrFpsmnD2vEkDHKNv
…peakeasy-api#5933) Closes DNO-960. Split from speakeasy-api#5929 so nothing here shares a rollback fate with the schema change. ## Why We ship an OpenClaw observability package but never told anyone how to install it. The generated ZIP README had sections for Claude Code, Cursor, Codex and OpenCode, and nothing for OpenClaw. There was also no runbook for the two things that silently break an OpenClaw install — both of which produce a *partly* working state rather than an obvious failure, which is the worst kind. ## What changed - **Generated package README**: an OpenClaw install section — the directory install, `--force` when replacing, the required `allowConversationAccess` block, the Gateway restart, and the coverage caveat - **`docs/runbooks/openclaw-install.md`**: laptop and server/CI install paths, verification, enforcement semantics, version-pin policy, and the operational gotchas from the DNO-950 spike - **`docs/plugins/overview.md`**: OpenClaw listed alongside the other observability packages and in the `downloadObservabilityPlugin` enum ## The two traps the runbook exists for 1. **`allowConversationAccess`.** Without it the conversation hooks silently never fire — no prompts, no assistant responses, no usage — while tool hooks keep working. A half-configured install looks partly fine. 2. **The harness split.** When a model routes through the Claude CLI harness (`agentRuntime: claude-cli`, which `openclaw models auth login` writes *by default* when a claude-cli profile exists), OpenClaw delegates the model and tool loop out-of-process and its tool/LLM hooks never fire. Those sessions are still covered — by our Claude Code hooks — so this is complementary coverage rather than a hole, but a customer in that mode will otherwise think the integration is broken. ## Notes - `generate.go` is a server implementation file, but the change is purely the customer-facing install text, so it belongs with the runbook rather than the feature PR. - No `hooksGeneratorVersion` bump needed: the README is not part of the rendered hook-plugin subtree that `check-generator-versions` diffs. Confirmed by rendering `export-hook-plugin -published` and checking the file list. - The runbook records that our fixtures were captured from OpenClaw 2026.6.34 while 2026.7.1-2 is what installs today, and documents the qualification procedure for that gap. The fixture-side half is speakeasy-api/agenthooks#24. - Deliberately no nightly canary against OpenClaw `latest`: it would page on OpenClaw's release cadence and registry flakes without telling us what we actually support — the same reasoning that keeps the LiteLLM real-proxy suite on `workflow_dispatch`. ## Related PRs Independent — all three branch from `main`, no stacking: - speakeasy-api#5929 — `feat:` source alias and turn correlation - speakeasy-api#5932 — `mig:` the materialized-view change 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01TiJy9DrFpsmnD2vEkDHKNv <!-- This is an auto-generated description by cubic. --> --- ## Summary by cubic Adds OpenClaw install docs for the observability plugin — the generated ZIP README now has an OpenClaw section, and a new runbook covers laptop and server/CI installs. Two configuration states silently break coverage without any error, and both are now documented. - The README's OpenClaw section is generated only when a hooks API key exists, so it never describes files that aren't in the ZIP. - The runbook details the two traps: missing `allowConversationAccess` drops prompt and usage capture while tool hooks keep working, and the Claude CLI harness routes sessions through Claude Code hooks, which capture them only if those hooks are also deployed on the machine. - Documents how coverage depends on model-auth mode, the enforcement timeout budgets, the version-pin qualification procedure, and that the hooks binary is fetched from the org's Gram server rather than GitHub. - Adds an OpenClaw package-format section in `docs/plugins/package-format.md`, and lists OpenClaw in `docs/plugins/overview.md`, the `downloadObservabilityPlugin` enum, and the observability slug table in `docs/plugins/publishing.md`. - Closes DNO-960. <sup>Written for commit 024520e. Summary will update on new commits.</sup> <a href="https://cubic.dev/pr/speakeasy-api/gram/pull/5933?utm_source=github" target="_blank" rel="noopener noreferrer" data-no-image-dialog="true"><picture><source media="(prefers-color-scheme: dark)" srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source media="(prefers-color-scheme: light)" srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img alt="Review in cubic" src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a> <!-- End of auto-generated description by cubic. --> --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Closes DNO-959.
Split out of the original combined PR so the schema change can be rolled forward or back independently — see the sibling PRs below.
Why
OpenClaw sessions weren't treated as a real, named source: the agent-type filter didn't list OpenClaw as one surface, and prompt-to-response turn correlation skipped it.
What changed
sourceAliases: anopenclawentry, so the agent-type filter lists it once rather than leaking raw casing variants as separate surfacescanonicalAgentTurnID: accept theopenclawadapter, plus the proxiedagent-turnprefix for LiteLLM-routed OpenClaw turnsThe issue gated turn correlation on OpenClaw's turn IDs being "proven stable." They are:
ctx.runIdis identical acrossbefore_agent_run,before_tool_call,llm_outputandagent_endin the DNO-950 spike fixtures. If a future OpenClaw build breaks that, correlation degrades silently rather than erroring, so it's called out in the fixture README (speakeasy-api/agenthooks#24).The dashboard display label (
openclaw: "OpenClaw") already existed.Related PRs
Independent of each other — all three branch from
main, no stacking:mig:the materialized-view change that makes OpenClaw count toward usage totalsdocs:customer install runbook and the generated package README sectionTesting
server/internal/hooksturn-correlation tests pass, including new cases for the direct and proxied OpenClaw turn IDs.🤖 Generated with Claude Code
https://claude.ai/code/session_01TiJy9DrFpsmnD2vEkDHKNv