Skip to content

Implement the runtime kernel: agent loop, session supervisor, and the first reference plugin - #9

Merged
scrothers merged 101 commits into
mainfrom
integration/runtime-kernel
Jul 25, 2026
Merged

Implement the runtime kernel: agent loop, session supervisor, and the first reference plugin#9
scrothers merged 101 commits into
mainfrom
integration/runtime-kernel

Conversation

@scrothers

Copy link
Copy Markdown
Member

What & why

Builds the runtime kernel — the thing every existing package was for. Before this, docs/specifications/ was settled, the sqlite state backend and the pkg/** plugin-author SDK had shipped, and roughly 21k lines of tested Go existed, but there was no running kernel: no cmd/ binary, no agent loop, no session supervisor, no real plugin. Everything that existed was a leaf — internal/policy evaluated rules nothing called, internal/pluginruntime launched subprocesses nothing chose, internal/statebackend persisted sessions nothing created — and internal/kernelcallback carried five codes.Unimplemented stubs blocked on machinery that had never been built.

This adds the trunk, plus the first real plugin to prove it end to end. ./bin/agent -prompt "…" now loads agent.hcl, launches provider plugins, runs the full 18-step RunTurn loop through the plan/apply gate, and persists a real session file.

~40 new packages, delivered in dependency order:

  • Pure domain (no slog/telemetry, ~95% covered): xdg, plugincache, sessionscope, callhash, cost, bounds, doomloop, retrypolicy, schemavalidate, circuitbreaker, hookpayload, streamaccum, modelrequest, tokencount
  • Interface + driver seams: plandecision, interactive, providercatalog
  • I/O collaborators: sessionstate, hookdispatch, tooldispatch, modelcall, contextassembly, plangate, pluginhost, providerresolve
  • Orchestration: turn (the 18-step algorithm), session (lifecycle, turn loop, bounds/doom-loop/breaker), kernel (composition root)
  • Reference plugin: internal/anthropic (+catalog, messages) and cmd/anthropic, hand-rolled on net/http so no vendor SDK enters the module graph every plugin author inherits
  • Binaries: cmd/agent, cmd/anthropic

Two additive, non-breaking proto changes: ToolSchema.terminates_turn and StreamEvent.redacted_thinking. Both go through buf generate; buf breaking is unaffected and ProtocolVersion is deliberately not bumped.

Checklist

  • Local gate passes: go mod tidy is a no-op, go build ./..., go vet ./..., gofmt -l -s . prints nothing, go test -race -covermode=atomic ./..., golangci-lint run
  • Observable behavior changes update the matching docs/specifications/ document in this same PR
  • No hand edits under pkg/*/proto/v1/.proto changes made in api/ and regenerated with buf generate
  • Doc cross-references are path + heading anchor (never section numbers); renamed headings were grepped for inbound anchors
  • New internal/ packages include README.md + CLAUDE.md in the same commit
  • No compiled artifacts outside bin/

Also run green locally: go test -race -tags=integration ./... (74 packages), gosec -exclude-generated, govulncheck, buf lint, buf format --diff --exit-code.

Notes for reviewers

Two tracked deviations from a spec MUST — please read these first

Both exist because no frontend plugin category is built yet, and both are wired at a single call site (internal/kernel/turnstack.go's newTurnDriver) so they are impossible to miss in review.

  1. ask plan decisions are auto-approved. plan-apply-gate.md#decision-semantics requires an ask to emit a permission-request and block until a human answers. There is nothing to ask. internal/plandecision/drivers/autoallow has no usable zero value — construction fails unless AcknowledgeUnsafeAutoAllow: true — logs a WARN at construction and one per resolution, resolves only ONCE (never SESSION/ALWAYS, so it can create no durable state a real resolver would have to reconcile), and permanently persists plan_items.decided_by = "UNSAFE-AUTO-ALLOW(no-frontend-attached)". In plain terms: a session run by this build executes mutating tool calls a human was supposed to approve, and the audit trail says so per item. The fix is the real drivers/frontend resolver, which drops in behind the same one-method Resolver interface with no call-site change.
  2. interactive-kind calls are auto-refused. internal/interactive/drivers/unattended returns a PERMISSION_DENIED tool result the model can observe and adapt to. The asymmetry with (1) is deliberate: an ask item has a defensible default (the call as proposed); an interactive call's whole payload is a human's answer, and any synthetic one would be a fabrication in the model's own history.

Scoped non-conformance

Cost rollup is single-session. turn-algorithm.md#cost-accounting requires rollup across a session tree; this build is root-sessions-only by agreed scope. bounds.Tracker carries the parent link from day one (NewTracker(limits, parent), Debit walks to the root) and is unit-tested against a synthetic parent even though production always passes nil, so nothing here gets rewritten when sub-agents land.

Spec gaps resolved by documented interpretation, not invention

Each is recorded in the owning package's CLAUDE.md with its reasoning, and each is the kind of thing worth disagreeing with if you read it differently:

  • Which hook points are veto-bearing. hook-dispatch.md uses the phrase three times and never enumerates the set. Resolved as {plan-ready, pre-tool-call} — the two points immediately preceding a blockable action. NewRegistry rejects a veto subscription anywhere else.
  • Hook ordering across multiple .hcl files. agent-profiles.md's textual-position rule assumes one file; the XDG layout permits several. Resolved as (lexicographic filename, byte offset) — never filesystem enumeration order.
  • Doom-loop and circuit-breaker trips have no SessionStatus of their own, so both terminate COMPLETED with the reason on Result.FinalAnswerReason. Reusing an error_max_* subtype would make session_meta lie about which limit fired. A dedicated status is proposed for a later revision.
  • pre-tool-call needs a plan_item before a Plan exists — resolved by minting a provisional PENDING item at step 7 and carrying it forward by identity into step 10, so the gate stamps a decision onto the same pointer a hook already saw.
  • Provider category is unknown before launch, but go-plugin needs one to dispense. Resolved by a lock-file category field, falling back to a seven-key Describe probe.

One spec gap this PR does not resolve

A session's own user prompt has nowhere legal to live in events, so replay reconstructs every assistant message, tool call, and plan but not the prompt that caused them. producer_category is the seven plugin categories; state-backend.md says every kind but hook_error is written by the producing plugin's own callback connection; and the reserved kernel producer is restricted to plan/apply. message is also the wrong shape — its payload carries usage/cost, extracted to cost_ledger at write time, and a prompt has neither. Fixing it is wire-visible (widen the kernel producer for a user-authored message, or give the user turn its own kind), so it is recorded in state-backend.md's Open questions rather than decided here. A local workaround — fabricated producer, zero-cost ledger row — would put a lie in the audit log.

Deliberately not built

No RunSession/session tree/sub-agent spawning (parent-link seams already exist in bounds.Tracker, sessionstate.Live); no real frontend; no internal/registry provider download or lock-file writing — bootstrap is dev_overrides plus already-cached binaries, and an unresolvable provider reports an aggregated MissingError rather than hanging.

Worth close reading

  • internal/turn/runturn.go — declaration order is the invariant everything else bends around. One pending slice built at step 7, never reordered; every grouping is a slice of pointers into it. post-tool-call dispatches and tool_result blocks must both emerge in tool_use order, not grouped by kind or completion order.
  • internal/tooldispatch — provider semaphore first, per-key second, always. That single fixed order is what makes the two-level scheme deadlock-free by construction.
  • internal/kernel/turnstack.go — the late-binding seam. plangate needs a session id at construction and one Breaker must be shared with tooldispatch, but internal/session mints the id inside Run, which is called with an already-built driver. Resolved with turnStack + sessionSink, modeled on pluginhost's callbackSlot.
  • internal/anthropic — held to exactly what a third party gets: a depguard rule forbids it importing any other internal/ package. That constraint is the point (it is what proves pkg/ is sufficient), and it is also why this package has no spans or metrics — no pkg/ surface exposes telemetry to a plugin author yet. Building it caught a real SDK gap (Sink could not emit redacted_thinking), fixed in pkg/model rather than worked around.

Review-driven fixes included

The last three commits are fixes from a review of the rest of the branch: persisted event payloads were marshaled without pinned proto map ordering (violating determinism.md — now centralized in statebackend.MarshalPayload); no kernel-originated event reached the event bus because the composition root bound the raw session handle past sessionstate.Live's republish; a tool's default_timeout was charging lock-queue time and could report a fabricated retryable TIMEOUT for a call that never ran; plus a missing clock injection, a span that always ended OK, and a circuit-breaker debit that preceded its own audit write.

scrothers added 30 commits July 24, 2026 20:58
Implements XDG Base Directory layout resolution for the kernel,
per architecture.md#xdg-layout. Resolves paths for project config,
global config, plugin cache, persistent data, and session state.

- Pure-domain package: I/O-free, deterministic, ~95% test coverage
- No logging/telemetry per logging-telemetry.md exemption
- Comprehensive table-driven tests with race/shuffle support
- Full documentation: doc.go, README.md, CLAUDE.md
The agent loop's opt-in explicit terminal-tool done detection
(agent-loop/turn-algorithm.md#done-detection) is specified as a
`terminates_turn: bool` schema annotation, but ToolSchema carried no
wire field for a provider to declare it, so the path was unimplementable.

Add `bool terminates_turn = 11` (next free field number) and document it
across the tool specs: a new data-types.md section covering the
resource-only constraint and the "layers on top of, never replaces,
implicit no-tool-calls detection" rule, the full-shape block and prose in
protocol.md#getschema, a MAY row in the conformance matrix, and a
cross-reference from turn-algorithm.md to where the field now lives.

Purely additive: no field number reused, no existing field changed, so
`buf breaking` stays clean and pkg/common.ProtocolVersion is unchanged.
content.v1.RedactedThinkingBlock has existed as a canonical content-block
type since the schema was written, but StreamEvent had no variant able to
produce one — so a model plugin had no way to surface a vendor-encrypted
reasoning block, and a vendor that requires such blocks be echoed back
verbatim rejects the entire conversation on the following turn.

Add `RedactedThinking redacted_thinking = 10` (next free oneof field
number) carrying a single opaque `bytes data`, mirroring
RedactedThinkingBlock. Unlike ThinkingDelta this is whole-block, not
incremental: the payload is opaque, so there is nothing to accumulate.
Document the variant in the model spec's StreamEvent block, expand the
canonical-content-block prose to cover it, and add a conformance row.

Purely additive: no field number reused, no existing variant changed, so
`buf breaking` stays clean and pkg/common.ProtocolVersion is unchanged.
Close gaps where docs/specifications already reference a kernel-configurable
knob with no agent.hcl field to set it:

- settings.event_bus{} with subscribe_queue_bound (default 1024).
- settings.doom_loop{} with window_size/threshold, defaulted from
  internal/doomloop.DefaultConfig so the numbers have one source of truth.
- settings.default_hook_timeout_ms / default_tool_timeout_ms, flat
  attributes defaulting to 5000/30000.
- settings.max_depth as a *int, carried as declared rather than defaulted
  here — agentprofile.RootRemainingDepth already resolves the unset case.
- hook{}'s optional timeout_ms per-subscriber override.

Both defaulting paths now share defaultSettings() so decode() and
decodeSettings cannot drift. Adds TelemetryConfig, bridging Settings into
internal/telemetry.Config; telemetry = false forces the noop backend
regardless of observability{}'s contents.
The fixed pkg/common.CallbackBrokerID is collision-free only while
broker.AcceptAndServe is called exactly once per launched subprocess.
That guarantee was held by a sync.Once on categoryPlugin — which is
per-category, not per-subprocess, so a launch whose plugin map carries
more than one category could race several goroutines onto the same fixed
broker ID.

Lift the Once, the callback server, and the telemetry provider onto a new
launchScope, shared by reference across every categoryPlugin built for
one Launch call, and make pluginMap variadic over categories. Single-
category launches behave exactly as before; the multi-category shape is
the primitive a later dev_overrides category probe needs, which keys one
subprocess by all seven categories because the binary's real category
isn't knowable ahead of time.

TestLaunchScope_serveOnce drives the once-guarded core concurrently
through a seven-entry plugin map and asserts a single serve results.
agent-loop/hook-dispatch.md requires the kernel dial
HookSubscriberService "on the same connection it already holds to that
plugin's category service" — go-plugin muxes several gRPC services over
one subprocess connection — but that connection did not survive past
categoryPlugin.GRPCClient, so no second service could be reached.

Record it on the launch scope, keep it on Plugin, and add
Plugin.HookClient. HookSubscriberService is the one service in the
protocol that is category-agnostic, so naming it costs this package none
of the category knowledge Dispensed()'s any return exists to avoid; the
raw *grpc.ClientConn stays unexported because go-plugin owns and closes
it.

The fixture's hook.Observer now logs back through the kernel callback, so
TestLaunch_hookClientSharesCategoryConnection proves a DispatchHook
issued through HookClient reached that subprocess over the same
connection its ToolServiceClient came from, and dies with it on Close.
Add NewEventID, EventKindText, and EventPayloadType for callers outside
this package.

NewEventID shares NewSessionID's mutex and monotonic ULID entropy source
via a common newULID, so the two can never mint the same value in one
millisecond and there is a single canonical event-id format.

EventKindText is a thin wrapper over encodeEventKind rather than a second
switch. EventPayloadType is the kind -> pluggableharness.event.v1 message
table from state-backend.md's kind enum, which had no code representation
before; a later Emit implementation needs both to build the reserved
kernel.event.{kind} bus topic and BusEvent.payload_type.
WAL keeps a concurrent reader from blocking the kernel's writes as a
steady state, but another process opening the same file briefly takes an
exclusive lock while sqlite initializes the -shm/-wal sidecars. With no
busy timeout a write landing in that window failed outright with
SQLITE_BUSY, which is the outcome state-backend.md's ordering and
concurrency section rules out.

Request busy_timeout(5000) via the DSN, alongside foreign_keys, so a
replacement pooled connection inherits it too. Surfaced as a flaky
TestSession_concurrentReaderDuringWrites under a loaded test suite.
Add Session.EventsMatching and the EventQuery it takes — kinds,
from_sequence, and limit, mirroring kernel-callbacks.md's ReadEvents
request field for field, so answering "the last N tool_result events"
does not mean reading a whole session log into memory.

Events is now EventsMatching with a zero-value query: one read path, one
SQL builder, no unfiltered fast path that can drift from the filtered
one. The existing Events tests are unchanged and cover the equivalence,
alongside an explicit field-for-field comparison of both call shapes over
the same session.

The IN (...) list is built by walking EventQuery.Kinds in caller order
with a seen map used only to skip duplicates, never by ranging a map, so
the rendered statement can never depend on Go map iteration order. A
negative limit is rejected rather than passed to sqlite, which reads one
as "no limit".
plan and apply events are assembled by the kernel from a turn's tool
calls spanning several tool providers, so no plugin owns them as
producer — yet events.producer_category/name/version are all NOT NULL.

KernelProducer returns the fixed identity for exactly those two kinds:
name "kernel", version "1" (the event.v1 payload generation, not a kernel
release, so a session's producers manifest is stable across upgrades),
category CATEGORY_UNSPECIFIED, empty source. Storing it is a composite
decision, never a category one: encodeProducer accepts the unspecified
category only when the name is exactly "kernel" and the kind is plan or
apply, and decodeProducer resolves the reserved "kernel" category text
only when paired with that name. encodeProducerCategory still rejects
CATEGORY_UNSPECIFIED outright and the reserved text is absent from both
category tables, so a real plugin producer's path is unchanged and
CATEGORY_UNSPECIFIED gains no new way to reach a written row.

hook_error stays out: the spec has its producer columns identify the
failing subscriber, not the kernel.
scrothers added 23 commits July 24, 2026 22:45
newCallbackServer never passed these through, so any real launched
plugin calling Emit/ReadEvents/GetSession/CountTokens would
nil-pointer-panic inside kernelcallback once those RPCs stopped being
stubs. Made all three MUST-be-set in Config.validate(), matching
kernelcallback's own convention. Also fixes a test that called
ReadEvents with a nil stream, which its former Unimplemented stub
tolerated but the real implementation does not.
Compose internal/contextassembly, hookdispatch, modelrequest,
modelcall, plangate, and tooldispatch into the numbered algorithm in
docs/specifications/agent-loop/turn-algorithm.md. The package owns no
algorithm of its own: it contributes the documented order, the
declaration-order bookkeeping that keeps every tool_result paired with
its tool_use block across the kind split, and the adapters that let
plangate keep declaring its own HookDispatcher and ApplyOutcome instead
of importing hookdispatch and tooldispatch.

Steps 16-18 and the session lifecycle stay out: Result hands the future
session driver the call hashes, spend, tripped providers, and done
status those checks need.

The conformance test runs a two-turn scenario through five hand-written
fakes sharing one ordered call log and asserts the sequence exactly,
which is conformance.md's first MUST.
Adds internal/session: the kernel's outer loop around internal/turn —
profile resolution (including the implicit default profile), model
routing and tool-scope expansion against the loaded provider catalog,
session creation and terminal-status persistence, session-lifetime
callback grants, session-start/session-end dispatch, and steps 16-18 of
turn-algorithm.md.

All three bound dimensions, the doom-loop detector, and the plan-gate
circuit breaker route through one limit-reached path: exactly one
final-answer turn naming what fired. Doom-loop and breaker trips map to
SESSION_STATUS_COMPLETED with the reason on Result.FinalAnswerReason,
since session.v1.SessionStatus has no subtype for either.
StartProviderCatalogBuild wraps providercatalog/drivers/plugin.New's
extraction pass; StartToolPreview's doc comment now covers its second
call site, the one-time SupportsPreview probe that driver performs.
Wraps a *pluginhost.Registry to satisfy providercatalog.Catalog against
real, live plugin subprocesses. Extraction is eager (safe because a
Registry is only mutated by Supervisor.Start, which completes before
New is ever called). ToolHandle.SupportsPreview is resolved by a
one-time live Preview probe per operation, checking specifically for
codes.Unimplemented per doc.go's reasoning. ContextHandle.Position is
set directly from Live.LaunchIndex, confirmed to already be agent.hcl
declaration order. ContextHandle.TokenBudget cannot reflect an
agent.hcl override yet — pluginhost retains no decoded provider
config outside Configure — documented as a known gap rather than
guessed at.

Corrects providercatalog/CLAUDE.md's stale note anticipating a
drivers/drivers.go selector once a second driver landed: drivers/fake
is a test double, never a second production option, so no selector is
warranted.
ValidatePricing accepts free = true with no tiers at all, but
modelcall's persist resolved a tier unconditionally, so every
completion from a legally-declared free provider failed with
ErrNoMatchingTier before its message was ever persisted.

Add cost.IsFree for exactly that shape and short-circuit on it.
A free Pricing that also declares tiers is unaffected: those
tiers are validated like any other and must still be resolved.
The composition root builds the turn stack's collaborators over the
same sole-writer *statebackend.Session that Live wraps, rather than
opening a second handle on the same file. Nothing above
internal/session can reach that handle otherwise, since Runner.Run
creates it.

Emit/EmitMessage/EmitPlan remain the only path for plugin-originated
events: they debit the budget tracker and republish onto the bus.
Config gained required Scopes/Sessions/Tokens fields; the
integration harness was never updated, so every test in it failed
NewSupervisor validation.
internal/kernel.Run brings the process up in dependency order, runs
exactly one non-interactive session, prints its final message, and
tears every phase back down in reverse — including after a bring-up
failure. cmd/agent parses flags, calls it once, and maps the result
to an exit code.

The per-session half of the turn stack is built lazily on the first
turn: plangate needs the session id at construction and five packages
need the open session handle, but internal/session mints both inside
Runner.Run. A sessionSink slot bridges the gap, mirroring
pluginhost's callbackSlot.

Both frontend-absent deviations are wired loudly at one call site:
autoallow for ask-decisions, unattended for interactive calls.
Translates a kernel StreamCompletionRequest + model.Spec into
Anthropic's request body: content-block translation with capability
gates, message coalescing, cache-breakpoint placement, thinking/effort
handling, and deterministic JSON Schema/tool-argument serialization.
Several event.v1 payloads reach a structpb.Struct — ToolCallEvent through
ToolCall.arguments, ToolResultEvent through ToolResult.payload and
ToolError.details, MessageEvent through every ToolUseBlock.arguments,
ContextContributionEvent through contributed content blocks. A proto map
marshals in randomized order unless Deterministic is set, so the same
session persisted to different bytes on every run, which determinism.md
forbids for any persisted payload.

Only internal/plangate set the option, via a package-local helper. Promote
it to statebackend.MarshalPayload as the single source of truth and route
modelcall, contextassembly, and plangate itself through it. tooldispatch's
two call sites move over in the following commit, alongside its clock.
ToolSchema.default_timeout is documented as the Invoke deadline, but the
context carrying it was derived before acquireLocks and passed into it, so
its clock ran while a call sat queued behind an exclusive safe:false
sibling. A short-timeout call that never ran came back as
TOOL_ERROR_CATEGORY_TIMEOUT with Retryable set — a fabricated provider
failure a caller would then retry. Acquire under the caller's ctx, derive
the deadline once the locks are held.

Also: add Config.Clock, matching plangate/hookdispatch/sessionstate/
modelcall, and read it once per persisted event so an event's ULID
timestamp and its Timestamp column stop being two different instants;
route both payloads through statebackend.MarshalPayload; and thread a
spanErr so the two persist-failure paths stop ending their span OK.
Live documented itself as a session's sole writer and republished every
committed event onto kernel.event.{kind}, but the composition root bound
the raw *statebackend.Session it wraps into the turn stack's sink. So the
five kernel collaborators wrote straight to sqlite: past Live.mu, and past
the republish. A plugin subscribed to kernel.event.* saw other plugins'
Emit calls and never a message, tool_call, tool_result, plan, or apply —
including kernel.event.message, which event-bus.md names as an example.

Give Live the three Append* methods those collaborators' sink interfaces
already declare. They take an already-built statebackend.Event, so callers
keep owning their event ids and timestamps, and they deliberately do not
debit the budget: internal/session debits once per turn and two debits
would compound silently. Drop the dead EmitMessage/EmitPlan, whose minted
ids and budget debit were the wrong contract for this path, and drop the
Session() accessor that made the old wiring possible.

The user prompt still has nowhere legal to live in events; the blocking
spec gap is now recorded in state-backend.md's open questions.
Copilot AI review requested due to automatic review settings July 25, 2026 19:32

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of files (300). Try reducing the number of changed files and requesting a review from Copilot again.

@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

event-bus.md linked determinism.md as a markdown link into .claude/rules/,
which is not part of the documentation site, so mkdocs build --strict
aborted on the unresolvable target. Every other spec file referencing a
.claude/rules/ document does so as plain text for exactly this reason;
match them.

Pre-existing on main. The Docs workflow is path-filtered on docs/**, so it
only ran once this branch touched a spec file.
Three Windows-only test failures, all in test code — the packages
themselves already use filepath.Join, os.UserHomeDir, and os.Stat, and are
portable as written.

xdg's fallback tests set HOME and expect os.UserHomeDir to follow, but
Windows reads %USERPROFILE%, so those tests silently resolved the runner's
real home directory instead of the temp one. setHome sets both.

plugincache's BinaryPath test compared a filepath.Join result against a raw
"/cache" literal, which only shares a prefix on POSIX; normalize the input
with filepath.FromSlash. Its permission-denied case provokes the error with
a 0o000 directory, which Windows ignores in favor of ACLs — skipped there,
since the behavior under test is real everywhere but that way of producing
it is not.

Latent since these packages landed; the test matrix runs Windows but this
branch had only ever been verified locally on Linux.
@scrothers
scrothers merged commit c8c08e5 into main Jul 25, 2026
15 checks passed
@scrothers
scrothers deleted the integration/runtime-kernel branch July 26, 2026 04:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants