Implement the runtime kernel: agent loop, session supervisor, and the first reference plugin - #9
Merged
Merged
Conversation
Implements XDG Base Directory layout resolution for the kernel, per architecture.md#xdg-layout. Resolves paths for project config, global config, plugin cache, persistent data, and session state. - Pure-domain package: I/O-free, deterministic, ~95% test coverage - No logging/telemetry per logging-telemetry.md exemption - Comprehensive table-driven tests with race/shuffle support - Full documentation: doc.go, README.md, CLAUDE.md
The agent loop's opt-in explicit terminal-tool done detection (agent-loop/turn-algorithm.md#done-detection) is specified as a `terminates_turn: bool` schema annotation, but ToolSchema carried no wire field for a provider to declare it, so the path was unimplementable. Add `bool terminates_turn = 11` (next free field number) and document it across the tool specs: a new data-types.md section covering the resource-only constraint and the "layers on top of, never replaces, implicit no-tool-calls detection" rule, the full-shape block and prose in protocol.md#getschema, a MAY row in the conformance matrix, and a cross-reference from turn-algorithm.md to where the field now lives. Purely additive: no field number reused, no existing field changed, so `buf breaking` stays clean and pkg/common.ProtocolVersion is unchanged.
content.v1.RedactedThinkingBlock has existed as a canonical content-block type since the schema was written, but StreamEvent had no variant able to produce one — so a model plugin had no way to surface a vendor-encrypted reasoning block, and a vendor that requires such blocks be echoed back verbatim rejects the entire conversation on the following turn. Add `RedactedThinking redacted_thinking = 10` (next free oneof field number) carrying a single opaque `bytes data`, mirroring RedactedThinkingBlock. Unlike ThinkingDelta this is whole-block, not incremental: the payload is opaque, so there is nothing to accumulate. Document the variant in the model spec's StreamEvent block, expand the canonical-content-block prose to cover it, and add a conformance row. Purely additive: no field number reused, no existing variant changed, so `buf breaking` stays clean and pkg/common.ProtocolVersion is unchanged.
Close gaps where docs/specifications already reference a kernel-configurable
knob with no agent.hcl field to set it:
- settings.event_bus{} with subscribe_queue_bound (default 1024).
- settings.doom_loop{} with window_size/threshold, defaulted from
internal/doomloop.DefaultConfig so the numbers have one source of truth.
- settings.default_hook_timeout_ms / default_tool_timeout_ms, flat
attributes defaulting to 5000/30000.
- settings.max_depth as a *int, carried as declared rather than defaulted
here — agentprofile.RootRemainingDepth already resolves the unset case.
- hook{}'s optional timeout_ms per-subscriber override.
Both defaulting paths now share defaultSettings() so decode() and
decodeSettings cannot drift. Adds TelemetryConfig, bridging Settings into
internal/telemetry.Config; telemetry = false forces the noop backend
regardless of observability{}'s contents.
The fixed pkg/common.CallbackBrokerID is collision-free only while broker.AcceptAndServe is called exactly once per launched subprocess. That guarantee was held by a sync.Once on categoryPlugin — which is per-category, not per-subprocess, so a launch whose plugin map carries more than one category could race several goroutines onto the same fixed broker ID. Lift the Once, the callback server, and the telemetry provider onto a new launchScope, shared by reference across every categoryPlugin built for one Launch call, and make pluginMap variadic over categories. Single- category launches behave exactly as before; the multi-category shape is the primitive a later dev_overrides category probe needs, which keys one subprocess by all seven categories because the binary's real category isn't knowable ahead of time. TestLaunchScope_serveOnce drives the once-guarded core concurrently through a seven-entry plugin map and asserts a single serve results.
agent-loop/hook-dispatch.md requires the kernel dial HookSubscriberService "on the same connection it already holds to that plugin's category service" — go-plugin muxes several gRPC services over one subprocess connection — but that connection did not survive past categoryPlugin.GRPCClient, so no second service could be reached. Record it on the launch scope, keep it on Plugin, and add Plugin.HookClient. HookSubscriberService is the one service in the protocol that is category-agnostic, so naming it costs this package none of the category knowledge Dispensed()'s any return exists to avoid; the raw *grpc.ClientConn stays unexported because go-plugin owns and closes it. The fixture's hook.Observer now logs back through the kernel callback, so TestLaunch_hookClientSharesCategoryConnection proves a DispatchHook issued through HookClient reached that subprocess over the same connection its ToolServiceClient came from, and dies with it on Close.
Add NewEventID, EventKindText, and EventPayloadType for callers outside
this package.
NewEventID shares NewSessionID's mutex and monotonic ULID entropy source
via a common newULID, so the two can never mint the same value in one
millisecond and there is a single canonical event-id format.
EventKindText is a thin wrapper over encodeEventKind rather than a second
switch. EventPayloadType is the kind -> pluggableharness.event.v1 message
table from state-backend.md's kind enum, which had no code representation
before; a later Emit implementation needs both to build the reserved
kernel.event.{kind} bus topic and BusEvent.payload_type.
WAL keeps a concurrent reader from blocking the kernel's writes as a steady state, but another process opening the same file briefly takes an exclusive lock while sqlite initializes the -shm/-wal sidecars. With no busy timeout a write landing in that window failed outright with SQLITE_BUSY, which is the outcome state-backend.md's ordering and concurrency section rules out. Request busy_timeout(5000) via the DSN, alongside foreign_keys, so a replacement pooled connection inherits it too. Surfaced as a flaky TestSession_concurrentReaderDuringWrites under a loaded test suite.
Add Session.EventsMatching and the EventQuery it takes — kinds, from_sequence, and limit, mirroring kernel-callbacks.md's ReadEvents request field for field, so answering "the last N tool_result events" does not mean reading a whole session log into memory. Events is now EventsMatching with a zero-value query: one read path, one SQL builder, no unfiltered fast path that can drift from the filtered one. The existing Events tests are unchanged and cover the equivalence, alongside an explicit field-for-field comparison of both call shapes over the same session. The IN (...) list is built by walking EventQuery.Kinds in caller order with a seen map used only to skip duplicates, never by ranging a map, so the rendered statement can never depend on Go map iteration order. A negative limit is rejected rather than passed to sqlite, which reads one as "no limit".
plan and apply events are assembled by the kernel from a turn's tool calls spanning several tool providers, so no plugin owns them as producer — yet events.producer_category/name/version are all NOT NULL. KernelProducer returns the fixed identity for exactly those two kinds: name "kernel", version "1" (the event.v1 payload generation, not a kernel release, so a session's producers manifest is stable across upgrades), category CATEGORY_UNSPECIFIED, empty source. Storing it is a composite decision, never a category one: encodeProducer accepts the unspecified category only when the name is exactly "kernel" and the kind is plan or apply, and decodeProducer resolves the reserved "kernel" category text only when paired with that name. encodeProducerCategory still rejects CATEGORY_UNSPECIFIED outright and the reserved text is absent from both category tables, so a real plugin producer's path is unchanged and CATEGORY_UNSPECIFIED gains no new way to reach a written row. hook_error stays out: the spec has its producer columns identify the failing subscriber, not the kernel.
newCallbackServer never passed these through, so any real launched plugin calling Emit/ReadEvents/GetSession/CountTokens would nil-pointer-panic inside kernelcallback once those RPCs stopped being stubs. Made all three MUST-be-set in Config.validate(), matching kernelcallback's own convention. Also fixes a test that called ReadEvents with a nil stream, which its former Unimplemented stub tolerated but the real implementation does not.
Compose internal/contextassembly, hookdispatch, modelrequest, modelcall, plangate, and tooldispatch into the numbered algorithm in docs/specifications/agent-loop/turn-algorithm.md. The package owns no algorithm of its own: it contributes the documented order, the declaration-order bookkeeping that keeps every tool_result paired with its tool_use block across the kind split, and the adapters that let plangate keep declaring its own HookDispatcher and ApplyOutcome instead of importing hookdispatch and tooldispatch. Steps 16-18 and the session lifecycle stay out: Result hands the future session driver the call hashes, spend, tripped providers, and done status those checks need. The conformance test runs a two-turn scenario through five hand-written fakes sharing one ordered call log and asserts the sequence exactly, which is conformance.md's first MUST.
Adds internal/session: the kernel's outer loop around internal/turn — profile resolution (including the implicit default profile), model routing and tool-scope expansion against the loaded provider catalog, session creation and terminal-status persistence, session-lifetime callback grants, session-start/session-end dispatch, and steps 16-18 of turn-algorithm.md. All three bound dimensions, the doom-loop detector, and the plan-gate circuit breaker route through one limit-reached path: exactly one final-answer turn naming what fired. Doom-loop and breaker trips map to SESSION_STATUS_COMPLETED with the reason on Result.FinalAnswerReason, since session.v1.SessionStatus has no subtype for either.
StartProviderCatalogBuild wraps providercatalog/drivers/plugin.New's extraction pass; StartToolPreview's doc comment now covers its second call site, the one-time SupportsPreview probe that driver performs.
Wraps a *pluginhost.Registry to satisfy providercatalog.Catalog against real, live plugin subprocesses. Extraction is eager (safe because a Registry is only mutated by Supervisor.Start, which completes before New is ever called). ToolHandle.SupportsPreview is resolved by a one-time live Preview probe per operation, checking specifically for codes.Unimplemented per doc.go's reasoning. ContextHandle.Position is set directly from Live.LaunchIndex, confirmed to already be agent.hcl declaration order. ContextHandle.TokenBudget cannot reflect an agent.hcl override yet — pluginhost retains no decoded provider config outside Configure — documented as a known gap rather than guessed at. Corrects providercatalog/CLAUDE.md's stale note anticipating a drivers/drivers.go selector once a second driver landed: drivers/fake is a test double, never a second production option, so no selector is warranted.
ValidatePricing accepts free = true with no tiers at all, but modelcall's persist resolved a tier unconditionally, so every completion from a legally-declared free provider failed with ErrNoMatchingTier before its message was ever persisted. Add cost.IsFree for exactly that shape and short-circuit on it. A free Pricing that also declares tiers is unaffected: those tiers are validated like any other and must still be resolved.
The composition root builds the turn stack's collaborators over the same sole-writer *statebackend.Session that Live wraps, rather than opening a second handle on the same file. Nothing above internal/session can reach that handle otherwise, since Runner.Run creates it. Emit/EmitMessage/EmitPlan remain the only path for plugin-originated events: they debit the budget tracker and republish onto the bus.
Config gained required Scopes/Sessions/Tokens fields; the integration harness was never updated, so every test in it failed NewSupervisor validation.
internal/kernel.Run brings the process up in dependency order, runs exactly one non-interactive session, prints its final message, and tears every phase back down in reverse — including after a bring-up failure. cmd/agent parses flags, calls it once, and maps the result to an exit code. The per-session half of the turn stack is built lazily on the first turn: plangate needs the session id at construction and five packages need the open session handle, but internal/session mints both inside Runner.Run. A sessionSink slot bridges the gap, mirroring pluginhost's callbackSlot. Both frontend-absent deviations are wired loudly at one call site: autoallow for ask-decisions, unattended for interactive calls.
Translates a kernel StreamCompletionRequest + model.Spec into Anthropic's request body: content-block translation with capability gates, message coalescing, cache-breakpoint placement, thinking/effort handling, and deterministic JSON Schema/tool-argument serialization.
Several event.v1 payloads reach a structpb.Struct — ToolCallEvent through ToolCall.arguments, ToolResultEvent through ToolResult.payload and ToolError.details, MessageEvent through every ToolUseBlock.arguments, ContextContributionEvent through contributed content blocks. A proto map marshals in randomized order unless Deterministic is set, so the same session persisted to different bytes on every run, which determinism.md forbids for any persisted payload. Only internal/plangate set the option, via a package-local helper. Promote it to statebackend.MarshalPayload as the single source of truth and route modelcall, contextassembly, and plangate itself through it. tooldispatch's two call sites move over in the following commit, alongside its clock.
ToolSchema.default_timeout is documented as the Invoke deadline, but the context carrying it was derived before acquireLocks and passed into it, so its clock ran while a call sat queued behind an exclusive safe:false sibling. A short-timeout call that never ran came back as TOOL_ERROR_CATEGORY_TIMEOUT with Retryable set — a fabricated provider failure a caller would then retry. Acquire under the caller's ctx, derive the deadline once the locks are held. Also: add Config.Clock, matching plangate/hookdispatch/sessionstate/ modelcall, and read it once per persisted event so an event's ULID timestamp and its Timestamp column stop being two different instants; route both payloads through statebackend.MarshalPayload; and thread a spanErr so the two persist-failure paths stop ending their span OK.
Live documented itself as a session's sole writer and republished every
committed event onto kernel.event.{kind}, but the composition root bound
the raw *statebackend.Session it wraps into the turn stack's sink. So the
five kernel collaborators wrote straight to sqlite: past Live.mu, and past
the republish. A plugin subscribed to kernel.event.* saw other plugins'
Emit calls and never a message, tool_call, tool_result, plan, or apply —
including kernel.event.message, which event-bus.md names as an example.
Give Live the three Append* methods those collaborators' sink interfaces
already declare. They take an already-built statebackend.Event, so callers
keep owning their event ids and timestamps, and they deliberately do not
debit the budget: internal/session debits once per turn and two debits
would compound silently. Drop the dead EmitMessage/EmitPlan, whose minted
ids and budget debit were the wrong contract for this path, and drop the
Session() accessor that made the old wiring possible.
The user prompt still has nowhere legal to live in events; the blocking
spec gap is now recorded in state-backend.md's open questions.
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
event-bus.md linked determinism.md as a markdown link into .claude/rules/, which is not part of the documentation site, so mkdocs build --strict aborted on the unresolvable target. Every other spec file referencing a .claude/rules/ document does so as plain text for exactly this reason; match them. Pre-existing on main. The Docs workflow is path-filtered on docs/**, so it only ran once this branch touched a spec file.
Three Windows-only test failures, all in test code — the packages themselves already use filepath.Join, os.UserHomeDir, and os.Stat, and are portable as written. xdg's fallback tests set HOME and expect os.UserHomeDir to follow, but Windows reads %USERPROFILE%, so those tests silently resolved the runner's real home directory instead of the temp one. setHome sets both. plugincache's BinaryPath test compared a filepath.Join result against a raw "/cache" literal, which only shares a prefix on POSIX; normalize the input with filepath.FromSlash. Its permission-denied case provokes the error with a 0o000 directory, which Windows ignores in favor of ACLs — skipped there, since the behavior under test is real everywhere but that way of producing it is not. Latent since these packages landed; the test matrix runs Windows but this branch had only ever been verified locally on Linux.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Builds the runtime kernel — the thing every existing package was for. Before this,
docs/specifications/was settled, the sqlite state backend and thepkg/**plugin-author SDK had shipped, and roughly 21k lines of tested Go existed, but there was no running kernel: nocmd/binary, no agent loop, no session supervisor, no real plugin. Everything that existed was a leaf —internal/policyevaluated rules nothing called,internal/pluginruntimelaunched subprocesses nothing chose,internal/statebackendpersisted sessions nothing created — andinternal/kernelcallbackcarried fivecodes.Unimplementedstubs blocked on machinery that had never been built.This adds the trunk, plus the first real plugin to prove it end to end.
./bin/agent -prompt "…"now loadsagent.hcl, launches provider plugins, runs the full 18-stepRunTurnloop through the plan/apply gate, and persists a real session file.~40 new packages, delivered in dependency order:
slog/telemetry, ~95% covered):xdg,plugincache,sessionscope,callhash,cost,bounds,doomloop,retrypolicy,schemavalidate,circuitbreaker,hookpayload,streamaccum,modelrequest,tokencountplandecision,interactive,providercatalogsessionstate,hookdispatch,tooldispatch,modelcall,contextassembly,plangate,pluginhost,providerresolveturn(the 18-step algorithm),session(lifecycle, turn loop, bounds/doom-loop/breaker),kernel(composition root)internal/anthropic(+catalog,messages) andcmd/anthropic, hand-rolled onnet/httpso no vendor SDK enters the module graph every plugin author inheritscmd/agent,cmd/anthropicTwo additive, non-breaking proto changes:
ToolSchema.terminates_turnandStreamEvent.redacted_thinking. Both go throughbuf generate;buf breakingis unaffected andProtocolVersionis deliberately not bumped.Checklist
go mod tidyis a no-op,go build ./...,go vet ./...,gofmt -l -s .prints nothing,go test -race -covermode=atomic ./...,golangci-lint rundocs/specifications/document in this same PRpkg/*/proto/v1/—.protochanges made inapi/and regenerated withbuf generateinternal/packages includeREADME.md+CLAUDE.mdin the same commitbin/Also run green locally:
go test -race -tags=integration ./...(74 packages),gosec -exclude-generated,govulncheck,buf lint,buf format --diff --exit-code.Notes for reviewers
Two tracked deviations from a spec MUST — please read these first
Both exist because no frontend plugin category is built yet, and both are wired at a single call site (
internal/kernel/turnstack.go'snewTurnDriver) so they are impossible to miss in review.askplan decisions are auto-approved.plan-apply-gate.md#decision-semanticsrequires anaskto emit apermission-requestand block until a human answers. There is nothing to ask.internal/plandecision/drivers/autoallowhas no usable zero value — construction fails unlessAcknowledgeUnsafeAutoAllow: true— logs a WARN at construction and one per resolution, resolves onlyONCE(neverSESSION/ALWAYS, so it can create no durable state a real resolver would have to reconcile), and permanently persistsplan_items.decided_by = "UNSAFE-AUTO-ALLOW(no-frontend-attached)". In plain terms: a session run by this build executes mutating tool calls a human was supposed to approve, and the audit trail says so per item. The fix is the realdrivers/frontendresolver, which drops in behind the same one-methodResolverinterface with no call-site change.interactive-kind calls are auto-refused.internal/interactive/drivers/unattendedreturns aPERMISSION_DENIEDtool result the model can observe and adapt to. The asymmetry with (1) is deliberate: anaskitem has a defensible default (the call as proposed); an interactive call's whole payload is a human's answer, and any synthetic one would be a fabrication in the model's own history.Scoped non-conformance
Cost rollup is single-session.
turn-algorithm.md#cost-accountingrequires rollup across a session tree; this build is root-sessions-only by agreed scope.bounds.Trackercarries the parent link from day one (NewTracker(limits, parent),Debitwalks to the root) and is unit-tested against a synthetic parent even though production always passesnil, so nothing here gets rewritten when sub-agents land.Spec gaps resolved by documented interpretation, not invention
Each is recorded in the owning package's
CLAUDE.mdwith its reasoning, and each is the kind of thing worth disagreeing with if you read it differently:hook-dispatch.mduses the phrase three times and never enumerates the set. Resolved as{plan-ready, pre-tool-call}— the two points immediately preceding a blockable action.NewRegistryrejects a veto subscription anywhere else..hclfiles.agent-profiles.md's textual-position rule assumes one file; the XDG layout permits several. Resolved as (lexicographic filename, byte offset) — never filesystem enumeration order.SessionStatusof their own, so both terminateCOMPLETEDwith the reason onResult.FinalAnswerReason. Reusing anerror_max_*subtype would makesession_metalie about which limit fired. A dedicated status is proposed for a later revision.pre-tool-callneeds aplan_itembefore aPlanexists — resolved by minting a provisionalPENDINGitem at step 7 and carrying it forward by identity into step 10, so the gate stamps a decision onto the same pointer a hook already saw.categoryfield, falling back to a seven-keyDescribeprobe.One spec gap this PR does not resolve
A session's own user prompt has nowhere legal to live in
events, so replay reconstructs every assistant message, tool call, and plan but not the prompt that caused them.producer_categoryis the seven plugin categories;state-backend.mdsays every kind buthook_erroris written by the producing plugin's own callback connection; and the reservedkernelproducer is restricted toplan/apply.messageis also the wrong shape — its payload carries usage/cost, extracted tocost_ledgerat write time, and a prompt has neither. Fixing it is wire-visible (widen the kernel producer for a user-authoredmessage, or give the user turn its own kind), so it is recorded instate-backend.md's Open questions rather than decided here. A local workaround — fabricated producer, zero-cost ledger row — would put a lie in the audit log.Deliberately not built
No
RunSession/session tree/sub-agent spawning (parent-link seams already exist inbounds.Tracker,sessionstate.Live); no real frontend; nointernal/registryprovider download or lock-file writing — bootstrap isdev_overridesplus already-cached binaries, and an unresolvable provider reports an aggregatedMissingErrorrather than hanging.Worth close reading
internal/turn/runturn.go— declaration order is the invariant everything else bends around. Onependingslice built at step 7, never reordered; every grouping is a slice of pointers into it.post-tool-calldispatches andtool_resultblocks must both emerge intool_useorder, not grouped by kind or completion order.internal/tooldispatch— provider semaphore first, per-key second, always. That single fixed order is what makes the two-level scheme deadlock-free by construction.internal/kernel/turnstack.go— the late-binding seam.plangateneeds a session id at construction and oneBreakermust be shared withtooldispatch, butinternal/sessionmints the id insideRun, which is called with an already-built driver. Resolved withturnStack+sessionSink, modeled onpluginhost'scallbackSlot.internal/anthropic— held to exactly what a third party gets: adepguardrule forbids it importing any otherinternal/package. That constraint is the point (it is what provespkg/is sufficient), and it is also why this package has no spans or metrics — nopkg/surface exposes telemetry to a plugin author yet. Building it caught a real SDK gap (Sinkcould not emitredacted_thinking), fixed inpkg/modelrather than worked around.Review-driven fixes included
The last three commits are fixes from a review of the rest of the branch: persisted event payloads were marshaled without pinned proto map ordering (violating
determinism.md— now centralized instatebackend.MarshalPayload); no kernel-originated event reached the event bus because the composition root bound the raw session handle pastsessionstate.Live's republish; a tool'sdefault_timeoutwas charging lock-queue time and could report a fabricated retryable TIMEOUT for a call that never ran; plus a missing clock injection, a span that always ended OK, and a circuit-breaker debit that preceded its own audit write.