Add kernel state backend (internal/statebackend) - #3
Merged
Conversation
Add package doc, sentinel errors, and ULID session-id helpers with unit and fuzz coverage, per docs/specifications/state-backend.md. Adds github.com/oklog/ulid/v2.
Implement the five-table DDL verbatim from state-backend.md, PRAGMA user_version handling with per-step transactional migrations, and the Store lifecycle: Create, Open (version check before any table access, ErrSchemaTooNew on newer files), List and Children via metadata-only directory scans that never migrate as a side effect. Adds modernc.org/sqlite as a direct dependency.
Implement sole-writer append transactions: AppendEvent with same-tx producer upsert, AppendMessage with cost_ledger row, AppendPlan with plan_items rows, SetStatus, and idempotent Close with WAL checkpoint. Event kinds, session statuses, plan decisions, and producer categories encode to the spec's lowercase text vocabulary; unspecified enum values are rejected. Add StartStateBackend* span helpers to internal/telemetry and instrument Create/Open. Race tests assert dense 1..N sequences under concurrent appenders.
Implement the read-side API: sequence-ordered Events replay iterator, deduplicated Producers set, TotalCostUSD, CostLedger, PlanItems, and Meta. Wire Open through a real integrity check: corrupt or unopenable files are renamed to .corrupt (never deleted), salvaged table-by-table into a fresh schema-correct file preserving sequence values, with slog WARN reporting; unsalvageable files return ErrUnrecoverable. Add an event round-trip fuzz target and query/recovery span helpers.
Expand the package godoc with the design decisions the spec left open, add README and package CLAUDE.md, enumerate producer_category's text vocabulary in state-backend.md, and fix forward the stale parent-to- children index claims in subagents.md, conformance.md, and turn-algorithm.md: parent linkage lives in session_meta, child lookup is a session_meta scan, and cost-rollup, depth threading, and cancellation are live in-memory mechanisms, not state-backend reads.
Pre-create the recovery scratch file at 0600 so recovered session files keep the documented mode instead of SQLite's 0644 default, move foreign_keys enforcement into the DSN so every pooled connection gets it rather than only the first, and chain the salvage failure cause through ErrUnrecoverable so callers keep both errors.Is matching and the underlying reason.
Dependency ReviewThe following issues were found:
License Issuesgo.mod
OpenSSF Scorecard
Scanned Files
|
There was a problem hiding this comment.
Pull request overview
This pull request introduces the kernel’s non-pluggable sqlite-per-session state backend per docs/specifications/state-backend.md, including schema/versioning, append/query APIs, corruption recovery, and end-to-end test coverage. It also extends kernel telemetry spans for state-backend operations and aligns several specifications with the “scan session_meta” tree-walking model.
Changes:
- Added
internal/statebackendimplementing per-session sqlite persistence (schema, migrations, append/query APIs, integrity-check + salvage recovery). - Added state-backend tracing spans to
internal/telemetry. - Updated specifications and module dependencies to support the new backend.
Reviewed changes
Copilot reviewed 26 out of 27 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| internal/telemetry/span.go | Adds tracing span names and StartStateBackend* helpers for state backend operations. |
| internal/statebackend/statebackend.go | Implements Store and SessionMeta plus create/open/list/children and metadata scanning. |
| internal/statebackend/statebackend_test.go | Tests Store behaviors (create/open/list/children), timestamp/status encoding, and openDB pragmas. |
| internal/statebackend/sessionid.go | Adds ULID-based session ID generation and canonical validation. |
| internal/statebackend/sessionid_test.go | Unit tests for session ID generation, ordering, concurrency uniqueness, and validation. |
| internal/statebackend/sessionid_fuzz_test.go | Fuzzes ValidateSessionID for robustness and round-trip properties. |
| internal/statebackend/session.go | Implements write-path APIs (AppendEvent/Message/Plan, SetStatus, Close) with transactional guarantees. |
| internal/statebackend/session_test.go | Tests write-path behavior, rollback semantics, WAL read/write concurrency, and closed-session behavior. |
| internal/statebackend/schema.go | Defines the spec-verbatim schema and migration scaffolding using PRAGMA user_version. |
| internal/statebackend/schema_test.go | Tests schema initialization and migration transactionality/versioning behavior. |
| internal/statebackend/README.md | Package-level README documenting responsibilities, API sketch, and recovery/testing notes. |
| internal/statebackend/query.go | Implements read-path APIs including replay iterator (Events) and structured queries. |
| internal/statebackend/query_test.go | Tests replay/query methods including ordering, decode errors, and totals/ledgers. |
| internal/statebackend/integrity.go | Implements PRAGMA integrity_check on open and salvage recovery into a new file. |
| internal/statebackend/integrity_test.go | Tests healthy-open, recoverable corruption, unrecoverable cases, and recovery artifact handling. |
| internal/statebackend/event.go | Defines event/cost/plan item types and enum/text encoding/decoding helpers. |
| internal/statebackend/event_test.go | Tests enum encode/decode round-trips and invalid-value rejection. |
| internal/statebackend/event_fuzz_test.go | Fuzzes event append/read round-tripping including payload and timestamp truncation behavior. |
| internal/statebackend/errors.go | Adds typed sentinel errors for state backend operations. |
| internal/statebackend/doc.go | Adds package godoc describing design decisions and how the backend is used. |
| internal/statebackend/CLAUDE.md | Adds agent notes capturing key invariants (spec-verbatim schema, append-only, ordering authority, recovery rules). |
| go.mod | Adds required dependencies for sqlite + ULID support. |
| go.sum | Updates dependency checksums for new direct/transitive deps. |
| docs/specifications/state-backend.md | Clarifies producer_category vocabulary in the schema section. |
| docs/specifications/agent-loop/turn-algorithm.md | Updates cost-rollup wording to reflect in-memory live tree state, not state-backend index. |
| docs/specifications/agent-loop/subagents.md | Updates session-hierarchy bookkeeping and cancellation propagation to the scan-based/no-index model. |
| docs/specifications/agent-loop/conformance.md | Updates conformance table to “session_meta scan (no separate index)”. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Unix mode bits aren't meaningful on Windows: created files report 666, directories 777, and read-only directory modes don't block file creation. Guard the permission assertions per-OS, skip the read-only directory test on Windows, close the session on an unexpected Create success so TempDir cleanup can't wedge on an open handle, and note in the package doc that the 0700/0600 guarantee is Unix-only.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
docs/specifications/state-backend.md: sqlite-per-session persistence with the five-table schema reproduced verbatim (append-onlyevents,session_meta,cost_ledger,plan_items,producers),PRAGMA user_versionschema versioning with per-step transactional migrations, and ULID session files that sort chronologically.modernc.org/sqlitekeeps the three-OS race CI cgo-free; sole-writer semantics are enforced mechanically viaSetMaxOpenConns(1)so commit order is sequence order.AppendEvent/AppendMessage/AppendPlaneach run in one transaction with their cost/plan/producer rows;session_metais the only mutable table. Read side: sequence-orderedEventsreplay iterator, deduplicatedProducersset,TotalCostUSD,CostLedger,PlanItems.Open:PRAGMA integrity_check, damaged files renamed to.corrupt(never deleted), table-by-table salvage into a fresh 0600 schema-correct file preserving sequence values, noisyslogWARN reporting,ErrUnrecoverable(with wrapped cause) when salvage fails.StartStateBackend*span helpers added tointernal/telemetry, wired through all I/O paths.producer_categoryvocabulary enumerated instate-backend.md, and the stale "queryable parent→children index" claims insubagents.md,conformance.md, andturn-algorithm.mdcorrected to the authoritativesession_metascan / live in-memory tree model.Test plan
go build ./...,go vet ./...,gofmt -l -s .,go mod tidyno-op — cleango test -race -shuffle=on -covermode=atomic ./...— pass;internal/statebackendat ~86% coverage (floor 80%), including concurrent-appender sequence-density race tests, WAL concurrent-reader tests, corruption/recovery tests, and migration testsFuzzValidateSessionIDandFuzzEventRoundTrip— 10s runs, no crashesgolangci-lint run— 0 issues;gosec -exclude-generated ./...— 0 issues;govulncheck ./...— clean;buf lint/buf format --diff— clean