Make the model plugin surface sufficient for out-of-tree providers - #14
Merged
Conversation
A third-party model provider currently cannot ship any vendor feature the kernel's proto has not already named, because every request field is strongly typed and adding one requires a change to this repository. That makes the out-of-tree provider surface effectively closed. Add StreamCompletionRequest.provider_options, a Struct the kernel passes through untouched — vendor knobs it has no semantics for, such as a service tier, a sampling seed, or a beta-feature flag. This extends proto.md's existing Struct precedent list rather than adding a third opaque-bytes carve-out: the justification is the same one ConfigureRequest.config already carries — the shape is the provider's schema, not the kernel's to name — applied per-request instead of once at configure time. Record the rule that keeps the list from becoming a general escape from strong typing: a Struct field is pass-through only, so a value the kernel branches on (routing, capability validation, cost, replay) must be a typed field or it does not work at all. Prompt-cache TTL is the worked example, since the kernel computes cost_usd and the TTL changes the rate. pkg/model gains Options, a nil-safe view with os.LookupEnv-style (value, ok) accessors, so each provider does not hand-roll structpb traversal. LookupInt64 rejects values that are not exactly representable rather than truncating an operator's value silently.
ThinkingMode was a single mutually-exclusive enum, so a model occupying
more than one position could only declare a half-truth. Three places in
this repository had already written that down:
- catalog.go calls Fable 5's declaration "a deliberate choice between
two modes that each capture half the truth" — it reasons adaptively,
cannot be disabled, AND exposes the full effort ladder.
- catalog.go calls Opus 5's conditional disable "a caveat the
ThinkingSpec shape cannot express", settling for the answer that is
"the larger lie".
- docs/first-party/providers/anthropic.md tells adapter authors to
"pick one canonical mode to declare" for Sonnet 4.6, whose second
mechanism is "not directly representable in a single ThinkingSpec".
buildThinking is the proof: for DISCRETE_EFFORT it already emitted both
thinking:{type:"adaptive"} and output_config.effort, because Anthropic's
effort ladder rides on top of adaptive reasoning rather than replacing
it. The enum named a model family, not a capability.
Replace it with four independent axes — an optional EffortControl, an
optional BudgetControl, adaptive_by_default, and a disable enum whose
CONDITIONAL value carries the Opus 5 case honestly. Each control owns its
own default, which also answers conformance.md's open question about a
per-model budget default separate from the range bounds.
Validation follows the axes: each generation param is checked against the
control that governs it, so a model declaring both accepts both. The
all-zero ThinkingSpec stays valid for a model that does not reason, since
that is the common case; only a positive claim that reasoning can be
turned off contradicts supported == false.
The Anthropic roster is re-expressed in the new shape without adding any
vendor claim it did not already make. Notably it still declares no budget
control on the 4.6 generation: the research suggests those models honor a
deprecated budget_tokens, but this roster never claimed it and that claim
needs its own pass against the live docs.
Also records an open question this surfaced: several vendors' reasoning
models reject temperature, and adapters currently infer that from the
thinking shape. The proxy holds today and will break on the first model
with an effort ladder that still accepts sampling params.
CachingMode forced explicit_markers and implicit_automatic to be mutually exclusive, but Gemini 2.5 and later run implicit automatic caching by default AND offer explicit manual declaration concurrently, at a deeper discount. docs/first-party/providers/google.md had already escalated this: "the current CachingSpec shape has no way to declare 'this model supports two caching modes concurrently, with different discount rates', which is precisely the situation Google's own docs describe. This is worth flagging back to the protocol's designers as a real gap." The gap had teeth beyond declaration accuracy. cache_breakpoints were gated on the enum naming EXPLICIT_MARKERS, so such a model — declaring the mode that accurately described its default behavior — was thereby required to discard breakpoints it could in fact have honored, losing a cache discount it was eligible for with no error anywhere. Replace the enum with two independent bools and gate breakpoints on the explicit_markers axis alone. Declaring caching now requires naming at least one mechanism, since declaring neither would read as "no caching" to every caller. Also records the pricing gap this exposes, as an open question rather than a fix: PricingTier carries one cache-rate pair, but a model can bill cached tokens at several rates depending on which mechanism served the request — Anthropic's 5-minute vs 1-hour TTLs, Gemini's implicit vs explicit discounts. That deliberately did not become a provider_options knob, because the kernel computes and persists cost_usd, so a rate- changing value riding in a pass-through field would produce silently wrong ledger rows forever. Fixing it means making the rate a function of the mechanism used, which is a Pricing redesign rather than a field.
CountTokensRequest carried a flat `text` field, but every vendor that exposes exact counting counts a whole request: Anthropic's /v1/messages/count_tokens takes messages plus system plus tools and returns that request's input-token total. The old shape could not express the question the kernel actually asks. Worse, it discarded the two things most likely to dominate the answer — tool schemas and the system preamble — so a count omitted exactly the weight that decides whether a turn fits in the context window. The round trip was lossy twice over: internal/tokencount flattened content blocks into a joined string, and the Anthropic client then re-wrapped that string back into a single user message to satisfy the endpoint. Replace it with messages + assembled_context + tools + model_id, mirroring StreamCompletionRequest's content-bearing fields minus everything that only affects generation. A caller holding loose content passes it as one user message, which is what the adapter had to construct anyway. BuildCountTokensRequest runs the same buildSystem/buildTools/ translateMessage path BuildRequest does, so a count is computed over exactly the content a completion would have carried; any divergence would make the count silently unrepresentative of the request it sizes. One deliberate consequence: internal/tokencount now forwards every block, not just the text ones, so the exact path and the ceil(bytes/4) fallback no longer measure the same input. That asymmetry is the correct one — the vendor charges for an image, so including it makes the exact count more accurate, while the fallback keeps under-estimating non-text content. That package's CLAUDE.md previously required the two to stay symmetric and now records why they no longer are.
Two things the kernel needs to read but had no field for. Usage.rate_limits reports the vendor's own rate-limit budgets. Without it the kernel cannot tell a user which ceiling stopped their session, and "you have 2% left" is unactionable without saying 2% of what. It is repeated because vendors publish several budgets at once and they exhaust independently — OpenAI and xAI return separate request and token headers, Anthropic reports input and output separately. Every numeric field is optional so an adapter reports the subset its vendor actually returned; synthesizing a snapshot from the adapter's own bookkeeping is forbidden, since a guessed budget is worse than none. StreamEvent.stream_start carries the vendor's request id. It is a separate early event rather than a field on stop because an id that arrives only on successful completion is absent in exactly the case it is needed — correlating a failure with the vendor's own logs. The Anthropic client emits stream_start from response headers, before any content streams. Reading a header rather than the body is what makes it safe to be best-effort: the spec permits omitting the event, so a header name that stops matching costs a correlation id, never a request. The e2e tier asserts the id is actually present, which is the only tier that can prove the vendor publishes it at all. A third field from this group is deliberately absent. GenerationParams gained no response_format: nothing in the kernel produces or consumes structured output today, so a typed field would be dead protocol surface and go-style.md forbids speculative generality. Until a caller exists it is provider_options territory, and promoting it to a typed field at that point is exactly the path proto.md's pass-through rule describes.
GetCapabilities required a built-in model list and no network call, which no gateway or locally-served provider can satisfy literally: an aggregator's roster is genuinely dynamic across many upstream vendors, and a local runtime's roster is whatever the operator has pulled. Separate the two things the old wording conflated. The real requirement is per-invocation cost — the kernel may call this before every routing decision — not where the roster originates. So: resolve once in Configure, which already does real work and is already where a bad configuration must fail, then serve every call from memory. A background refresh is allowed but must never block the call, since a stale roster served instantly beats a fresh one that stalls routing. Also records that a credential attribute may only be declared required when every supported deployment needs one. A locally-served runtime on loopback typically has no auth while the same plugin pointed at that vendor's hosted tier does, and a required api_key makes the former impossible to configure at all.
internal/anthropic/CLAUDE.md states the rule this applies: wanting something from internal/ means it belongs in pkg/, so every plugin author gets it. Two pieces of that package were never Anthropic-specific, and all four planned providers would otherwise have rewritten them. pkg/sse decodes SSE frames. The framing is the wire format, not any vendor's dialect, and the one subtlety worth centralizing is that bufio.Scanner's 64 KiB default silently TRUNCATES an over-long line rather than erroring — a corrupted frame that still parses is far worse than a failed read, and real vendor frames routinely exceed it. Writing that bound correctly turned out to need its own care: bufio's effective limit is max(limit, cap(initial)), so a larger initial buffer silently raises the ceiling, which a test now pins. The package yields frames and never interprets one. Vendors disagree about every interpretation — Anthropic treats the payload's own type field as authoritative and ignores event: entirely, while OpenAI-compatible vendors dispatch on event: and end with a literal [DONE] — so a framing package taking a position on either would be wrong for somebody. Both fields are surfaced; the caller decides. pkg/model.ClassifyHTTPStatus maps an HTTP status to the conformance taxonomy. Those are HTTP semantics, and centralizing them makes the two mistakes that actually hurt once instead of per vendor: a retryable 403 burns quota against a request that can never succeed, and a 413 read as a generic invalid request loses the kernel's chance to shrink context and retry. Its 5xx fallback is also what makes a vendor-specific overload code — Anthropic's 529 — classify correctly with no entry. Anthropic keeps only what is genuinely its own: which field decides an event's type, and its error.type vocabulary. Its existing tests pass unchanged, which is the evidence that both extractions preserve behavior.
Nothing in this repository proved that pkg/ is usable from outside this module. internal/anthropic's depguard rule forbids it importing other internal/ packages, which simulates that isolation — but a simulation cannot catch an unexported type leaking through an exported signature, or a pkg/ package that only compiles because something else in the main module already resolved a dependency for it. examples/provider is its own Go module with a replace directive back to the working tree, so CI builds it against pkg/ as it exists in the commit under review rather than the last published release. Go excludes a nested module from the parent's ./... automatically, so it costs the main module's build and test nothing. It is also the reference an out-of-band session starts from, so it demonstrates the things that are easy to get wrong rather than the minimum that compiles: the zero ThinkingSpec/CachingSpec as the valid declaration for a model that does neither, a Configure written to be safely re-callable, cancellation returned unwrapped as normal control flow, provider_options read as a pass-through vendor knob, and a CountTokens that counts tool declarations rather than only message text. Adds the companion check the example cannot make on its own: pkg/ must not import internal/, since that compiles here and fails for every downstream author. pkg/telemetry stays the one sanctioned exception, already documented in its own source. The tidied go.mod is worth reading as output: it shows the real dependency tax a third-party plugin author pays for pkg/ — grpc, go-plugin, hclog, yamux, and the otel SDK.
An out-of-band session building a provider had no way to know it was
correct short of re-reading the spec. pkg/model/modeltest turns
conformance.md's MUST/SHOULD matrix into assertions, in two drive modes
sharing one implementation so they cannot drift: in-process over a real
gRPC round trip, and against a built binary through a real handshake.
The round trip is deliberate. Most of what this checks lives in the
pkg/model service adapter and the wire types — terminal-event
bookkeeping, error-to-status mapping, the conversion layer — and a direct
method call on a Provider would exercise none of it. RunBinary is the
only mode that reaches a plugin's own main() wiring, and the only one
that works on a plugin written in another language.
Assertions produce Findings rather than driving *testing.T. That is what
lets the suite's own tests prove it rejects a bad provider — a
conformance suite that cannot be shown to fail is worth very little — and
what will let a non-test binary reuse it. Skips are reported rather than
omitted, so a check the run could not reach never reads as a pass.
Writing the tests found four real defects, three in the suite itself:
- WithExpectedIdentity could never fail in-process, because Check was
serving the very identity it then compared against. It now serves a
fixed identity and reports the expectation as unverifiable in that
mode, since a check that cannot fail is worse than no check.
- checkCancellation had no bound, so a provider ignoring its context
would hang the suite forever. Worse, the assertion it claimed to make
is impossible from a black-box client: gRPC returns codes.Canceled to
the canceling client whatever the server does, so a provider that
keeps generating is invisible from this side. The check now asserts
only what it can and documents the limit rather than implying
coverage it does not have.
- A BudgetControl whose range admits only zero was accepted.
- The example provider silently accepted image and document blocks on a
model declaring neither, which is exactly the violation the suite
exists to catch. Fixed, and it now exercises the capability gates the
Anthropic roster skips — between the two, every gate is covered.
Both real providers pass: internal/anthropic against a canned in-process
vendor, and examples/provider from its own module, which also proves
modeltest is reachable by a third party.
internal/pluginruntime launches every plugin with PATH/HOME/TMPDIR and
nothing else, deliberately: ambient inheritance would leak every variable
the kernel holds, including secrets meant for other plugins, into every
subprocess. Config.ExtraEnv existed to widen that per launch, but nothing
ever populated it and there was no operator surface at all.
The consequence was a hard block on real providers. A plugin behind a
corporate proxy could not see HTTPS_PROXY, one resolving an ambient cloud
credential chain could not see its SDK's variables, and pkg/telemetry's
Bootstrap — which reads OTEL exporter config from the process
environment — could never be configured, so a plugin could start spans
that went nowhere.
provider{} gains an environment{} block, declared per provider so one
plugin's variables stay invisible to every other. The kernel lifts it out
before decoding the rest against the provider's own ConfigSchema, since
it is the one name in that body the kernel owns and leaving it in would
collide with a provider declaring an attribute of the same name. Entries
are emitted sorted: a subprocess environment assembled in Go map order
differs run to run, which is what determinism.md exists to prevent.
Two things this surfaced that were not obvious up front:
HCL's remain body from PartialContent does NOT hide an already-consumed
block from JustAttributes. validateSensitiveAttrs used JustAttributes, so
the first working version rejected every provider that declared
environment{} at all — including, and especially, the ones carrying a
credential. It now asks for the specific sensitive attributes by name,
and a test pins the secret-plus-environment combination end to end.
The "=" and empty-name checks are unreachable from agent.hcl, because
HCL's own grammar forbids a quoted argument name. They are kept as
defense in depth and tested directly rather than through a fixture, with
the reason recorded so a later reader does not mistake them for a
reachable path.
common.v1.ProducerRef.protocol_version was documented as the go-plugin handshake version, with the rule that bumping it "always accompanies a proto package version bump for that category". That coupling is wrong, and the first v* tag would have frozen it. A handshake-version mismatch rejects a plugin before any category RPC is issued. So under the old rule, a breaking change to the model protocol would bump the shared handshake version and thereby reject every tool, context, and memory plugin ever published — none of which changed — until each was rebuilt. Categories already version independently at the proto level, where a v2 lands alongside v1 rather than replacing it, so the handshake was the one place that made them move in lockstep. Separate them. common.ProtocolVersion now versions only the runtime contract it actually describes: the handshake, the fixed callback broker id, and service muxing. Each category SDK carries its own ProtocolVersion constant, and Describe reports that one. Correctness never depended on the field: a category's proto package version is part of its gRPC service name, so a plugin serving pluggableharness.model.v2.ModelService and a kernel dispensing v1 already cannot match. What the field buys is a clear version error at bring-up instead of an opaque "unimplemented service" on the first real call, and a lock file that records it lets preflightVersionCheck — a documented no-op today — reject a plugin before spawning it at all. The identity test previously asserted the field equalled common.ProtocolVersion, which was the coupling expressed as a test. It now covers two categories reporting different protocol versions, which is the property that has to hold.
Configure was specified as the place a bad configuration must fail, but said nothing about being called more than once. The kernel calls it exactly once at bring-up today, so a provider that only works the first time looks perfectly healthy in production — right up until a credential rotation or endpoint change needs a second call, which is when a failure is most expensive and least expected. State the requirement now rather than when that path is built. A plugin written against the weaker "called exactly once" reading has to be reworked, not merely re-invoked, so the cost of leaving it unstated grows with every provider written in the meantime. The rule is replace-wholesale, not merge: a second call carries the operator's complete intent, so a field absent from it is absent rather than inherited from the first. A provider holding a vendor client rebuilds it here for the same reason — a client built from one configuration alongside a setting from another is silently inconsistent, and surfaces as vendor errors that look like anything but a config problem. The conformance suite now calls Configure twice, which is the entire check. internal/anthropic and examples/provider both already satisfied it; a deliberately single-use provider is the regression guard proving the check bites. This is the contract half of the re-Configure work. The kernel-side trigger — noticing a changed agent.hcl and re-invoking — is a separate change and is called out as not yet existing rather than implied.
A thin wrapper over pkg/model/modeltest.CheckBinary, for the two cases a Go author's own `go test` cannot reach: a plugin not written in Go, and an operator checking a binary they did not build. It launches the plugin the way the kernel does and speaks nothing but the wire protocol, so the implementation language never comes up. The assertions are modeltest's, unchanged. That is deliberate — a second copy of the rules would eventually disagree with the first, and the whole point of the suite is that there is one answer to "is this conformant". Exit 1 and exit 2 are kept distinct: a binary that will not start has not failed the suite, it has failed to be tested, and a CI job conflating the two reports a conformance regression when the real problem is a bad path or a missing execute bit. Three lint findings on the first version were worth fixing rather than suppressing: main deferred a signal-context release directly above an os.Exit that would skip it; the report was printed piecemeal with six unchecked writes; and a flag-parse failure returned a nil error, which would have made a usage mistake exit silently. The report is now built as one string and written once, so there is a single write whose error is actually checked, and a parse failure returns a sentinel main recognizes and does not double-print.
The provider catalog still described the old single-mode ThinkingSpec and
CachingSpec, and in three places instructed adapter authors to declare
something the protocol can now express truthfully:
- anthropic.md told authors to "pick one canonical mode" for Sonnet
4.6, whose second mechanism it called "not directly representable in
a single ThinkingSpec value". Both controls are now declarable at
once, with budget.deprecated marking the transitional one.
- google.md carried an open escalation — that CachingSpec "has no way
to declare 'this model supports two caching modes concurrently'" and
that this was "worth flagging back to the protocol's designers". That
gap is closed, so the page now shows the declaration rather than the
workaround, and records why it mattered: breakpoints were gated on
the mode, so a model forced to declare implicit caching had to
discard breakpoints it could have honored.
- xai.md mandated thinking.can_disable/mode/default fields that no
longer exist.
google.md keeps one open item rather than dropping it: PricingTier still
carries a single cache-rate pair, so a model billing implicit and
explicit hits at different rates can declare both mechanisms but cannot
price them separately. That is now cross-referenced to conformance.md's
open questions instead of being restated as a protocol-shape complaint.
Also fixes three links into .claude/rules/ that I had added in earlier
commits. Those files are outside the published docs tree, so they broke
`mkdocs build --strict` — caught by running it, not by CI, since the Docs
workflow is path-filtered and deliberately not a required check. They are
now plain references rather than links.
pkg/telemetry documented the hole in its own source: Instruments() and Config() are excluded because both return internal/telemetry types an out-of-tree plugin cannot name, so "a plugin author has no path to either through this package today". Spans already had a route — NewSpanExporter relays them through the kernel — and metrics had none. pkg/kernel.Client.Metrics closes it, as the metrics counterpart to that exporter: observations travel to the kernel via RecordMetrics, which records them against an instrument named from the calling plugin's server-derived identity. Count and Histogram cover the common shapes; RecordBatch exists because each call is a round trip and a plugin reporting per-request metrics should not pay one per metric. Relaying rather than re-exporting internal/telemetry's instrument API is the point, not a limitation. The kernel owns the instruments and bounds their attribute cardinality — properties of the whole system's metric store, which no single plugin can decide for itself — and observability.md's relay model exists precisely so a plugin never exports off-process directly. A pkg/ wrapper over the internal instrument API would have been a second path to the same place, and the wrong one. An observation missing a name or a kind is rejected locally rather than sent for the kernel to reject, so the caller learns which observation was malformed instead of getting a batch-level failure. An empty batch is a no-op: the wire requires a non-empty one, so forwarding it would be a guaranteed rejection for a caller whose loop simply found nothing to report. pkg/telemetry's note is updated from "a known, tracked gap" to what is now true.
A `go build ./...` names its output after the package and writes it to the working directory. The existing patterns cover *.exe, *.so, and *.test, none of which match a bare ELF binary on Linux or macOS — so two of them (a 22 MB providerconform at the repo root, and the example provider inside its own module) were committed before being caught. Listed by name rather than by a broad pattern, because anything wide enough to catch an extensionless binary would also ignore real source files.
Dependency ReviewThe following issues were found:
License Issuesexamples/provider/go.mod
OpenSSF ScorecardScorecard details
Scanned Files
|
There was a problem hiding this comment.
Pull request overview
Strengthens the model-provider plugin protocol, runtime, and pkg/ SDK so third-party model providers can be implemented out-of-tree (in separate repos) with enough surface area and a reusable conformance suite to prove correctness.
Changes:
- Expands
model.v1(provider options passthrough, request-shapedCountTokens, richer streaming/usage signals, thinking/caching axes) and updates specs/docs accordingly. - Improves
pkg/for out-of-tree use (shared SSE framing, HTTP status classification, per-category protocol versions inDescribeidentity, kernel metric relay). - Adds a model-provider conformance suite (
pkg/model/modeltest) and a standalone checker binary (cmd/providerconform), plus CI checks and an out-of-tree example module.
Reviewed changes
Copilot reviewed 106 out of 108 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pkg/widget/server.go | Describe reports category protocol version |
| pkg/widget/doc.go | Adds widget ProtocolVersion constant |
| pkg/tool/server.go | Describe reports category protocol version |
| pkg/tool/doc.go | Adds tool ProtocolVersion constant |
| pkg/telemetry/telemetry.go | Clarifies metrics path via kernel relay |
| pkg/sse/scanner.go | New vendor-neutral SSE frame scanner |
| pkg/sse/doc.go | Documents SSE scanner scope/usage |
| pkg/slashcommand/server.go | Describe reports category protocol version |
| pkg/slashcommand/doc.go | Adds slashcommand ProtocolVersion constant |
| pkg/plugin/identity.go | ProducerRef now takes per-category version |
| pkg/plugin/identity_test.go | Tests per-category protocol version behavior |
| pkg/model/stream.go | Adds StreamStart event helper on Sink |
| pkg/model/server.go | Updates Describe + request-shaped CountTokens |
| pkg/model/server_test.go | Updates CountTokens tests for new request |
| pkg/model/provideroptions.go | Adds typed helpers over provider_options |
| pkg/model/provideroptions_test.go | Tests provider_options lookup semantics |
| pkg/model/modeltest/runbinary_integration_test.go | Integration test drives example provider binary |
| pkg/model/modeltest/run.go | Conformance runner (in-proc + binary) |
| pkg/model/modeltest/report_test.go | Tests report ordering/formatting |
| pkg/model/modeltest/options.go | Conformance run option plumbing |
| pkg/model/modeltest/options_test.go | Tests options propagation semantics |
| pkg/model/modeltest/launch.go | Subprocess launch helpers for conformance |
| pkg/model/modeltest/launch_test.go | Tests env allowlist + logger behavior |
| pkg/model/modeltest/finding.go | Report/finding types and rendering |
| pkg/model/modeltest/doc.go | Documents modeltest drive modes/limits |
| pkg/model/doc.go | Adds model ProtocolVersion constant |
| pkg/model/convert.go | Updates thinking/caching/usage conversions |
| pkg/model/classify.go | Adds shared HTTP status classifier |
| pkg/model/classify_test.go | Tests shared HTTP status classifier |
| pkg/model/capabilities.go | Validates new thinking/caching shapes |
| pkg/model/capabilities_test.go | Updates capability validation tests |
| pkg/memory/server.go | Describe reports category protocol version |
| pkg/memory/doc.go | Adds memory ProtocolVersion constant |
| pkg/kernel/metrics.go | Adds kernel metrics relay client API |
| pkg/kernel/metrics_test.go | Tests metrics relay request construction |
| pkg/kernel/helpers_test.go | Extends fake kernel callback server for metrics |
| pkg/kernel/doc.go | Documents Client.Metrics |
| pkg/frontend/server.go | Describe reports category protocol version |
| pkg/frontend/doc.go | Adds frontend ProtocolVersion constant |
| pkg/context/server.go | Describe reports category protocol version |
| pkg/context/doc.go | Adds context ProtocolVersion constant |
| pkg/common/proto/v1/types.pb.go | Regenerated ProducerRef docs for protocol version |
| pkg/common/plugin.go | Clarifies handshake vs category versions |
| internal/tokencount/tokencount.go | CountTokens request now carries messages |
| internal/tokencount/tokencount_test.go | Updates tests for request-shaped CountTokens |
| internal/tokencount/CLAUDE.md | Updates invariants for exact vs fallback counting |
| internal/pluginhost/supervisor.go | Plumbs per-provider env passthrough to launcher |
| internal/modelrequest/params.go | Validates thinking params against per-axis controls |
| internal/modelrequest/params_test.go | Tests new per-axis thinking validation |
| internal/modelrequest/CLAUDE.md | Updates thinking validation invariants |
| internal/modelrequest/cache.go | Gates breakpoints on explicit_markers axis |
| internal/modelrequest/cache_test.go | Tests caching axes + breakpoint behavior |
| internal/kernel/testdata/plugin/main.go | Updates fixture model capability declarations |
| internal/kernel/bringup.go | Passes provider env config into supervisor |
| internal/cost/pricing_test.go | Updates thinking default in pricing tests |
| internal/config/types.go | Adds ProviderEnv to resolved config |
| internal/config/providerenv.go | Extracts provider environment{} block |
| internal/config/load.go | Lifts environment{} before schema decode |
| internal/config/errors.go | Adds provider env validation errors |
| internal/config/bridge.go | Sensitive-attr validation tolerates kernel blocks |
| internal/anthropic/provider.go | Provider CountTokens takes request shape |
| internal/anthropic/provider_test.go | Updates provider CountTokens tests |
| internal/anthropic/provider_e2e_test.go | Live tests cover stream_start + CountTokens |
| internal/anthropic/messages/sse.go | Switches to shared pkg/sse framing |
| internal/anthropic/messages/request.go | Adds BuildCountTokensRequest + thinking rules |
| internal/anthropic/messages/request_test.go | Tests count-tokens request includes tools/system |
| internal/anthropic/messages/events.go | Adds StreamStart to EventSink |
| internal/anthropic/messages/events_test.go | Updates fake sink for StreamStart |
| internal/anthropic/messages/client.go | Emits StreamStart + request-shaped CountTokens |
| internal/anthropic/messages/client_test.go | Updates CountTokens client tests |
| internal/anthropic/messages/classify.go | Uses shared HTTP classifier fallback |
| internal/anthropic/conformance_test.go | Runs shared conformance suite on Anthropic provider |
| internal/anthropic/catalog/catalog.go | Updates roster thinking/caching declarations |
| internal/anthropic/catalog/catalog_test.go | Updates tests for new thinking/caching shapes |
| examples/provider/go.mod | Separate module to validate out-of-tree build |
| examples/provider/conformance_test.go | Example provider runs modeltest suite |
| docs/specifications/model/protocol.md | Updates CountTokens/Configure/contracts |
| docs/specifications/model/conformance.md | Updates MUST/SHOULD matrix and open questions |
| docs/specifications/configuration/blocks-reference.md | Documents provider environment{} block |
| docs/first-party/providers/xai.md | Updates to new thinking/caching fields |
| docs/first-party/providers/openai.md | Updates to new thinking/caching fields |
| docs/first-party/providers/google.md | Updates caching mechanism expressiveness notes |
| docs/first-party/providers/anthropic.md | Updates thinking/caching mapping description |
| cmd/providerconform/main.go | New CLI wrapper for binary conformance |
| cmd/providerconform/main_test.go | Tests CLI behavior and config loading |
| api/pluggableharness/model/v1/rpc_request.proto | Adds provider_options + request CountTokens |
| api/pluggableharness/model/v1/events.proto | Adds stream_start event |
| api/pluggableharness/common/v1/types.proto | ProducerRef protocol_version semantics updated |
| .gitignore | Ignores stray local build artifacts |
| .github/workflows/ci.yml | Adds pkg/internal import check + example module build |
| .claude/rules/proto.md | Documents Struct precedent + pass-through rule |
| .claude/rules/plugin-runtime.md | Documents decoupled handshake vs category versions |
| .claude/rules/go-layout.md | Adds pkg/sse to repo layout guidance |
Comment on lines
+21
to
+25
| // mirroring what the real launcher does: a plugin that only works because | ||
| // it inherited a credential from the test runner's environment would pass | ||
| // here and fail under the kernel, which is the opposite of what a | ||
| // conformance run is for. A provider needing configuration receives it | ||
| // through Configure, via WithConfig. |
Comment on lines
+181
to
+183
| func (s *Scanner) IsDone() bool { | ||
| return string(s.data) == doneSentinel | ||
| } |
Comment on lines
+135
to
+139
| schema := &hcl.BodySchema{Attributes: make([]hcl.AttributeSchema, 0, len(sensitiveAttrs))} | ||
| for name := range sensitiveAttrs { | ||
| schema.Attributes = append(schema.Attributes, hcl.AttributeSchema{Name: name}) | ||
| } | ||
| content, _, diags := body.PartialContent(schema) |
6 tasks
scrothers
added a commit
that referenced
this pull request
Jul 27, 2026
CI on main is red. The github-actions bump in #13 also moved the root go.mod, but examples/provider is a separate module — by design, since it exists to prove pkg/ builds from outside the main one — so its pinned versions were left behind. The CI step added in #14 runs `go mod tidy` there and fails on any diff, which is exactly what happened. Tidying it fixes today. The reason it happened is that Dependabot resolves a directory, not a repository, and its gomod entry only listed "/" — so the nested module goes stale silently on every root bump, and every future dependency PR would turn main red the same way. Adding /examples/provider to that entry fixes the cause. This is the same failure mode the config's own composite-action note already describes: "each one's directory has to be listed, or the SHAs pinned inside it go stale silently while the workflows around it stay current." The dependency versions here are Dependabot's from #13, not chosen by this change — `go mod tidy` resolves them from the root module's graph.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Provider plugins for Anthropic, OpenAI, xAI, and OpenRouter are going to be built out of band, in their own repos. They can't be today: the model protocol can't express what real vendors do, the runtime starves a subprocess of things a real provider needs,
pkg/has holes with no way to detect new ones, and there was no way to check a provider for correctness short of re-reading the spec.This makes the protocol, the runtime, and the
pkg/SDK strong enough that a session with no access to this tree can build a provider correctly and prove it did.Timing is the reason this is one PR rather than several.
git tag -lis empty, sobuf breakinghas no base —model.v1is freely mutable right now, and every protocol change here is av1edit that becomes av2package the moment the firstv*tag exists.Protocol (
model.v1)provider_options— a pass-throughStructfor vendor knobs the kernel has no semantics for. Without it, every third-party provider feature is gated on a PR to this repo. Extendsproto.md's existingStructprecedent list rather than adding a third opaque-bytes carve-out, and records the rule that keeps that list honest: a field the kernel reads must be typed or it doesn't work.ThinkingSpec→ independent axes. The enum forced a half-truth, and this repo had already written that down three times —catalog.gocalls Fable 5's declaration "a deliberate choice between two modes that each capture half the truth" and Opus 5's conditional disable "a caveat the ThinkingSpec shape cannot express", settling for what it calls "the larger lie".buildThinkingwas the proof: it already emittedthinking:{type:"adaptive"}andoutput_config.efforttogether, because the enum named a model family, not a capability.CachingSpec→ independent axes.docs/first-party/providers/google.mdhad escalated this: "the current CachingSpec shape has no way to declare 'this model supports two caching modes concurrently'… worth flagging back to the protocol's designers." It had teeth beyond accuracy — breakpoints were gated on the mode, so a Gemini-shaped model declaring implicit caching was required to discard breakpoints it could have honored, losing a discount with no error anywhere.CountTokenscounts a request, not a string. Every vendor's endpoint counts messages + system + tools; the old shape discarded exactly the weight that decides whether a turn fits in the context window.Usage.rate_limits(repeated — vendors meter several budgets that exhaust independently) andstream_start(emitted early, so a failed stream is still correlatable to vendor logs).ConfigureMUST be re-callable, and a gateway/locally-served provider'sGetCapabilitiescontract is specced.pkg/sufficiencypkg/sseandpkg/model.ClassifyHTTPStatuspromoted out ofinternal/anthropic. Its tests pass unchanged — the evidence both extractions preserve behavior.pkg/kernel.Client.Metricscloses the holepkg/telemetrydocumented in its own source ("a plugin author has no path to either through this package today"). Spans had a relay; metrics had none.examples/provideris a real separate module with areplaceback to the tree, built and tested in CI. Thedepguardrule only simulates out-of-tree isolation; a simulation can't catch an unexported type leaking through an exported signature. Paired with a check thatpkg/never importsinternal/.Conformance
pkg/model/modeltestturnsconformance.md's MUST matrix into assertions — in-process over a real gRPC round trip, and against a built binary through a real handshake, sharing one implementation so the modes can't drift.cmd/providerconformwraps it for plugins written in any language. Both real providers pass.Runtime
Per-provider
environment{}passthrough.pluginruntimedeliberately never inherits the kernel's environment, andConfig.ExtraEnvexisted but nothing ever populated it — so noHTTPS_PROXY, no ambient cloud credential chain, andpkg/telemetry.Bootstrapcould never be configured.Category protocol versions decoupled from the go-plugin handshake. Under the old coupling a breaking change to
modelwould bump the shared handshake version and thereby reject every tool, context, and memory plugin ever published, none of which changed.Checklist
go mod tidyis a no-op,go build ./...,go vet ./...,gofmt -l -s .prints nothing,go test -race -covermode=atomic ./...,golangci-lint rundocs/specifications/document in this same PRpkg/*/proto/v1/—.protochanges made inapi/and regenerated withbuf generateinternal/packages includeREADME.md+CLAUDE.mdin the same commit (no newinternal/packages in this PR)bin/Also verified beyond the checklist:
buf lint,buf format --exit-code, generated-code drift,-tags=integration,go vet -tags=e2e,mkdocs build --strict, and the example module's own build/vet/test.Notes for reviewers
Worth close reading:
docs/specifications/model/data-types.md(the two restructured capability types) andpkg/model/modeltest/stream.go'scheckCancellation, whose comment documents a real limit rather than implying coverage it doesn't have.Four defects the work surfaced — three in the conformance suite itself. Writing the suite's own negative controls found that
WithExpectedIdentitycould never fail in-process (Checkwas serving the very identity it then compared against); thatcheckCancellationhad no bound and claimed an assertion impossible from a black-box client, since gRPC returnsCanceledto the canceling client whatever the server does; that a zero-widthBudgetControlrange was accepted; and thatexamples/providersilently accepted image/document blocks — the exact violation the suite exists to catch.Two deviations from the agreed scope, both deliberate. No
response_format: nothing in the kernel produces or consumes structured output, so a typed field would be dead protocol surface. Re-Configureis delivered as contract + conformance check only — the kernel-side trigger is a separate change and is called out as not existing rather than implied.Deferred with reasons, in
conformance.mdopen questions.PricingTiercarries one cache-rate pair but vendors bill cached tokens at several rates (Anthropic 5m vs 1h TTL, Gemini implicit vs explicit). That deliberately did not become aprovider_optionsknob — the kernel persistscost_usd, so a rate-changing pass-through value would write silently wrong ledger rows forever. Fixing it is aPricingredesign. Also recorded: reasoning models rejectingtemperature, which adapters currently infer from the thinking shape.One HCL trap, caught by a test.
PartialContent's remain body does not hide a consumed block fromJustAttributes, so the first workingenvironment{}broke the secret check for every provider using it — especially the ones carrying a credential.validateSensitiveAttrsnow asks for named attributes.Pre-existing flake, not from this PR.
TestServer_Subscribe_backpressureCloses(internal/kernelcallback) failed once under full parallel-shuffleload, then passed 5× isolated and 10× on the base commit. Its own comment documents it as timing-sensitive. Untouched here.History note. Two compiled binaries were committed mid-work (
go build ./...without-onames its output after the package and drops it in the working directory;.gitignore's*.exe/*.so/*.testpatterns don't match a bare ELF file). One was 22 MB. They were rewritten out of the branch before any push — this branch has never been force-pushed and no one else has fetched it — and.gitignorenow names them.