From 74fbd20d4ea0e80084f31752ad5aa9c4cc85d8b9 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 11:48:17 -0400 Subject: [PATCH 01/16] model: add provider_options escape hatch MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A third-party model provider currently cannot ship any vendor feature the kernel's proto has not already named, because every request field is strongly typed and adding one requires a change to this repository. That makes the out-of-tree provider surface effectively closed. Add StreamCompletionRequest.provider_options, a Struct the kernel passes through untouched — vendor knobs it has no semantics for, such as a service tier, a sampling seed, or a beta-feature flag. This extends proto.md's existing Struct precedent list rather than adding a third opaque-bytes carve-out: the justification is the same one ConfigureRequest.config already carries — the shape is the provider's schema, not the kernel's to name — applied per-request instead of once at configure time. Record the rule that keeps the list from becoming a general escape from strong typing: a Struct field is pass-through only, so a value the kernel branches on (routing, capability validation, cost, replay) must be a typed field or it does not work at all. Prompt-cache TTL is the worked example, since the kernel computes cost_usd and the TTL changes the rate. pkg/model gains Options, a nil-safe view with os.LookupEnv-style (value, ok) accessors, so each provider does not hand-roll structpb traversal. LookupInt64 rejects values that are not exactly representable rather than truncating an operator's value silently. --- .claude/rules/proto.md | 25 +++- .../model/v1/rpc_request.proto | 24 ++++ docs/specifications/model/conformance.md | 1 + docs/specifications/model/data-types.md | 11 ++ pkg/model/proto/v1/rpc_request.pb.go | 53 +++++-- pkg/model/provideroptions.go | 129 +++++++++++++++++ pkg/model/provideroptions_test.go | 131 ++++++++++++++++++ 7 files changed, 361 insertions(+), 13 deletions(-) create mode 100644 pkg/model/provideroptions.go create mode 100644 pkg/model/provideroptions_test.go diff --git a/.claude/rules/proto.md b/.claude/rules/proto.md index 1279f24..dd60e77 100644 --- a/.claude/rules/proto.md +++ b/.claude/rules/proto.md @@ -53,10 +53,29 @@ strongly typed as the Go code that implements it. enumerate: `log.v1.LogEntry.fields` (mirrors `slog.Attr`'s open key/value model), `config.v1`'s `ConfigureRequest.config` and `kernel.v1.GetConfigResult.config` (already-decoded `agent.hcl` values, whose shape is the *provider's* schema, - not the kernel's to name), and `trace.v1.Span`/`SpanEvent`'s `attributes` + not the kernel's to name), `trace.v1.Span`/`SpanEvent`'s `attributes` (an OTel span's attribute set, open-ended per call site by the same - reasoning as `log.v1.LogEntry.fields`) are the precedents. A field whose - keys are actually fixed and enumerable belongs in a real message instead. + reasoning as `log.v1.LogEntry.fields`), and + `model.v1.StreamCompletionRequest.provider_options` (vendor-specific + request knobs the kernel has no semantics for — the same "shape is the + provider's, not the kernel's to name" reasoning as `ConfigureRequest.config`, + applied per-request rather than once at configure time) are the + precedents. A field whose keys are actually fixed and enumerable belongs + in a real message instead. +- **A `Struct` field is pass-through only: if the kernel reads it, it must + be a typed field instead.** This is what keeps the precedent list above + from becoming a general-purpose escape from the strong-typing rule. Every + entry on it is data the kernel *carries* — logs it, relays it, hands it + to the plugin that owns it — never data the kernel *branches on*. The + moment a value affects kernel behavior (routing, capability validation, + cost computation, replay), a `Struct` is the wrong home for it, because + nothing about a `Struct` key is discoverable, validatable, or versionable + at the wire contract level. `model.v1`'s prompt-cache TTL is the worked + example: it looks like an ordinary vendor knob, but the kernel computes + `cost_usd` and the TTL changes the rate, so it cannot live in + `provider_options` and must be a typed field or not exist + (`docs/specifications/model/conformance.md`'s open questions records why + it is currently the latter). - Every field that has a natural bounded domain (status, kind, risk class, error category) is an `enum`, not a `string`. `docs/specifications/tool/conformance.md`'s `ToolErrorCategory`, and `docs/specifications/tool/data-types.md`'s `RiskClass` diff --git a/api/pluggableharness/model/v1/rpc_request.proto b/api/pluggableharness/model/v1/rpc_request.proto index ee76144..dc30d62 100644 --- a/api/pluggableharness/model/v1/rpc_request.proto +++ b/api/pluggableharness/model/v1/rpc_request.proto @@ -90,6 +90,30 @@ message StreamCompletionRequest { // natural stable-prefix boundaries — see // model/protocol.md#cache-breakpoint-placement-policy. repeated CacheBreakpoint cache_breakpoints = 7; + + // Vendor-specific request knobs the kernel has no semantics for, passed + // through untouched: the kernel never reads a key, validates one, or + // assigns meaning to one. This is the escape hatch that lets a + // third-party provider ship a vendor feature — a service tier, a + // sampling seed, a beta-feature flag, a conversation-retention id — + // without a change to this protocol. Values originate in the provider's + // own ConfigSchema and the operator's provider{} block, and the provider + // documents its own accepted keys; two providers MAY use the same key + // name for unrelated things. + // + // A Struct for the same reason ConfigureRequest.config is one — the + // shape is the provider's schema, not the kernel's to name — applied + // per-request rather than once at configure time (.claude/rules/proto.md's + // Struct precedent list). + // + // MUST NOT carry anything the kernel reads. Pass-through is the whole + // contract, so a value affecting routing, capability validation, cost + // computation, or replay is a typed field on this protocol or it does + // not work at all — a provider smuggling one through here gets silence, + // not kernel behavior. Promoting such a knob to a typed field in a later + // revision is the fix; teaching the kernel to read this field is not. + // See model/data-types.md#provider_options. + optional google.protobuf.Struct provider_options = 8; } // CountTokensRequest is CountTokens' request: the raw text to count, per diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index 1e95410..f73ca6d 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -34,6 +34,7 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | `StreamEvent.redacted_thinking` | MUST, for a vendor that emits vendor-encrypted reasoning blocks | [`data-types.md#streamevent`](data-types.md#streamevent) — a whole block, never fragmented; stored and echoed back verbatim or the vendor rejects the whole conversation on a later turn | | Prompt caching | MAY, capability-gated via `CachingSpec` | declare `mode` (explicit vs. implicit) | | Cache breakpoints (`StreamCompletionRequest.cache_breakpoints`) | MUST honor where `CachingSpec.mode = CACHING_MODE_EXPLICIT_MARKERS`; MUST ignore otherwise | [`protocol.md#cache-breakpoint-placement-policy`](protocol.md#cache-breakpoint-placement-policy) — placement is a kernel decision, never the plugin's | +| `StreamCompletionRequest.provider_options` | MAY consume; kernel MUST pass through untouched | [`data-types.md#provider_options`](data-types.md#provider_options) — vendor knobs the kernel has no semantics for. A value the kernel reads MUST be a typed field instead, never smuggled through here | | Parallel tool calls in one turn | SHOULD declare via `supports_parallel_tool_calls` | kernel serializes calls if absent/false | | Tool-choice constraint (`GenerationParams.tool_choice`) | MAY, capability-gated via `ModelSpec.supported_tool_choice_modes` | kernel MUST NOT send a mode absent from the declared list, mirroring `ThinkingSpec` validation | | `Render` | MAY | generic fallback exists; `RenderRequest.schema_version` MUST be set when implemented | diff --git a/docs/specifications/model/data-types.md b/docs/specifications/model/data-types.md index fb18327..25f5066 100644 --- a/docs/specifications/model/data-types.md +++ b/docs/specifications/model/data-types.md @@ -190,6 +190,7 @@ StreamCompletionRequest { assembled_context []ContextSection // MUST — the kernel-assembled context chain, see below call_context CallContext // MUST — session/turn/working-directory attribution, see below cache_breakpoints []CacheBreakpoint // MAY be empty — see below + provider_options Struct? // MAY — vendor-specific pass-through, see below } ``` @@ -223,6 +224,16 @@ CacheBreakpoint { **Breakpoint placement is a kernel decision, not the plugin's.** The kernel knows each `assembled_context` section's `Stability` (`content/v1/types.proto`'s `Stability` enum: `STABILITY_STATIC` vs. `STABILITY_DYNAMIC`) and each message's position in the conversation, so it places breakpoints at natural stable-prefix boundaries — the same tools → system → static-project-context → conversation-tail ordering that governs `assembled_context`'s own chain order. In practice this means: a breakpoint after `after_tools` when the tool declaration list is stable turn to turn, and a breakpoint after `after_assembled_context` when the whole chain's leading sections are `STABILITY_STATIC` — since that's usually the longest stable prefix a vendor's prompt cache can actually reuse. A plugin never invents its own placement; it only translates the breakpoints the kernel already decided into vendor-native markers. +### `provider_options` + +`provider_options` is an optional `google.protobuf.Struct` carrying vendor-specific request knobs the kernel has no semantics for. It is the escape hatch that lets a third-party provider ship a vendor feature without a change to this protocol — a vendor's `service_tier`, a sampling `seed`, a beta-feature header flag, a conversation-retention id, an incremental-response id. The kernel passes it through untouched: it never reads a key, never validates one, and never assigns meaning to one. + +Values originate the same way every other provider-specific value does — from the provider's own `ConfigSchema` and the operator's `provider "" { ... }` block ([`configuration/blocks-reference.md`](../configuration/blocks-reference.md)) — and the provider documents its own accepted keys. Two providers MAY use the same key name for unrelated things; nothing in this protocol coordinates them. + +**The rule that keeps this honest: a field the kernel reads MUST NOT live here.** `provider_options` is pass-through by construction, so anything the kernel acts on — a value affecting routing, capability validation, cost computation, or replay — is a typed field on this protocol or it does not work at all. A provider that smuggles such a value through `provider_options` gets no kernel behavior from it, only silence. Concretely: prompt-cache TTL selection cannot live here (the kernel computes `cost_usd` and the TTL changes the rate), a thinking-effort level cannot live here (the kernel validates it against [`ThinkingSpec`](#thinkingspec) and routes fallbacks on it), and a token count cannot live here. When a knob turns out to need kernel behavior, the fix is to promote it to a typed field in a later protocol revision — not to teach the kernel to read `provider_options`. + +This is a `Struct` for the same reason `ConfigureRequest.config` is one — the shape is the provider's schema, not the kernel's to name — applied per-request rather than once at configure time. It is deliberately *not* one of the opaque `bytes` payloads (Emit→Render→Paint, event-bus publish), which exist so a producer's payload format can evolve independently of the kernel; `provider_options` exists because the kernel has no opinion about the values at all. See [`.claude/rules/proto.md`](../../../.claude/rules/proto.md)'s strong-typing section, whose pass-through rule this section restates from the protocol side. + ## `GenerationParams` `GenerationParams` carries per-request overrides of otherwise model-default generation behavior: diff --git a/pkg/model/proto/v1/rpc_request.pb.go b/pkg/model/proto/v1/rpc_request.pb.go index 2cf50b7..1b3cffb 100644 --- a/pkg/model/proto/v1/rpc_request.pb.go +++ b/pkg/model/proto/v1/rpc_request.pb.go @@ -206,8 +206,31 @@ type StreamCompletionRequest struct { // natural stable-prefix boundaries — see // model/protocol.md#cache-breakpoint-placement-policy. CacheBreakpoints []*CacheBreakpoint `protobuf:"bytes,7,rep,name=cache_breakpoints,json=cacheBreakpoints,proto3" json:"cache_breakpoints,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Vendor-specific request knobs the kernel has no semantics for, passed + // through untouched: the kernel never reads a key, validates one, or + // assigns meaning to one. This is the escape hatch that lets a + // third-party provider ship a vendor feature — a service tier, a + // sampling seed, a beta-feature flag, a conversation-retention id — + // without a change to this protocol. Values originate in the provider's + // own ConfigSchema and the operator's provider{} block, and the provider + // documents its own accepted keys; two providers MAY use the same key + // name for unrelated things. + // + // A Struct for the same reason ConfigureRequest.config is one — the + // shape is the provider's schema, not the kernel's to name — applied + // per-request rather than once at configure time (.claude/rules/proto.md's + // Struct precedent list). + // + // MUST NOT carry anything the kernel reads. Pass-through is the whole + // contract, so a value affecting routing, capability validation, cost + // computation, or replay is a typed field on this protocol or it does + // not work at all — a provider smuggling one through here gets silence, + // not kernel behavior. Promoting such a knob to a typed field in a later + // revision is the fix; teaching the kernel to read this field is not. + // See model/data-types.md#provider_options. + ProviderOptions *structpb.Struct `protobuf:"bytes,8,opt,name=provider_options,json=providerOptions,proto3,oneof" json:"provider_options,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *StreamCompletionRequest) Reset() { @@ -289,6 +312,13 @@ func (x *StreamCompletionRequest) GetCacheBreakpoints() []*CacheBreakpoint { return nil } +func (x *StreamCompletionRequest) GetProviderOptions() *structpb.Struct { + if x != nil { + return x.ProviderOptions + } + return nil +} + // CountTokensRequest is CountTokens' request: the raw text to count, per // model.md §2.1. type CountTokensRequest struct { @@ -417,7 +447,7 @@ const file_pluggableharness_model_v1_rpc_request_proto_rawDesc = "" + "\x16GetCapabilitiesRequest\"C\n" + "\x10ConfigureRequest\x12/\n" + "\x06config\x18\x01 \x01(\v2\x17.google.protobuf.StructR\x06config\"\x11\n" + - "\x0fDescribeRequest\"\x8c\x04\n" + + "\x0fDescribeRequest\"\xea\x04\n" + "\x17StreamCompletionRequest\x12@\n" + "\bmessages\x18\x01 \x03(\v2$.pluggableharness.content.v1.MessageR\bmessages\x12\x19\n" + "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12@\n" + @@ -425,8 +455,10 @@ const file_pluggableharness_model_v1_rpc_request_proto_rawDesc = "" + "\x06params\x18\x04 \x01(\v2+.pluggableharness.model.v1.GenerationParamsH\x00R\x06params\x88\x01\x01\x12X\n" + "\x11assembled_context\x18\x05 \x03(\v2+.pluggableharness.content.v1.ContextSectionR\x10assembledContext\x12J\n" + "\fcall_context\x18\x06 \x01(\v2'.pluggableharness.common.v1.CallContextR\vcallContext\x12W\n" + - "\x11cache_breakpoints\x18\a \x03(\v2*.pluggableharness.model.v1.CacheBreakpointR\x10cacheBreakpointsB\t\n" + - "\a_params\"C\n" + + "\x11cache_breakpoints\x18\a \x03(\v2*.pluggableharness.model.v1.CacheBreakpointR\x10cacheBreakpoints\x12G\n" + + "\x10provider_options\x18\b \x01(\v2\x17.google.protobuf.StructH\x01R\x0fproviderOptions\x88\x01\x01B\t\n" + + "\a_paramsB\x13\n" + + "\x11_provider_options\"C\n" + "\x12CountTokensRequest\x12\x12\n" + "\x04text\x18\x01 \x01(\tR\x04text\x12\x19\n" + "\bmodel_id\x18\x02 \x01(\tR\amodelId\"P\n" + @@ -470,11 +502,12 @@ var file_pluggableharness_model_v1_rpc_request_proto_depIdxs = []int32{ 10, // 4: pluggableharness.model.v1.StreamCompletionRequest.assembled_context:type_name -> pluggableharness.content.v1.ContextSection 11, // 5: pluggableharness.model.v1.StreamCompletionRequest.call_context:type_name -> pluggableharness.common.v1.CallContext 12, // 6: pluggableharness.model.v1.StreamCompletionRequest.cache_breakpoints:type_name -> pluggableharness.model.v1.CacheBreakpoint - 7, // [7:7] is the sub-list for method output_type - 7, // [7:7] is the sub-list for method input_type - 7, // [7:7] is the sub-list for extension type_name - 7, // [7:7] is the sub-list for extension extendee - 0, // [0:7] is the sub-list for field type_name + 6, // 7: pluggableharness.model.v1.StreamCompletionRequest.provider_options:type_name -> google.protobuf.Struct + 8, // [8:8] is the sub-list for method output_type + 8, // [8:8] is the sub-list for method input_type + 8, // [8:8] is the sub-list for extension type_name + 8, // [8:8] is the sub-list for extension extendee + 0, // [0:8] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_rpc_request_proto_init() } diff --git a/pkg/model/provideroptions.go b/pkg/model/provideroptions.go new file mode 100644 index 0000000..1d7246a --- /dev/null +++ b/pkg/model/provideroptions.go @@ -0,0 +1,129 @@ +package model + +import ( + "math" + "sort" + + "google.golang.org/protobuf/types/known/structpb" + + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// Options is a read-only view over a request's provider_options +// (docs/specifications/model/data-types.md#provider_options) — the +// vendor-specific knobs the kernel passes through untouched. +// +// The zero value is usable and empty, so a Provider never has to nil-check +// before reading: a request that carried no provider_options answers false +// to every lookup, which is the same answer an absent key gives. That is +// deliberate — an adapter reading an optional knob wants one branch ("did +// the operator set this?"), not two ("was the field present, and if so was +// the key present?"). +// +// Every lookup follows os.LookupEnv's (value, ok) convention. ok is false +// for an absent key AND for a key whose value is the wrong JSON type, so a +// misconfigured value falls back to the adapter's default rather than +// silently becoming a zero value. A Provider that needs to reject a +// wrong-typed value rather than ignore it should check Has first. +type Options struct { + s *structpb.Struct +} + +// ProviderOptions returns a view over req's provider_options. A nil req, or +// one with no provider_options set, yields an empty Options rather than an +// error — absent options are the ordinary case, not a failure. +func ProviderOptions(req *modelv1.StreamCompletionRequest) Options { + return Options{s: req.GetProviderOptions()} +} + +// Has reports whether key is present, regardless of its value's type. +// Use it to distinguish "the operator set this to something unusable" from +// "the operator did not set this", which the (value, ok) lookups below +// deliberately collapse. +func (o Options) Has(key string) bool { + _, ok := o.s.GetFields()[key] + return ok +} + +// Keys returns the keys present, sorted. +// +// Sorted because a provider may include them in a log line or an error +// message, and Go map iteration order is randomized — an unsorted list +// would make otherwise-identical runs differ (.claude/rules/determinism.md's +// serialization rule). +func (o Options) Keys() []string { + fields := o.s.GetFields() + if len(fields) == 0 { + return nil + } + keys := make([]string, 0, len(fields)) + for k := range fields { + keys = append(keys, k) + } + sort.Strings(keys) + return keys +} + +// LookupString returns key's string value. +func (o Options) LookupString(key string) (string, bool) { + v, ok := o.s.GetFields()[key] + if !ok { + return "", false + } + if _, isString := v.GetKind().(*structpb.Value_StringValue); !isString { + return "", false + } + return v.GetStringValue(), true +} + +// LookupBool returns key's boolean value. +func (o Options) LookupBool(key string) (bool, bool) { + v, ok := o.s.GetFields()[key] + if !ok { + return false, false + } + if _, isBool := v.GetKind().(*structpb.Value_BoolValue); !isBool { + return false, false + } + return v.GetBoolValue(), true +} + +// LookupFloat64 returns key's numeric value. +func (o Options) LookupFloat64(key string) (float64, bool) { + v, ok := o.s.GetFields()[key] + if !ok { + return 0, false + } + if _, isNumber := v.GetKind().(*structpb.Value_NumberValue); !isNumber { + return 0, false + } + return v.GetNumberValue(), true +} + +// LookupInt64 returns key's numeric value as an int64. +// +// A Struct carries every number as a float64 (it is JSON's model, not +// protobuf's), so this rejects a value that is not exactly representable as +// an int64: a fractional value, a NaN or infinity, or one outside int64's +// range all report false rather than truncating. Silently truncating a +// token budget or a retry count to something the operator did not write is +// worse than falling back to the adapter's default. +func (o Options) LookupInt64(key string) (int64, bool) { + f, ok := o.LookupFloat64(key) + if !ok { + return 0, false + } + if math.IsNaN(f) || math.IsInf(f, 0) || f != math.Trunc(f) { + return 0, false + } + // Compared as float64 against the exact powers of two bracketing + // int64's range, not against math.MaxInt64: converting math.MaxInt64 to + // float64 rounds it *up* to 2^63, so a direct f > float64(math.MaxInt64) + // comparison would let 2^63 itself through and overflow the conversion + // below. + const twoPow63 = float64(1 << 63) + if f >= twoPow63 || f < -twoPow63 { + return 0, false + } + return int64(f), true +} diff --git a/pkg/model/provideroptions_test.go b/pkg/model/provideroptions_test.go new file mode 100644 index 0000000..c8b5463 --- /dev/null +++ b/pkg/model/provideroptions_test.go @@ -0,0 +1,131 @@ +package model_test + +import ( + "math" + "slices" + "testing" + + "google.golang.org/protobuf/types/known/structpb" + + "github.com/pluggableharness/agent/pkg/model" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// optionsFrom builds a request carrying raw as provider_options. +func optionsFrom(t *testing.T, raw map[string]any) model.Options { + t.Helper() + s, err := structpb.NewStruct(raw) + if err != nil { + t.Fatalf("structpb.NewStruct(%v): %v", raw, err) + } + return model.ProviderOptions(&modelv1.StreamCompletionRequest{ProviderOptions: s}) +} + +func TestProviderOptions_absentSourcesAreEmptyNotPanicking(t *testing.T) { + t.Parallel() + + // The zero value, a nil request, and a request with no provider_options + // must all behave identically: a Provider reading an optional knob gets + // one branch, not three. + for name, opts := range map[string]model.Options{ + "zero value": {}, + "nil request": model.ProviderOptions(nil), + "request without it": model.ProviderOptions(&modelv1.StreamCompletionRequest{}), + } { + if opts.Has("anything") { + t.Errorf("%s: Has = true, want false", name) + } + if got := opts.Keys(); got != nil { + t.Errorf("%s: Keys = %v, want nil", name, got) + } + if v, ok := opts.LookupString("anything"); ok || v != "" { + t.Errorf("%s: LookupString = (%q, %v), want (\"\", false)", name, v, ok) + } + } +} + +func TestProviderOptions_lookupsReturnTypedValues(t *testing.T) { + t.Parallel() + + opts := optionsFrom(t, map[string]any{ + "tier": "priority", + "beta": true, + "seed": float64(42), + "top_p": 0.95, + "nothing": nil, + }) + + if got, ok := opts.LookupString("tier"); !ok || got != "priority" { + t.Errorf("LookupString(tier) = (%q, %v), want (priority, true)", got, ok) + } + if got, ok := opts.LookupBool("beta"); !ok || !got { + t.Errorf("LookupBool(beta) = (%v, %v), want (true, true)", got, ok) + } + if got, ok := opts.LookupInt64("seed"); !ok || got != 42 { + t.Errorf("LookupInt64(seed) = (%d, %v), want (42, true)", got, ok) + } + if got, ok := opts.LookupFloat64("top_p"); !ok || got != 0.95 { + t.Errorf("LookupFloat64(top_p) = (%v, %v), want (0.95, true)", got, ok) + } +} + +func TestProviderOptions_wrongTypeReportsAbsentButHasStaysTrue(t *testing.T) { + t.Parallel() + + // A wrong-typed value must fall back to the adapter's default rather + // than becoming a zero value — but Has still reports the key, which is + // how a Provider that wants to reject the config instead can tell the + // two apart. + opts := optionsFrom(t, map[string]any{"seed": "not-a-number"}) + + if got, ok := opts.LookupInt64("seed"); ok || got != 0 { + t.Errorf("LookupInt64 = (%d, %v), want (0, false)", got, ok) + } + if !opts.Has("seed") { + t.Error("Has(seed) = false, want true — the key is present, only its type is wrong") + } +} + +func TestProviderOptions_lookupInt64RejectsInexactValues(t *testing.T) { + t.Parallel() + + // Every case here would corrupt an operator's value if truncated. + // 2^63 is the specific trap: float64(math.MaxInt64) rounds UP to 2^63, + // so a naive `f > float64(math.MaxInt64)` bound lets it through and the + // int64 conversion overflows. + const twoPow63 = float64(1 << 63) + for name, v := range map[string]float64{ + "fractional": 1.5, + "NaN": math.NaN(), + "+Inf": math.Inf(1), + "-Inf": math.Inf(-1), + "exactly 2^63": twoPow63, + "above 2^63": twoPow63 * 2, + "below -2^63": -twoPow63 * 2, + } { + opts := optionsFrom(t, map[string]any{"n": v}) + if got, ok := opts.LookupInt64("n"); ok { + t.Errorf("LookupInt64(%s = %v) = (%d, true), want ok=false", name, v, got) + } + } + + // The largest value that IS exactly representable must still pass, so + // the bound above isn't rejecting legitimate input. + opts := optionsFrom(t, map[string]any{"n": -twoPow63}) + if got, ok := opts.LookupInt64("n"); !ok || got != math.MinInt64 { + t.Errorf("LookupInt64(-2^63) = (%d, %v), want (%d, true)", got, ok, int64(math.MinInt64)) + } +} + +func TestProviderOptions_keysAreSorted(t *testing.T) { + t.Parallel() + + // Sorted so a provider logging or erroring with these produces + // identical output across runs (determinism.md's serialization rule). + opts := optionsFrom(t, map[string]any{"zeta": 1.0, "alpha": 2.0, "mu": 3.0}) + + want := []string{"alpha", "mu", "zeta"} + if got := opts.Keys(); !slices.Equal(got, want) { + t.Errorf("Keys = %v, want %v", got, want) + } +} From 62626a862956a899b28990d5540241aa8d9e86af Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:02:54 -0400 Subject: [PATCH 02/16] model: make ThinkingSpec axes independent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ThinkingMode was a single mutually-exclusive enum, so a model occupying more than one position could only declare a half-truth. Three places in this repository had already written that down: - catalog.go calls Fable 5's declaration "a deliberate choice between two modes that each capture half the truth" — it reasons adaptively, cannot be disabled, AND exposes the full effort ladder. - catalog.go calls Opus 5's conditional disable "a caveat the ThinkingSpec shape cannot express", settling for the answer that is "the larger lie". - docs/first-party/providers/anthropic.md tells adapter authors to "pick one canonical mode to declare" for Sonnet 4.6, whose second mechanism is "not directly representable in a single ThinkingSpec". buildThinking is the proof: for DISCRETE_EFFORT it already emitted both thinking:{type:"adaptive"} and output_config.effort, because Anthropic's effort ladder rides on top of adaptive reasoning rather than replacing it. The enum named a model family, not a capability. Replace it with four independent axes — an optional EffortControl, an optional BudgetControl, adaptive_by_default, and a disable enum whose CONDITIONAL value carries the Opus 5 case honestly. Each control owns its own default, which also answers conformance.md's open question about a per-model budget default separate from the range bounds. Validation follows the axes: each generation param is checked against the control that governs it, so a model declaring both accepts both. The all-zero ThinkingSpec stays valid for a model that does not reason, since that is the common case; only a positive claim that reasoning can be turned off contradicts supported == false. The Anthropic roster is re-expressed in the new shape without adding any vendor claim it did not already make. Notably it still declares no budget control on the 4.6 generation: the research suggests those models honor a deprecated budget_tokens, but this roster never claimed it and that claim needs its own pass against the live docs. Also records an open question this surfaced: several vendors' reasoning models reject temperature, and adapters currently infer that from the thinking shape. The proxy holds today and will break on the first model with an effort ladder that still accepts sampling params. --- api/pluggableharness/model/v1/types.proto | 140 ++++-- docs/specifications/model/conformance.md | 7 +- docs/specifications/model/data-types.md | 59 ++- docs/specifications/model/protocol.md | 7 +- internal/anthropic/catalog/catalog.go | 80 +-- internal/anthropic/catalog/catalog_test.go | 48 +- internal/anthropic/messages/request.go | 70 ++- internal/anthropic/messages/request_test.go | 28 +- internal/cost/pricing_test.go | 2 +- internal/kernel/testdata/plugin/main.go | 11 +- internal/modelrequest/CLAUDE.md | 18 +- internal/modelrequest/params.go | 44 +- internal/modelrequest/params_test.go | 103 +++- pkg/model/capabilities.go | 82 +++- pkg/model/capabilities_test.go | 101 +++- pkg/model/convert.go | 65 ++- pkg/model/convert_test.go | 128 ++++- pkg/model/model.go | 78 ++- pkg/model/proto/v1/types.pb.go | 519 +++++++++++++------- pkg/model/server_test.go | 2 +- 20 files changed, 1116 insertions(+), 476 deletions(-) diff --git a/api/pluggableharness/model/v1/types.proto b/api/pluggableharness/model/v1/types.proto index 4a8c695..a0c5a59 100644 --- a/api/pluggableharness/model/v1/types.proto +++ b/api/pluggableharness/model/v1/types.proto @@ -110,30 +110,33 @@ message ModelSpec { bool supports_documents = 12; } -// ThinkingMode enumerates the shapes of extended-reasoning control found -// across researched vendors (model.md §2) — a plain supports_thinking -// bool would lose information the kernel needs to build a correct -// request, since some vendors (e.g. Anthropic) expose more than one mode -// across their own model lineup. -enum ThinkingMode { +// ThinkingDisableSupport describes whether a model's reasoning can be +// turned off, per model/data-types.md#thinkingspec. A plain bool cannot +// express the real answer for every model: Anthropic's Opus 5 accepts an +// explicit disable at effort "high" or below and returns a 400 at "xhigh" +// or "max", so both true and false are wrong for it. +enum ThinkingDisableSupport { // Zero value. Never valid when ThinkingSpec.supported is true; its // presence on the wire means a caller forgot to set the field. - THINKING_MODE_UNSPECIFIED = 0; - // The model has no extended-reasoning capability. Pairs with - // ThinkingSpec.supported == false. - THINKING_MODE_NONE = 1; - // The model always reasons, adaptively, with no caller-selectable - // effort level or budget. - THINKING_MODE_ALWAYS_ON_ADAPTIVE = 2; - // The caller selects one of a fixed set of named effort levels - // (ThinkingSpec.effort_levels). - THINKING_MODE_DISCRETE_EFFORT = 3; - // The caller selects a token budget within ThinkingSpec.budget_range. - THINKING_MODE_CONTINUOUS_BUDGET = 4; + THINKING_DISABLE_SUPPORT_UNSPECIFIED = 0; + // Reasoning cannot be turned off in any configuration — a researched + // Grok model defaults reasoning on with no off switch, and Anthropic's + // Fable 5 returns a 400 for an explicit disable. Also the correct value + // when ThinkingSpec.supported is false: there is nothing to disable. + THINKING_DISABLE_SUPPORT_NEVER = 1; + // An explicit disable is accepted in every configuration. + THINKING_DISABLE_SUPPORT_ALWAYS = 2; + // An explicit disable is accepted in some configurations and rejected in + // others. This protocol deliberately does not model WHICH: the condition + // is vendor-specific and expressing it would need a general constraint + // language. What this value buys the kernel is knowing that a failed + // disable attempt is a vendor policy response, not an adapter bug. + THINKING_DISABLE_SUPPORT_CONDITIONAL = 3; } -// ThinkingBudgetRange bounds the token budget a caller may request when -// ThinkingMode is THINKING_MODE_CONTINUOUS_BUDGET. +// ThinkingBudgetRange bounds the token budget a caller may request on a +// model whose ThinkingSpec declares a BudgetControl. Both bounds are +// inclusive. message ThinkingBudgetRange { // The smallest thinking-token budget this model accepts. int64 min = 1; @@ -141,38 +144,81 @@ message ThinkingBudgetRange { int64 max = 2; } +// EffortControl declares that a model accepts a named reasoning-effort +// level, and which levels, per model/data-types.md#thinkingspec. +message EffortControl { + // The selectable effort levels, e.g. ["low","medium","high","xhigh", + // "max"]. MUST be non-empty — a model with no selectable levels omits + // the whole EffortControl instead. + repeated string levels = 1; + + // The level the vendor applies when a request omits effort entirely. + // MUST be set, and MUST be one of `levels` — makes the vendor's actual + // default behavior visible/auditable via GetCapabilities rather than + // hidden in adapter code, so a kernel wanting deterministic behavior can + // always send an explicit override. + string default = 2; +} + +// BudgetControl declares that a model accepts an explicit reasoning-token +// budget, and its bounds, per model/data-types.md#thinkingspec. +message BudgetControl { + // The accepted token-budget range. MUST be present. + ThinkingBudgetRange range = 1; + + // The budget the vendor applies when a request omits one. MAY be + // omitted, which means the vendor reasons zero tokens by default. + optional int64 default = 2; + + // Whether the vendor still honors this control but steers callers to + // effort/adaptive instead, and MAY remove it in a later model. This + // status is per-model, never per-vendor: Anthropic's Opus 4.6 and + // Sonnet 4.6 accept a deprecated budget, Haiku 4.5 accepts ONLY the + // budget, and Opus 4.7 onward reject it with a 400 (declared by omitting + // BudgetControl entirely, not by setting this flag). + bool deprecated = 3; +} + // ThinkingSpec describes one model's extended-reasoning capability, per -// model.md §2. +// model/data-types.md#thinkingspec. +// +// These are independent axes, not one-of-N modes. A model MAY reason +// adaptively AND expose an effort ladder (Anthropic Opus 4.8, Sonnet 5), +// or accept an effort level AND a deprecated token budget (Opus 4.6, +// Sonnet 4.6). An earlier revision modeled this as a single mutually- +// exclusive enum, which forced every such model to declare a half-truth. message ThinkingSpec { - // Whether this model has any extended-reasoning capability at all. + reserved 2 to 6; + + reserved "budget_range", "can_disable", "default", "effort_levels", "mode"; + + // Whether this model has any extended-reasoning capability at all. When + // false, effort and budget MUST both be absent and adaptive_by_default + // MUST be false; `disable` is meaningless (there is nothing to disable), + // so UNSPECIFIED and NEVER are equivalent and a reader MUST treat them + // identically. That equivalence is what keeps an all-zero ThinkingSpec a + // valid declaration for a model that does not reason — the common case. + // Only a positive claim that reasoning CAN be turned off (ALWAYS or + // CONDITIONAL) contradicts supported == false. bool supported = 1; - // Which reasoning-control shape this model uses. MUST be - // THINKING_MODE_NONE when supported == false. - ThinkingMode mode = 2; + // The named-effort-level control, present iff this model accepts one. + // Absent means sending GenerationParams.thinking_effort to this model is + // a kernel-level reject, not something forwarded to the vendor. + optional EffortControl effort = 7; - // The selectable effort levels, e.g. ["low","medium","high","xhigh", - // "max"]. MUST be non-empty when mode == THINKING_MODE_DISCRETE_EFFORT; - // meaningless otherwise. - repeated string effort_levels = 3; - - // The selectable token-budget range. MUST be present when mode == - // THINKING_MODE_CONTINUOUS_BUDGET; meaningless otherwise. - optional ThinkingBudgetRange budget_range = 4; - - // Whether reasoning can be turned off once enabled. MUST be set - // accurately — some vendors' reasoning cannot be disabled (e.g. a - // researched Grok model defaults reasoning on with no off switch). - bool can_disable = 5; - - // The effort level (discrete_effort) or budget-token value - // (continuous_budget), as a string, the vendor applies when a request - // omits thinking config entirely. MUST be set when mode != - // THINKING_MODE_NONE — makes the vendor's actual default behavior - // visible/auditable via GetCapabilities rather than hidden in adapter - // code, so a kernel wanting deterministic behavior can always send an - // explicit override. - optional string default = 6; + // The explicit-token-budget control, present iff this model accepts one. + // A model that never had one, and a model whose vendor removed it, + // both declare it absent. + optional BudgetControl budget = 8; + + // Whether omitting every thinking control still produces reasoning. + // False means an unconfigured request reasons zero tokens. + bool adaptive_by_default = 9; + + // Whether, and when, reasoning can be turned off. MUST be set when + // supported is true. + ThinkingDisableSupport disable = 10; } // CachingMode enumerates the prompt-caching mechanics found across diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index f73ca6d..3c07506 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -30,7 +30,9 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | `tool_use` / `tool_result` | MUST, if any served model has `supports_tool_use = true` | | | `image` (vision) | MUST support where `supports_vision = true`; MUST reject cleanly where `false` | | | `document` | MUST support where `supports_documents = true`; MUST reject cleanly where `false` | [`data-types.md#canonical-message--content-block-schema`](data-types.md#canonical-message--content-block-schema) — mirrors `image`/`supports_vision`'s rule | -| Extended thinking/reasoning | MAY, capability-gated via `ThinkingSpec` | declare `mode` precisely, don't collapse to a bool | +| Extended thinking/reasoning | MAY, capability-gated via `ThinkingSpec` | declare each axis it actually accepts, don't collapse to a bool or to one mutually-exclusive mode | +| `ThinkingSpec.effort.default` / `budget.default` | MUST when that control is present | [`data-types.md#thinkingspec`](data-types.md#thinkingspec) — `effort.default` names a level; `budget.default` MAY be omitted, meaning zero reasoning tokens by default | +| `ThinkingSpec.adaptive_by_default` / `disable` | MUST | [`data-types.md#thinkingspec`](data-types.md#thinkingspec) — `disable = conditional` tells the kernel a disable attempt MAY legitimately fail, so such a failure is vendor policy, not an adapter bug | | `StreamEvent.redacted_thinking` | MUST, for a vendor that emits vendor-encrypted reasoning blocks | [`data-types.md#streamevent`](data-types.md#streamevent) — a whole block, never fragmented; stored and echoed back verbatim or the vendor rejects the whole conversation on a later turn | | Prompt caching | MAY, capability-gated via `CachingSpec` | declare `mode` (explicit vs. implicit) | | Cache breakpoints (`StreamCompletionRequest.cache_breakpoints`) | MUST honor where `CachingSpec.mode = CACHING_MODE_EXPLICIT_MARKERS`; MUST ignore otherwise | [`protocol.md#cache-breakpoint-placement-policy`](protocol.md#cache-breakpoint-placement-policy) — placement is a kernel decision, never the plugin's | @@ -39,7 +41,6 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | Tool-choice constraint (`GenerationParams.tool_choice`) | MAY, capability-gated via `ModelSpec.supported_tool_choice_modes` | kernel MUST NOT send a mode absent from the declared list, mirroring `ThinkingSpec` validation | | `Render` | MAY | generic fallback exists; `RenderRequest.schema_version` MUST be set when implemented | | `CountTokens` | SHOULD | kernel falls back to [`kernel-callbacks.md`](../kernel-callbacks.md#the-fallback-heuristic)'s heuristic when absent, treated as a last resort; `CountTokensRequest.model_id` MUST be set | -| `ThinkingSpec.default` | MUST when `mode != none` | [`data-types.md`](data-types.md#thinkingspec) | | `CachingSpec.keepalive_supported` | MUST (field); actual keepalive loop MAY | [`data-types.md`](data-types.md#cachingspec) | | `Pricing.tiers`, time-bounded/tiered/input-size-bounded rates | MUST | [`data-types.md`](data-types.md#pricing) — exactly one tier MUST match any given `(timestamp, input_token_count)` pair | | `Pricing` on every `ModelSpec` | MUST | required even for `free: true` models | @@ -52,7 +53,7 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded ## Open questions - Whether `supports_parallel_tool_calls` needs a per-request override (some vendors may allow disabling parallel calls per-call even when generally supported). -- Whether `ThinkingSpec.budget_range` needs a per-model default separate from the overall min/max (several vendors default to a specific level like `medium`/`HIGH` rather than "off"). +- Whether a model needs to declare that it rejects `GenerationParams.temperature`. Several vendors' reasoning models reject non-default sampling parameters outright (Anthropic's effort-ladder models return a 400; other vendors' reasoning models ignore the value silently). There is no field for this, so an adapter facing such a model can only drop the operator's `temperature` on the floor — which is the right behavior, but it happens invisibly, and the kernel cannot tell a dropped parameter from an honored one. Today adapters infer it from the thinking shape, which is a proxy that will be wrong for the first model that has an effort ladder *and* accepts temperature. The same question applies to `top_p` and any other sampling parameter added later, so the fix is probably a general "declared sampling parameters" list rather than a per-parameter bool. - Retry/backoff policy specifics (exponential backoff parameters) — likely belongs in the kernel's routing logic rather than this protocol, but needs to be decided somewhere; see [`configuration/blocks-reference.md`](../configuration/blocks-reference.md)'s `settings{}` retry defaults for the current kernel-side values. - Whether `content_filtered` needs sub-categories (input filtered vs. output filtered) — there isn't enough vendor detail yet to decide. - `Pricing.currency` is declared as a string but v1 only ever acts on `"USD"` — no conversion mechanism, no mixed-currency cost aggregation across providers with different currencies. Fine while vendors generally price in USD; would need real design work the moment that stops being true. diff --git a/docs/specifications/model/data-types.md b/docs/specifications/model/data-types.md index 25f5066..269dffe 100644 --- a/docs/specifications/model/data-types.md +++ b/docs/specifications/model/data-types.md @@ -23,29 +23,54 @@ ModelSpec { } ``` -Rationale for the sum-type shape of `ThinkingSpec`/`CachingSpec` below: all three thinking modes and both caching modes are in active use across real vendors, sometimes multiple modes on different models *from the same vendor* (Anthropic's newer models use adaptive thinking; Haiku 4.5 only supports the older discrete `budget_tokens` style). A boolean `supports_thinking` flag would lose information the kernel actually needs to build a correct request. +Rationale for the shape of `ThinkingSpec`/`CachingSpec` below: a boolean `supports_thinking`/`supports_caching` flag would lose information the kernel actually needs to build a correct request, because real vendors differ in *how* the capability is controlled, not only in whether it exists — sometimes across models from the same vendor. + +**Both are sets of independent axes, not one-of-N modes.** This is the correction to an earlier design that modeled each as a single mutually-exclusive enum. Real models occupy more than one position at once, and a single-valued enum forces a provider to declare a half-truth: + +- A model MAY reason adaptively *and* expose a named effort ladder simultaneously. Anthropic's Opus 4.8 and Sonnet 5 are exactly this — omitting thinking config runs adaptive reasoning, and `output_config.effort` selects a level on top of it. An adapter for such a model sends both controls in one request. +- A model MAY accept a named effort level *and* an explicit token budget, with the budget deprecated but still functional. Anthropic's Opus 4.6 and Sonnet 4.6 are this; Haiku 4.5 accepts only the budget and errors on effort; Opus 4.7 and later reject the budget with a 400. The status is per-model, never per-vendor, and a spec shape that cannot say so pushes the distinction into adapter code where no caller can see it. +- Whether reasoning can be turned off is not always a yes or no. Anthropic's Opus 5 accepts an explicit disable at effort `high` or below and rejects it at `xhigh` or `max` — so both `true` and `false` are wrong, and the honest answer needs a third value. + +Each axis below is therefore declared on its own. A model declares every control it actually accepts, and the kernel validates a requested parameter against the specific control that governs it rather than against a mode that stands in for a whole model family. ### `ThinkingSpec` ```protobuf ThinkingSpec { - supported bool - mode enum { none, always_on_adaptive, discrete_effort, continuous_budget } - effort_levels []string // required if mode == discrete_effort, e.g. ["low","medium","high","xhigh","max"] - budget_range {min, max} // required if mode == continuous_budget (token count range) - can_disable bool // MUST — some vendors' reasoning cannot be turned off (e.g. a - // Grok model defaulting reasoning on with no off switch) - default string? // MUST when mode != none — the effort level (discrete_effort) - // or budget-token value (continuous_budget, as a string) the - // vendor applies when a request omits thinking config entirely. - // Makes the actual default behavior visible/auditable in - // GetCapabilities instead of hidden in adapter code — a kernel - // wanting deterministic behavior can read this and always send - // an explicit override rather than guessing what "unspecified" - // means for a given model. + supported bool // MUST — whether this model reasons at all + effort EffortControl? // present iff the model accepts a named effort level + budget BudgetControl? // present iff the model accepts an explicit token budget + adaptive_by_default bool // MUST — whether omitting every thinking control still reasons + disable enum { unspecified, never, always, conditional } // MUST when supported +} + +EffortControl { + levels []string // MUST be non-empty, e.g. ["low","medium","high","xhigh","max"] + default string // MUST — the level the vendor applies when a request omits effort +} + +BudgetControl { + range {min, max} // MUST — the accepted token-budget range, inclusive + default int64? // MAY — the budget the vendor applies when a request omits one; + // omitted means the vendor reasons zero tokens by default + deprecated bool // MUST — the vendor still honors this control but steers callers + // to effort/adaptive instead, and MAY remove it in a later model } ``` +`supported == false` means the model has no reasoning capability: `effort` and `budget` MUST both be absent and `adaptive_by_default` MUST be false. `disable` is meaningless in that case — there is nothing to disable — so both `unspecified` and `never` are accepted and mean the same thing, and a reader MUST treat them identically. This keeps the all-zero `ThinkingSpec` a valid declaration for a model that does not reason, which is the common case; only a positive claim that reasoning *can* be turned off (`always` or `conditional`) contradicts `supported == false` and MUST be rejected. Every other combination is a real, declarable position: + +| Axis | Meaning when present/true | +|---|---| +| `effort` | The caller MAY select one of `levels`. Absent means this model has no effort ladder — sending an effort level is a kernel-level reject, not something forwarded. | +| `budget` | The caller MAY select a token budget inside `range`. Absent means this model has no budget control; a model that once had one and had it removed declares it absent, not `deprecated`. | +| `adaptive_by_default` | Omitting both controls still produces reasoning. False means an unconfigured request reasons zero tokens. This is what makes the vendor's actual default behavior auditable through `GetCapabilities` rather than hidden in adapter code — a kernel wanting deterministic behavior reads it and sends an explicit override instead of guessing what "unspecified" means. | +| `disable = never` | Reasoning cannot be turned off at all (a Grok model defaulting reasoning on with no off switch; Anthropic's Fable 5, where an explicit disable is a 400). | +| `disable = always` | An explicit disable is accepted in every configuration. | +| `disable = conditional` | An explicit disable is accepted in some configurations and rejected in others — Anthropic's Opus 5 accepts it at effort `high` or below and returns a 400 at `xhigh` or `max`. The protocol deliberately does not model *which* configurations: the condition is vendor-specific and would need a general constraint language to express. What `conditional` buys the kernel is the knowledge that a disable attempt MAY legitimately fail, so such a failure is a vendor policy response and not an adapter bug. | + +`EffortControl.default` and `BudgetControl.default` replace a single `default` string that had to encode either kind of value. Each now sits on the control it belongs to, in that control's own type, which also settles the question of what a per-model budget default means separately from the range's bounds. + ### `CachingSpec` ```protobuf @@ -240,8 +265,8 @@ This is a `Struct` for the same reason `ConfigureRequest.config` is one — the ```protobuf GenerationParams { - thinking_effort string? // one of ThinkingSpec.effort_levels; THINKING_MODE_DISCRETE_EFFORT only - thinking_budget_tokens int64? // within ThinkingSpec.budget_range; THINKING_MODE_CONTINUOUS_BUDGET only + thinking_effort string? // one of ThinkingSpec.effort.levels; requires effort to be present + thinking_budget_tokens int64? // within ThinkingSpec.budget.range; requires budget to be present max_output_tokens int64? // per-request override of ModelSpec.max_output_tokens temperature double? // sampling temperature; vendor-specific range/semantics, passed through as-is stop_sequences []string // sequences that MUST stop generation before they're produced diff --git a/docs/specifications/model/protocol.md b/docs/specifications/model/protocol.md index 5826c23..50a86b8 100644 --- a/docs/specifications/model/protocol.md +++ b/docs/specifications/model/protocol.md @@ -46,7 +46,12 @@ A plugin whose backend does not natively stream (batch-only) MUST still implemen ### Generation-parameter validation and capability-aware routing -`GenerationParams.thinking_effort`/`thinking_budget_tokens` MUST be validated against the resolved model's declared [`ThinkingSpec`](data-types.md#thinkingspec) before the request is dispatched to the plugin — an effort level outside `ThinkingSpec.effort_levels`, or a budget outside `ThinkingSpec.budget_range`, is a kernel-level reject-or-fallback, not something sent to the vendor and left to surface as a raw API error three layers up the stack. A caller (the turn loop, a sub-agent spawn) that needs a parameter the resolved model doesn't support MUST either drop back to that model's default behavior or fail the selection, never forward an invalid combination. +`GenerationParams.thinking_effort`/`thinking_budget_tokens` MUST be validated against the resolved model's declared [`ThinkingSpec`](data-types.md#thinkingspec) before the request is dispatched to the plugin — each against the specific control that governs it, since the two are independent axes and a model MAY declare either, both, or neither: + +- `thinking_effort` requires `ThinkingSpec.effort` to be present, and MUST be one of its `levels`. +- `thinking_budget_tokens` requires `ThinkingSpec.budget` to be present, and MUST fall inside its `range`. + +A parameter naming a control the resolved model does not declare, or a value outside that control's declared domain, is a kernel-level reject-or-fallback — not something sent to the vendor and left to surface as a raw API error three layers up the stack. A caller (the turn loop, a sub-agent spawn) that needs a parameter the resolved model doesn't support MUST either drop back to that model's default behavior or fail the selection, never forward an invalid combination. Sending both parameters to a model declaring both controls is legal; how the vendor reconciles them is that adapter's concern. `GenerationParams.tool_choice.mode` follows the identical rule against `ModelSpec.supported_tool_choice_modes`: a mode the resolved model doesn't declare support for MUST NOT be forwarded to the vendor — reject or fall back to `TOOL_CHOICE_MODE_AUTO` (equivalent to omitting `tool_choice`) at the kernel level, same as an out-of-range thinking param. See [`data-types.md#generationparams`](data-types.md#generationparams). diff --git a/internal/anthropic/catalog/catalog.go b/internal/anthropic/catalog/catalog.go index 4379322..96c8d07 100644 --- a/internal/anthropic/catalog/catalog.go +++ b/internal/anthropic/catalog/catalog.go @@ -116,23 +116,39 @@ func base(id string, contextWindow, maxOutput int64) model.Spec { } } -// effortThinking builds the ThinkingSpec for a model whose reasoning is -// controlled by a named effort level (output_config.effort) rather than a -// token budget. canDisable reports whether thinking can be turned off at -// all — see fable5 and opus5 for the two models where that answer is not -// a plain yes. +// effortThinking builds the ThinkingSpec for a model that reasons +// adaptively and exposes a named effort ladder (output_config.effort) on +// top of it. disable says whether, and when, that reasoning can be turned +// off — see fable5 and opus5 for the two models where the answer is not a +// plain yes. +// +// AdaptiveByDefault is true for every model built here: omitting thinking +// config entirely still reasons, and the adapter sends +// thinking:{type:"adaptive"} alongside the effort level rather than +// instead of it. Both facts are declarable now that ThinkingSpec's axes +// are independent; the earlier single-mode shape could only say one, and +// said the effort half. +// +// None of these models declares a BudgetControl. Anthropic removed +// budget_tokens outright on Opus 4.7 and later, and while the 4.6 +// generation reportedly still honors it transitionally, this roster has +// never claimed that and adding the claim needs its own pass against the +// live docs — see this package's CLAUDE.md on never writing a capability +// here from memory. // // levels is copied rather than aliased: effortLevels4/effortLevels5 are // package-level slices shared by several models, so handing one straight // to a caller would let that caller's mutation reach every later Models() // call, defeating the whole point of rebuilding the roster per call. -func effortThinking(levels []string, canDisable bool) model.ThinkingSpec { +func effortThinking(levels []string, disable modelv1.ThinkingDisableSupport) model.ThinkingSpec { return model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT, - EffortLevels: append([]string(nil), levels...), - CanDisable: canDisable, - Default: defaultEffort, + Supported: true, + Effort: &model.EffortControl{ + Levels: append([]string(nil), levels...), + Default: defaultEffort, + }, + AdaptiveByDefault: true, + Disable: disable, } } @@ -190,7 +206,7 @@ func opusPricing() model.Pricing { // make it a routing candidate that fails at request time. func fable5() model.Spec { s := base("claude-fable-5", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels5, false) + s.Thinking = effortThinking(effortLevels5, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER) s.Pricing = flatPricing(10.00, 50.00, 12.50, 1.00, 5.00, 25.00) return s } @@ -208,7 +224,7 @@ func fable5() model.Spec { // kernel explicitly asks for both. func opus5() model.Spec { s := base("claude-opus-5", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels5, true) + s.Thinking = effortThinking(effortLevels5, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL) s.Pricing = opusPricing() return s } @@ -217,7 +233,7 @@ func opus5() model.Spec { // current and the recommended fallback target for an Opus 5 refusal. func opus48() model.Spec { s := base("claude-opus-4-8", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels5, true) + s.Thinking = effortThinking(effortLevels5, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS) s.Pricing = opusPricing() return s } @@ -225,7 +241,7 @@ func opus48() model.Spec { // opus47 is Claude Opus 4.7. func opus47() model.Spec { s := base("claude-opus-4-7", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels5, true) + s.Thinking = effortThinking(effortLevels5, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS) s.Pricing = opusPricing() return s } @@ -234,7 +250,7 @@ func opus47() model.Spec { // level existed — hence effortLevels4 rather than effortLevels5. func opus46() model.Spec { s := base("claude-opus-4-6", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels4, true) + s.Thinking = effortThinking(effortLevels4, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS) s.Pricing = opusPricing() return s } @@ -250,7 +266,7 @@ func opus46() model.Spec { // model.NewCapabilities checks for overlap. func sonnet5() model.Spec { s := base("claude-sonnet-5", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels5, true) + s.Thinking = effortThinking(effortLevels5, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS) introEnd := sonnet5IntroEnd standardStart := sonnet5IntroEnd @@ -292,7 +308,7 @@ func sonnet5() model.Spec { // sonnet46 is Claude Sonnet 4.6. func sonnet46() model.Spec { s := base("claude-sonnet-4-6", contextWindow1M, maxOutput128K) - s.Thinking = effortThinking(effortLevels4, true) + s.Thinking = effortThinking(effortLevels4, modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS) s.Pricing = flatPricing(3.00, 15.00, 3.75, 0.30, 1.50, 7.50) return s } @@ -303,22 +319,28 @@ func sonnet46() model.Spec { // // The budget range's upper bound is one token below MaxOutputTokens // because Anthropic requires budget_tokens < max_tokens; the lower bound -// is Anthropic's documented 1024 minimum. Default is "0" rather than an -// effort name because omitting the thinking parameter on this model means -// no thinking at all — ThinkingSpec.default documents what the vendor -// actually does with an unconfigured request, and for Haiku 4.5 that is -// zero reasoning tokens. +// is Anthropic's documented 1024 minimum. +// +// It declares no EffortControl — output_config.effort errors on this +// model — and AdaptiveByDefault is false, because omitting the thinking +// parameter here means no thinking at all rather than adaptive reasoning. +// That pair is the exact opposite of every other model in this roster, and +// it is the case the older single-mode ThinkingSpec handled worst: a +// nil BudgetControl.Default now says "zero reasoning tokens by default" +// directly, where before it had to be smuggled through a Default field +// typed as a string holding "0". func haiku45() model.Spec { s := base("claude-haiku-4-5", contextWindow200K, maxOutput64K) s.Thinking = model.ThinkingSpec{ Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET, - BudgetRange: &model.ThinkingBudgetRange{ - Min: 1024, - Max: maxOutput64K - 1, + Budget: &model.BudgetControl{ + Range: model.ThinkingBudgetRange{ + Min: 1024, + Max: maxOutput64K - 1, + }, }, - CanDisable: true, - Default: "0", + AdaptiveByDefault: false, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, } s.Pricing = flatPricing(1.00, 5.00, 1.25, 0.10, 0.50, 2.50) return s diff --git a/internal/anthropic/catalog/catalog_test.go b/internal/anthropic/catalog/catalog_test.go index 5c27ce5..894e773 100644 --- a/internal/anthropic/catalog/catalog_test.go +++ b/internal/anthropic/catalog/catalog_test.go @@ -58,7 +58,7 @@ func TestModels_returnsAFreshCopy(t *testing.T) { first := Models() first[0].ID = "mutated" first[0].Pricing.Tiers[0].InputPerMtok = 999 - first[0].Thinking.EffortLevels[0] = "mutated" + first[0].Thinking.Effort.Levels[0] = "mutated" second := Models() if second[0].ID == "mutated" { @@ -67,8 +67,8 @@ func TestModels_returnsAFreshCopy(t *testing.T) { if second[0].Pricing.Tiers[0].InputPerMtok == 999 { t.Error("mutating a returned PricingTier changed the next call's roster") } - if second[0].Thinking.EffortLevels[0] == "mutated" { - t.Error("mutating a returned EffortLevels changed the next call's roster") + if second[0].Thinking.Effort.Levels[0] == "mutated" { + t.Error("mutating a returned effort Levels slice changed the next call's roster") } } @@ -184,7 +184,7 @@ func tierCovers(tier model.PricingTier, at time.Time) bool { // already enforces — specifically that an effort-controlled model quotes a // ladder containing its own declared default, and that a budget-controlled // model's range is ordered and fits inside its output ceiling. -func TestThinking_modeMatchesTheDeclaredControls(t *testing.T) { +func TestThinking_declaredControlsAreInternallyConsistent(t *testing.T) { t.Parallel() for _, m := range Models() { @@ -192,14 +192,17 @@ func TestThinking_modeMatchesTheDeclaredControls(t *testing.T) { t.Errorf("%s: every model in this roster reasons; Supported is false", m.ID) continue } - switch m.Thinking.Mode { - case modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT: - if !contains(m.Thinking.EffortLevels, m.Thinking.Default) { - t.Errorf("%s: default effort %q is absent from %v", - m.ID, m.Thinking.Default, m.Thinking.EffortLevels) + if m.Thinking.Effort == nil && m.Thinking.Budget == nil { + t.Errorf("%s: reasoning declared with neither an effort nor a budget control", m.ID) + continue + } + if e := m.Thinking.Effort; e != nil { + if !contains(e.Levels, e.Default) { + t.Errorf("%s: default effort %q is absent from %v", m.ID, e.Default, e.Levels) } - case modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET: - r := m.Thinking.BudgetRange + } + if b := m.Thinking.Budget; b != nil { + r := b.Range if r.Min <= 0 || r.Min >= r.Max { t.Errorf("%s: budget range [%d,%d] is not an ordered positive range", m.ID, r.Min, r.Max) } @@ -207,8 +210,27 @@ func TestThinking_modeMatchesTheDeclaredControls(t *testing.T) { t.Errorf("%s: budget max %d is not below max output %d, which the vendor rejects", m.ID, r.Max, m.MaxOutputTokens) } - default: - t.Errorf("%s: unexpected thinking mode %v", m.ID, m.Thinking.Mode) + } + if m.Thinking.Disable == modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_UNSPECIFIED { + t.Errorf("%s: reasoning declared without a disable value", m.ID) + } + } +} + +// TestThinking_effortModelsAreAdaptiveByDefault pins the pairing the older +// single-mode ThinkingSpec could not express and that +// internal/anthropic/messages relies on: Anthropic's effort ladder rides +// on top of adaptive reasoning rather than replacing it, so buildThinking +// emits thinking:{type:"adaptive"} AND output_config.effort together. +func TestThinking_effortModelsAreAdaptiveByDefault(t *testing.T) { + t.Parallel() + + for _, m := range Models() { + if m.Thinking.Effort == nil { + continue + } + if !m.Thinking.AdaptiveByDefault { + t.Errorf("%s: declares an effort ladder but not AdaptiveByDefault", m.ID) } } } diff --git a/internal/anthropic/messages/request.go b/internal/anthropic/messages/request.go index 1760963..1611181 100644 --- a/internal/anthropic/messages/request.go +++ b/internal/anthropic/messages/request.go @@ -105,11 +105,19 @@ func BuildRequest(in *modelv1.StreamCompletionRequest, spec model.Spec) (*Reques return nil, err } - // Models on the discrete-effort thinking ladder reject temperature - // outright (a 400 from the vendor); models on continuous-budget - // thinking (or with no thinking capability at all) still accept it. + // Models on the effort ladder reject temperature outright (a 400 from + // the vendor); models on a token budget, or with no thinking capability + // at all, still accept it. + // + // Presence of an EffortControl is a proxy for "rejects sampling + // params", not a statement about thinking as such — the protocol has no + // field for the latter, which + // docs/specifications/model/conformance.md's open questions records. + // The proxy holds for every Anthropic model in this roster and will + // need revisiting for the first model that has an effort ladder and + // still accepts temperature. var temperature *float64 - if params != nil && params.Temperature != nil && spec.Thinking.Mode != modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT { + if params != nil && params.Temperature != nil && spec.Thinking.Effort == nil { t := *params.Temperature temperature = &t } @@ -203,35 +211,49 @@ func buildToolChoice(params *modelv1.GenerationParams) (*ToolChoice, error) { } // buildThinking translates params' thinking-control fields into Anthropic's -// Thinking/OutputConfig pair, driven by spec's declared ThinkingSpec.Mode. +// Thinking/OutputConfig pair, against whichever controls spec declares. +// +// The effort and budget controls are independent axes +// (docs/specifications/model/data-types.md#thinkingspec), so this checks +// each requested param against the control that governs it rather than +// switching on one mode. Effort is checked first: no Anthropic model +// currently declares both, but if one ever does, sending the effort ladder +// alongside adaptive thinking is the shape Anthropic documents, and a +// budget would then be the deprecated path. +// +// Note that an effort request yields BOTH thinking:{type:"adaptive"} and +// output_config.effort — Anthropic's effort ladder rides on top of +// adaptive reasoning rather than replacing it, which is exactly the fact +// the old single-mode ThinkingSpec could not declare. func buildThinking(params *modelv1.GenerationParams, spec model.Spec) (*Thinking, *OutputConfig, error) { - switch spec.Thinking.Mode { - case modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT: - if params == nil || params.ThinkingEffort == nil { - return nil, nil, nil - } + if params == nil { + return nil, nil, nil + } + + if effort := spec.Thinking.Effort; effort != nil && params.ThinkingEffort != nil { level := *params.ThinkingEffort - if !slices.Contains(spec.Thinking.EffortLevels, level) { - return nil, nil, newInvalidRequestError("thinking effort %q is not one of model %q's effort levels %v", level, spec.ID, spec.Thinking.EffortLevels) + if !slices.Contains(effort.Levels, level) { + return nil, nil, newInvalidRequestError("thinking effort %q is not one of model %q's effort levels %v", level, spec.ID, effort.Levels) } return &Thinking{Type: thinkingTypeAdaptive}, &OutputConfig{Effort: level}, nil + } - case modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET: - if params == nil || params.ThinkingBudgetTokens == nil { - return nil, nil, nil - } - budget := *params.ThinkingBudgetTokens - if budget == 0 && spec.Thinking.CanDisable { + if budget := spec.Thinking.Budget; budget != nil && params.ThinkingBudgetTokens != nil { + want := *params.ThinkingBudgetTokens + // A budget of zero is how a caller asks for no reasoning at all, + // which Anthropic expresses as an explicit disable rather than as a + // zero budget — and which is only legal where the model actually + // permits disabling. + if want == 0 && spec.Thinking.Disable != modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER { return &Thinking{Type: thinkingTypeDisabled}, nil, nil } - if spec.Thinking.BudgetRange == nil || budget < spec.Thinking.BudgetRange.Min || budget > spec.Thinking.BudgetRange.Max { - return nil, nil, newInvalidRequestError("thinking budget %d is outside model %q's budget range", budget, spec.ID) + if want < budget.Range.Min || want > budget.Range.Max { + return nil, nil, newInvalidRequestError("thinking budget %d is outside model %q's budget range", want, spec.ID) } - return &Thinking{Type: thinkingTypeEnabled, BudgetTokens: &budget}, nil, nil - - default: - return nil, nil, nil + return &Thinking{Type: thinkingTypeEnabled, BudgetTokens: &want}, nil, nil } + + return nil, nil, nil } // applyCacheBreakpoints translates the kernel's cache breakpoints into diff --git a/internal/anthropic/messages/request_test.go b/internal/anthropic/messages/request_test.go index 1fe1fd1..ccbc706 100644 --- a/internal/anthropic/messages/request_test.go +++ b/internal/anthropic/messages/request_test.go @@ -27,11 +27,13 @@ func fullSpec() model.Spec { SupportsVision: true, SupportsDocuments: true, Thinking: model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT, - EffortLevels: []string{"low", "medium", "high"}, - CanDisable: false, - Default: "medium", + Supported: true, + Effort: &model.EffortControl{ + Levels: []string{"low", "medium", "high"}, + Default: "medium", + }, + AdaptiveByDefault: true, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER, }, Caching: model.CachingSpec{ Supported: true, @@ -44,16 +46,20 @@ func fullSpec() model.Spec { // discrete effort, used to exercise the budget-token path and the // temperature-inclusion rule (only the effort ladder rejects temperature). func budgetSpec(canDisable bool) model.Spec { + disable := modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER + if canDisable { + disable = modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS + } return model.Spec{ ID: "claude-legacy", MaxOutputTokens: 4096, SupportsToolUse: true, Thinking: model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET, - BudgetRange: &model.ThinkingBudgetRange{Min: 1024, Max: 32000}, - CanDisable: canDisable, - Default: "4096", + Supported: true, + Budget: &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 1024, Max: 32000}, + }, + Disable: disable, }, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, } @@ -66,7 +72,7 @@ func minimalSpec() model.Spec { return model.Spec{ ID: "claude-minimal", MaxOutputTokens: 2048, - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + Thinking: model.ThinkingSpec{}, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, } } diff --git a/internal/cost/pricing_test.go b/internal/cost/pricing_test.go index fcfd469..69c7f0c 100644 --- a/internal/cost/pricing_test.go +++ b/internal/cost/pricing_test.go @@ -356,7 +356,7 @@ func TestGapDetectionCatchesWhatPkgModelMisses(t *testing.T) { } spec := model.Spec{ ID: "gap-fixture-model", - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + Thinking: model.ThinkingSpec{}, Pricing: sdkSide, } if _, err := model.NewCapabilities([]model.Spec{spec}, &configv1.ConfigSchema{}); err != nil { diff --git a/internal/kernel/testdata/plugin/main.go b/internal/kernel/testdata/plugin/main.go index 55eedfe..a5ba4cc 100644 --- a/internal/kernel/testdata/plugin/main.go +++ b/internal/kernel/testdata/plugin/main.go @@ -80,10 +80,13 @@ func (*fixtureProvider) Capabilities(context.Context) (*model.Capabilities, erro MaxOutputTokens: 8192, SupportsToolUse: true, SupportsStreaming: true, - // Both capability sub-specs must name an explicit NONE mode: the - // zero value is THINKING_MODE_UNSPECIFIED/CACHING_MODE_UNSPECIFIED, - // which NewCapabilities rejects rather than guessing at. - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + // A model with no thinking capability leaves ThinkingSpec at its + // zero value: Supported false, no effort or budget control, and + // disable NEVER — there is nothing to disable. CachingSpec must + // still name an explicit NONE mode, whose zero value is + // CACHING_MODE_UNSPECIFIED and which NewCapabilities rejects rather + // than guessing at. + Thinking: model.ThinkingSpec{}, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, Pricing: model.Pricing{Currency: "USD", Free: true}, SupportedToolChoiceModes: []modelv1.ToolChoiceMode{ diff --git a/internal/modelrequest/CLAUDE.md b/internal/modelrequest/CLAUDE.md index 9c469aa..5260657 100644 --- a/internal/modelrequest/CLAUDE.md +++ b/internal/modelrequest/CLAUDE.md @@ -19,14 +19,16 @@ remove the special case to "simplify" it into a single `slices.Contains` call, or a model declaring an empty `supported_tool_choice_modes` would incorrectly reject an explicit `AUTO` request. -- **Thinking validation is mode-gated, not just range-gated.** A - `thinking_budget_tokens` value can sit numerically inside some - `ThinkingBudgetRange` and still be invalid, if the resolved model's - `ThinkingSpec.mode` isn't `THINKING_MODE_CONTINUOUS_BUDGET` (same for - `thinking_effort` against `THINKING_MODE_DISCRETE_EFFORT`). Check the - mode first in `budgetInRange`/the effort branch of `ValidateParams`, - not just the numeric/list membership — `TestValidateParamsThinkingEffort`'s - "mode mismatch" case is the regression guard for this. +- **Thinking validation is control-gated, and the two controls are + independent.** `ThinkingSpec` declares an optional `effort` control and + an optional `budget` control as separate axes — a model may declare + either, both, or neither — so each param is validated against the + control that governs it, never against a single mode standing in for + the whole model. `budgetInRange` checks `GetBudget() == nil` *before* + reading the range: a nil control's zero-valued range would otherwise + coincidentally accept a budget of exactly 0. The effort branch needs no + such guard only because `GetEffort().GetLevels()` on a nil control is an + empty slice, which fails membership for every value. - **`ValidateContent` returns the *first* violation, in message-then-block order.** It does not collect every unsupported block in one request — matching the brief's "returns `ErrUnsupportedContent` naming the first diff --git a/internal/modelrequest/params.go b/internal/modelrequest/params.go index a0ba72d..6b916b4 100644 --- a/internal/modelrequest/params.go +++ b/internal/modelrequest/params.go @@ -39,13 +39,16 @@ type Params struct { // protocol.md#generation-parameter-validation-and-capability-aware-routing's // fallback rules: // -// - Resolved.thinking_effort is cleared unless spec's ThinkingSpec.mode -// is THINKING_MODE_DISCRETE_EFFORT and the requested value appears in -// ThinkingSpec.effort_levels. -// - Resolved.thinking_budget_tokens is cleared unless spec's -// ThinkingSpec.mode is THINKING_MODE_CONTINUOUS_BUDGET and the -// requested value falls within ThinkingSpec.budget_range (inclusive of -// both bounds). +// - Resolved.thinking_effort is cleared unless spec declares a +// ThinkingSpec.effort control and the requested value appears in its +// levels. +// - Resolved.thinking_budget_tokens is cleared unless spec declares a +// ThinkingSpec.budget control and the requested value falls within its +// range (inclusive of both bounds). +// +// The two thinking controls are independent axes, so each is validated +// against the control that governs it. A model declaring both is legal and +// both params survive; a model declaring neither falls back on both. // - Resolved.tool_choice is cleared (equivalent to // TOOL_CHOICE_MODE_AUTO, i.e. omitting tool_choice entirely) unless // its mode is TOOL_CHOICE_MODE_AUTO — which never needs a capability @@ -86,8 +89,11 @@ func ValidateParams(req *modelv1.GenerationParams, spec *modelv1.ModelSpec) Para thinking := spec.GetThinking() if resolved.ThinkingEffort != nil { - if thinking.GetMode() != modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT || - !slices.Contains(thinking.GetEffortLevels(), resolved.GetThinkingEffort()) { + // GetEffort() is nil for a model with no effort ladder, and + // GetLevels() on a nil control is an empty slice — so a model that + // declares no effort control fails the membership check and falls + // back, which is the intended outcome. + if !slices.Contains(thinking.GetEffort().GetLevels(), resolved.GetThinkingEffort()) { resolved.ThinkingEffort = nil fellBackThinking = true } @@ -113,18 +119,20 @@ func ValidateParams(req *modelv1.GenerationParams, spec *modelv1.ModelSpec) Para } } -// budgetInRange reports whether budget falls within thinking's -// budget_range, inclusive of both bounds, and thinking's mode actually -// governs a token budget at all. A thinking with mode != -// THINKING_MODE_CONTINUOUS_BUDGET has no meaningful budget_range per -// data-types.md's ThinkingSpec doc ("required if mode == -// continuous_budget"), so any explicit budget request against such a -// model is out of range regardless of the numeric value. +// budgetInRange reports whether budget falls within thinking's declared +// budget control, inclusive of both bounds. +// +// A model declaring no budget control has no meaningful range, so any +// explicit budget request against it is out of range regardless of the +// numeric value — the nil control is checked first rather than relying on +// a nil range's zero bounds, which would coincidentally accept a budget of +// exactly 0. func budgetInRange(budget int64, thinking *modelv1.ThinkingSpec) bool { - if thinking.GetMode() != modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET { + b := thinking.GetBudget() + if b == nil { return false } - r := thinking.GetBudgetRange() + r := b.GetRange() if r == nil { return false } diff --git a/internal/modelrequest/params_test.go b/internal/modelrequest/params_test.go index 49c9838..3c32b95 100644 --- a/internal/modelrequest/params_test.go +++ b/internal/modelrequest/params_test.go @@ -9,22 +9,50 @@ import ( func strPtr(s string) *string { return &s } func i64Ptr(v int64) *int64 { return &v } +// discreteEffortSpec declares a model with an effort ladder and no budget +// control. func discreteEffortSpec(levels ...string) *modelv1.ModelSpec { + def := "" + if len(levels) > 0 { + def = levels[0] + } return &modelv1.ModelSpec{ Thinking: &modelv1.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT, - EffortLevels: levels, + Supported: true, + Effort: &modelv1.EffortControl{Levels: levels, Default: def}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, }, } } +// continuousBudgetSpec declares a model with a budget control and no +// effort ladder. func continuousBudgetSpec(lo, hi int64) *modelv1.ModelSpec { return &modelv1.ModelSpec{ Thinking: &modelv1.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET, - BudgetRange: &modelv1.ThinkingBudgetRange{Min: lo, Max: hi}, + Supported: true, + Budget: &modelv1.BudgetControl{ + Range: &modelv1.ThinkingBudgetRange{Min: lo, Max: hi}, + }, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + }, + } +} + +// bothControlsSpec declares a model carrying an effort ladder AND a budget +// control at once — the position the earlier single-mode ThinkingSpec +// could not express, and the reason this package validates each param +// against its own control instead of switching on one mode. +func bothControlsSpec(levels []string, lo, hi int64) *modelv1.ModelSpec { + return &modelv1.ModelSpec{ + Thinking: &modelv1.ThinkingSpec{ + Supported: true, + Effort: &modelv1.EffortControl{Levels: levels, Default: levels[0]}, + Budget: &modelv1.BudgetControl{ + Range: &modelv1.ThinkingBudgetRange{Min: lo, Max: hi}, + Deprecated: true, + }, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, }, } } @@ -70,7 +98,7 @@ func TestValidateParamsThinkingEffort(t *testing.T) { wantFallen: true, }, { - name: "mode mismatch falls back even for a plausible-looking value", + name: "effort against a budget-only model falls back even for a plausible-looking value", spec: continuousBudgetSpec(1024, 32000), effort: "high", wantEffort: "", @@ -78,7 +106,7 @@ func TestValidateParamsThinkingEffort(t *testing.T) { }, { name: "thinking unsupported at all falls back", - spec: &modelv1.ModelSpec{Thinking: &modelv1.ThinkingSpec{Supported: false, Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}}, + spec: &modelv1.ModelSpec{Thinking: &modelv1.ThinkingSpec{Supported: false}}, effort: "low", wantEffort: "", wantFallen: true, @@ -144,12 +172,13 @@ func TestValidateParamsThinkingBudget(t *testing.T) { } } -func TestValidateParamsThinkingBudgetModeMismatch(t *testing.T) { +func TestValidateParamsThinkingBudgetAgainstEffortOnlyModel(t *testing.T) { t.Parallel() // A numerically plausible budget still falls back when the resolved - // model's ThinkingSpec.mode isn't THINKING_MODE_CONTINUOUS_BUDGET — - // mirrors the effort-side "mode mismatch" case above. + // model declares no budget control — mirrors the effort-side case + // above. Each param is checked against the control that governs it, + // so declaring one control never implies the other. spec := discreteEffortSpec("low", "high") req := &modelv1.GenerationParams{ThinkingBudgetTokens: i64Ptr(5000)} @@ -163,14 +192,58 @@ func TestValidateParamsThinkingBudgetModeMismatch(t *testing.T) { } } +func TestValidateParamsBothControlsDeclaredKeepsBothParams(t *testing.T) { + t.Parallel() + + // The two axes are independent, so a model declaring both controls + // accepts both params in one request and neither falls back. Under the + // earlier single-mode ThinkingSpec this model was inexpressible, and + // whichever mode it picked would have silently dropped the other param. + spec := bothControlsSpec([]string{"low", "high"}, 1024, 32000) + req := &modelv1.GenerationParams{ + ThinkingEffort: strPtr("high"), + ThinkingBudgetTokens: i64Ptr(5000), + } + + got := ValidateParams(req, spec) + + if got.FellBackThinking { + t.Fatalf("FellBackThinking = true, want false") + } + if got.Resolved.GetThinkingEffort() != "high" { + t.Errorf("ThinkingEffort = %q, want %q", got.Resolved.GetThinkingEffort(), "high") + } + if got.Resolved.GetThinkingBudgetTokens() != 5000 { + t.Errorf("ThinkingBudgetTokens = %v, want 5000", got.Resolved.GetThinkingBudgetTokens()) + } +} + +func TestValidateParamsBudgetZeroAgainstNilControlFallsBack(t *testing.T) { + t.Parallel() + + // Regression guard: a nil BudgetControl's zero-valued range would + // accept a budget of exactly 0 if the nil check were skipped, which is + // the one value where "no control declared" and "in range" collide. + spec := &modelv1.ModelSpec{Thinking: &modelv1.ThinkingSpec{Supported: false}} + req := &modelv1.GenerationParams{ThinkingBudgetTokens: i64Ptr(0)} + + got := ValidateParams(req, spec) + + if !got.FellBackThinking { + t.Fatalf("FellBackThinking = false, want true") + } + if got.Resolved.ThinkingBudgetTokens != nil { + t.Fatalf("ThinkingBudgetTokens = %v, want cleared", got.Resolved.GetThinkingBudgetTokens()) + } +} + func TestValidateParamsThinkingBudgetMissingRange(t *testing.T) { t.Parallel() - // A ThinkingSpec claiming CONTINUOUS_BUDGET mode but omitting - // BudgetRange (a malformed capability declaration) must still fall - // back rather than panic. + // A BudgetControl declared with no Range (a malformed capability + // declaration) must still fall back rather than panic. spec := &modelv1.ModelSpec{ - Thinking: &modelv1.ThinkingSpec{Supported: true, Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET}, + Thinking: &modelv1.ThinkingSpec{Supported: true, Budget: &modelv1.BudgetControl{}}, } req := &modelv1.GenerationParams{ThinkingBudgetTokens: i64Ptr(5000)} got := ValidateParams(req, spec) diff --git a/pkg/model/capabilities.go b/pkg/model/capabilities.go index ddf6cc1..2d82895 100644 --- a/pkg/model/capabilities.go +++ b/pkg/model/capabilities.go @@ -2,6 +2,7 @@ package model import ( "fmt" + "slices" "time" commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" @@ -93,24 +94,79 @@ func validateModelSpec(m Spec) error { } // validateThinkingSpec checks t against -// docs/specifications/model/data-types.md#thinkingspec: effort_levels -// required for THINKING_MODE_DISCRETE_EFFORT, budget_range required for -// THINKING_MODE_CONTINUOUS_BUDGET, and default required whenever mode is -// not THINKING_MODE_NONE -// (docs/specifications/model/conformance.md's "ThinkingSpec.default MUST -// be set when mode != none" row). +// docs/specifications/model/data-types.md#thinkingspec. +// +// The axes are independent, so this validates each control that is present +// on its own terms rather than deriving requirements from one mode value. +// The one cross-axis rule is the unsupported case: a model that cannot +// reason at all MUST NOT declare a control for reasoning it does not do. func validateThinkingSpec(t ThinkingSpec) error { - if t.Mode == modelv1.ThinkingMode_THINKING_MODE_NONE { + if !t.Supported { + switch { + case t.Effort != nil: + return fmt.Errorf("%w: effort control declared on a model with thinking unsupported", ErrInvalidCapabilities) + case t.Budget != nil: + return fmt.Errorf("%w: budget control declared on a model with thinking unsupported", ErrInvalidCapabilities) + case t.AdaptiveByDefault: + return fmt.Errorf("%w: adaptive_by_default set on a model with thinking unsupported", ErrInvalidCapabilities) + // UNSPECIFIED and NEVER are both accepted here, and that is + // deliberate: the zero ThinkingSpec{} must be a valid declaration + // for "this model does not reason", which is the most common case + // by far and the one an author writes without thinking about it. + // Only a positive claim that reasoning CAN be disabled is a real + // contradiction worth rejecting. + case t.Disable == modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + t.Disable == modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL: + return fmt.Errorf("%w: disable claims reasoning can be turned off on a model with thinking unsupported", ErrInvalidCapabilities) + } return nil } - if t.Mode == modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT && len(t.EffortLevels) == 0 { - return fmt.Errorf("%w: effort_levels required for THINKING_MODE_DISCRETE_EFFORT", ErrInvalidCapabilities) + + if t.Disable == modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_UNSPECIFIED { + return fmt.Errorf("%w: disable required when thinking is supported", ErrInvalidCapabilities) + } + if err := validateEffortControl(t.Effort); err != nil { + return err + } + return validateBudgetControl(t.Budget) +} + +// validateEffortControl checks a declared effort ladder. A nil control is +// valid — it means the model has no effort ladder, which is a normal +// position, not an omission. +func validateEffortControl(e *EffortControl) error { + if e == nil { + return nil + } + if len(e.Levels) == 0 { + return fmt.Errorf("%w: effort control declared with no levels", ErrInvalidCapabilities) + } + if e.Default == "" { + return fmt.Errorf("%w: effort control declared with no default level", ErrInvalidCapabilities) + } + // The default must name a real level, or a kernel sending it back as an + // explicit override — the whole reason the field exists — would send a + // value the vendor rejects. + if !slices.Contains(e.Levels, e.Default) { + return fmt.Errorf("%w: effort default %q is not one of the declared levels %v", + ErrInvalidCapabilities, e.Default, e.Levels) + } + return nil +} + +// validateBudgetControl checks a declared token-budget control. A nil +// control is valid, for the same reason a nil effort control is. +func validateBudgetControl(b *BudgetControl) error { + if b == nil { + return nil } - if t.Mode == modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET && t.BudgetRange == nil { - return fmt.Errorf("%w: budget_range required for THINKING_MODE_CONTINUOUS_BUDGET", ErrInvalidCapabilities) + if b.Range.Min > b.Range.Max { + return fmt.Errorf("%w: budget range min %d exceeds max %d", + ErrInvalidCapabilities, b.Range.Min, b.Range.Max) } - if t.Default == "" { - return fmt.Errorf("%w: default required when mode is not THINKING_MODE_NONE", ErrInvalidCapabilities) + if b.Default != nil && (*b.Default < b.Range.Min || *b.Default > b.Range.Max) { + return fmt.Errorf("%w: budget default %d is outside the declared range [%d, %d]", + ErrInvalidCapabilities, *b.Default, b.Range.Min, b.Range.Max) } return nil } diff --git a/pkg/model/capabilities_test.go b/pkg/model/capabilities_test.go index 6abd166..b70183a 100644 --- a/pkg/model/capabilities_test.go +++ b/pkg/model/capabilities_test.go @@ -23,7 +23,7 @@ func validModelSpec() model.Spec { SupportsToolUse: true, SupportsVision: true, SupportsStreaming: true, - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + Thinking: model.ThinkingSpec{}, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, Pricing: model.Pricing{ Currency: "USD", @@ -34,6 +34,19 @@ func validModelSpec() model.Spec { } } +// thinkingWith returns a thinking-supported ThinkingSpec carrying the +// given controls, with the fields that are not under test set to values +// that satisfy their own invariants — so a failing case fails on the +// control it names, never on an unrelated omission. +func thinkingWith(effort *model.EffortControl, budget *model.BudgetControl) model.ThinkingSpec { + return model.ThinkingSpec{ + Supported: true, + Effort: effort, + Budget: budget, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } +} + func validConfigSchema(t *testing.T) *configv1.ConfigSchema { t.Helper() return &configv1.ConfigSchema{} @@ -83,46 +96,102 @@ func TestNewCapabilities_Invalid(t *testing.T) { wantErr: model.ErrInvalidCapabilities, }, { - name: "discrete effort without effort levels", + name: "effort control with no levels", models: func() []model.Spec { m := validModelSpec() - m.Thinking = model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT, - Default: "medium", - } + m.Thinking = thinkingWith(&model.EffortControl{Default: "medium"}, nil) return []model.Spec{m} }(), schema: &configv1.ConfigSchema{}, wantErr: model.ErrInvalidCapabilities, }, { - name: "continuous budget without budget range", + name: "effort control with no default", models: func() []model.Spec { m := validModelSpec() - m.Thinking = model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET, - Default: "1024", - } + m.Thinking = thinkingWith(&model.EffortControl{Levels: []string{"low", "high"}}, nil) + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, + { + // The default exists so a kernel can send it back as an explicit + // override; naming a level the vendor does not accept would make + // that override a guaranteed 400. + name: "effort default is not one of the declared levels", + models: func() []model.Spec { + m := validModelSpec() + m.Thinking = thinkingWith(&model.EffortControl{ + Levels: []string{"low", "high"}, + Default: "medium", + }, nil) + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, + { + name: "budget range inverted", + models: func() []model.Spec { + m := validModelSpec() + m.Thinking = thinkingWith(nil, &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 32000, Max: 1024}, + }) return []model.Spec{m} }(), schema: &configv1.ConfigSchema{}, wantErr: model.ErrInvalidCapabilities, }, { - name: "thinking mode set without default", + name: "budget default outside the declared range", + models: func() []model.Spec { + m := validModelSpec() + def := int64(64000) + m.Thinking = thinkingWith(nil, &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 1024, Max: 32000}, + Default: &def, + }) + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, + { + name: "thinking supported without a disable value", + models: func() []model.Spec { + m := validModelSpec() + m.Thinking = model.ThinkingSpec{Supported: true} + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, + { + // A model that cannot reason must not declare controls for + // reasoning it does not do — the one cross-axis rule. + name: "control declared on a model with thinking unsupported", models: func() []model.Spec { m := validModelSpec() m.Thinking = model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_ALWAYS_ON_ADAPTIVE, + Supported: false, + Effort: &model.EffortControl{Levels: []string{"low"}, Default: "low"}, } return []model.Spec{m} }(), schema: &configv1.ConfigSchema{}, wantErr: model.ErrInvalidCapabilities, }, + { + name: "adaptive_by_default set on a model with thinking unsupported", + models: func() []model.Spec { + m := validModelSpec() + m.Thinking = model.ThinkingSpec{Supported: false, AdaptiveByDefault: true} + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, { name: "pricing missing currency", models: func() []model.Spec { diff --git a/pkg/model/convert.go b/pkg/model/convert.go index cb6ba0e..fce704a 100644 --- a/pkg/model/convert.go +++ b/pkg/model/convert.go @@ -82,20 +82,33 @@ func modelSpecFromProto(in *modelv1.ModelSpec) Spec { // thinkingSpecToProto converts t into the generated wire type. func thinkingSpecToProto(t ThinkingSpec) *modelv1.ThinkingSpec { out := &modelv1.ThinkingSpec{ - Supported: t.Supported, - Mode: t.Mode, - EffortLevels: t.EffortLevels, - CanDisable: t.CanDisable, - } - if t.BudgetRange != nil { - out.BudgetRange = &modelv1.ThinkingBudgetRange{ - Min: t.BudgetRange.Min, - Max: t.BudgetRange.Max, + Supported: t.Supported, + AdaptiveByDefault: t.AdaptiveByDefault, + Disable: t.Disable, + } + if t.Effort != nil { + out.Effort = &modelv1.EffortControl{ + // Copied rather than aliased: a Spec's Levels slice is + // typically a package-level roster value shared across models, + // and handing it to the wire type would let a later mutation + // reach every model that shares it. + Levels: append([]string(nil), t.Effort.Levels...), + Default: t.Effort.Default, } } - if t.Default != "" { - def := t.Default - out.Default = &def + if t.Budget != nil { + budget := &modelv1.BudgetControl{ + Range: &modelv1.ThinkingBudgetRange{ + Min: t.Budget.Range.Min, + Max: t.Budget.Range.Max, + }, + Deprecated: t.Budget.Deprecated, + } + if t.Budget.Default != nil { + def := *t.Budget.Default + budget.Default = &def + } + out.Budget = budget } return out } @@ -106,14 +119,26 @@ func thinkingSpecFromProto(in *modelv1.ThinkingSpec) ThinkingSpec { return ThinkingSpec{} } out := ThinkingSpec{ - Supported: in.GetSupported(), - Mode: in.GetMode(), - EffortLevels: in.GetEffortLevels(), - CanDisable: in.GetCanDisable(), - Default: in.GetDefault(), - } - if br := in.GetBudgetRange(); br != nil { - out.BudgetRange = &ThinkingBudgetRange{Min: br.GetMin(), Max: br.GetMax()} + Supported: in.GetSupported(), + AdaptiveByDefault: in.GetAdaptiveByDefault(), + Disable: in.GetDisable(), + } + if e := in.GetEffort(); e != nil { + out.Effort = &EffortControl{ + Levels: append([]string(nil), e.GetLevels()...), + Default: e.GetDefault(), + } + } + if b := in.GetBudget(); b != nil { + budget := &BudgetControl{ + Range: ThinkingBudgetRange{Min: b.GetRange().GetMin(), Max: b.GetRange().GetMax()}, + Deprecated: b.GetDeprecated(), + } + if b.Default != nil { + def := b.GetDefault() + budget.Default = &def + } + out.Budget = budget } return out } diff --git a/pkg/model/convert_test.go b/pkg/model/convert_test.go index 802eeb0..837703a 100644 --- a/pkg/model/convert_test.go +++ b/pkg/model/convert_test.go @@ -28,11 +28,13 @@ func TestConvert_ModelSpecRoundTrip(t *testing.T) { SupportsStreaming: true, SupportsParallelToolCalls: true, Thinking: model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_DISCRETE_EFFORT, - EffortLevels: []string{"low", "medium", "high"}, - CanDisable: true, - Default: "medium", + Supported: true, + Effort: &model.EffortControl{ + Levels: []string{"low", "medium", "high"}, + Default: "medium", + }, + AdaptiveByDefault: true, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL, }, Caching: model.CachingSpec{ Supported: true, @@ -73,11 +75,23 @@ func TestConvert_ModelSpecRoundTrip(t *testing.T) { if !back.SupportsParallelToolCalls { t.Errorf("SupportsParallelToolCalls = false, want true") } - if back.Thinking.Default != "medium" { - t.Errorf("Thinking.Default = %q, want %q", back.Thinking.Default, "medium") + if back.Thinking.Effort == nil { + t.Fatal("Thinking.Effort = nil, want the declared effort control") + } + if back.Thinking.Effort.Default != "medium" { + t.Errorf("Thinking.Effort.Default = %q, want %q", back.Thinking.Effort.Default, "medium") + } + if len(back.Thinking.Effort.Levels) != 3 { + t.Errorf("len(Thinking.Effort.Levels) = %d, want 3", len(back.Thinking.Effort.Levels)) + } + // The two axes are independent, so a spec declaring an effort ladder + // AND adaptive-by-default must round-trip both — the exact pair the + // earlier single-mode shape could not carry. + if !back.Thinking.AdaptiveByDefault { + t.Error("Thinking.AdaptiveByDefault = false, want true") } - if len(back.Thinking.EffortLevels) != 3 { - t.Errorf("len(Thinking.EffortLevels) = %d, want 3", len(back.Thinking.EffortLevels)) + if back.Thinking.Disable != modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL { + t.Errorf("Thinking.Disable = %v, want CONDITIONAL", back.Thinking.Disable) } if !back.Caching.Supported || back.Caching.Mode != modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS { t.Errorf("Caching = %+v, want supported explicit_markers", back.Caching) @@ -115,32 +129,94 @@ func TestConvert_ModelSpecFromProtoNil(t *testing.T) { } } -func TestConvert_ThinkingSpecNoBudgetRange(t *testing.T) { +func TestConvert_ThinkingSpecBudgetControl(t *testing.T) { t.Parallel() - in := model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE} - wire := model.ThinkingSpecToProtoForTest(in) - if wire.GetBudgetRange() != nil { - t.Errorf("BudgetRange = %v, want nil", wire.GetBudgetRange()) + // A model with no thinking at all carries neither control. + wire := model.ThinkingSpecToProtoForTest(model.ThinkingSpec{}) + if wire.GetBudget() != nil { + t.Errorf("Budget = %v, want nil", wire.GetBudget()) + } + if wire.GetEffort() != nil { + t.Errorf("Effort = %v, want nil", wire.GetEffort()) } - back := model.ThinkingSpecFromProtoForTest(wire) - if back.BudgetRange != nil { - t.Errorf("round-tripped BudgetRange = %v, want nil", back.BudgetRange) + if back := model.ThinkingSpecFromProtoForTest(wire); back.Budget != nil || back.Effort != nil { + t.Errorf("round-tripped controls = (%v, %v), want both nil", back.Effort, back.Budget) } + def := int64(4096) inBudget := model.ThinkingSpec{ - Supported: true, - Mode: modelv1.ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET, - BudgetRange: &model.ThinkingBudgetRange{Min: 1024, Max: 32000}, - Default: "4096", + Supported: true, + Budget: &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 1024, Max: 32000}, + Default: &def, + Deprecated: true, + }, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, } wireBudget := model.ThinkingSpecToProtoForTest(inBudget) - if wireBudget.GetBudgetRange().GetMin() != 1024 || wireBudget.GetBudgetRange().GetMax() != 32000 { - t.Errorf("BudgetRange = %+v, want {1024 32000}", wireBudget.GetBudgetRange()) + if got := wireBudget.GetBudget().GetRange(); got.GetMin() != 1024 || got.GetMax() != 32000 { + t.Errorf("Budget.Range = %+v, want {1024 32000}", got) + } + if !wireBudget.GetBudget().GetDeprecated() { + t.Error("Budget.Deprecated = false, want true") } + backBudget := model.ThinkingSpecFromProtoForTest(wireBudget) - if backBudget.BudgetRange == nil || backBudget.BudgetRange.Min != 1024 || backBudget.BudgetRange.Max != 32000 { - t.Errorf("round-tripped BudgetRange = %+v, want {1024 32000}", backBudget.BudgetRange) + if backBudget.Budget == nil { + t.Fatal("round-tripped Budget = nil, want the declared control") + } + if backBudget.Budget.Range.Min != 1024 || backBudget.Budget.Range.Max != 32000 { + t.Errorf("round-tripped Budget.Range = %+v, want {1024 32000}", backBudget.Budget.Range) + } + if backBudget.Budget.Default == nil || *backBudget.Budget.Default != def { + t.Errorf("round-tripped Budget.Default = %v, want %d", backBudget.Budget.Default, def) + } + if !backBudget.Budget.Deprecated { + t.Error("round-tripped Budget.Deprecated = false, want true") + } +} + +func TestConvert_ThinkingSpecBudgetDefaultAbsentIsNotZero(t *testing.T) { + t.Parallel() + + // An omitted default means "the vendor reasons zero tokens by + // default", which is a different statement from "the vendor's default + // budget is the number 0" — collapsing the two would lose the + // distinction the pointer exists to carry. + in := model.ThinkingSpec{ + Supported: true, + Budget: &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 1024, Max: 32000}, + }, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + wire := model.ThinkingSpecToProtoForTest(in) + if wire.GetBudget().Default != nil { + t.Errorf("Budget.Default = %v, want nil", wire.GetBudget().Default) + } + if back := model.ThinkingSpecFromProtoForTest(wire); back.Budget.Default != nil { + t.Errorf("round-tripped Budget.Default = %v, want nil", back.Budget.Default) + } +} + +func TestConvert_ThinkingSpecEffortLevelsAreCopied(t *testing.T) { + t.Parallel() + + // A roster typically shares one levels slice across several models, so + // aliasing it into the wire type would let a mutation through one + // model's spec reach every other model that shares it. + levels := []string{"low", "high"} + in := model.ThinkingSpec{ + Supported: true, + Effort: &model.EffortControl{Levels: levels, Default: "low"}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + wire := model.ThinkingSpecToProtoForTest(in) + levels[0] = "mutated" + + if got := wire.GetEffort().GetLevels()[0]; got != "low" { + t.Errorf("wire levels[0] = %q after mutating the source slice, want %q", got, "low") } } @@ -236,7 +312,7 @@ func TestConvert_CapabilitiesRoundTrip(t *testing.T) { caps := &model.Capabilities{ Models: []model.Spec{{ ID: "claude-test", - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + Thinking: model.ThinkingSpec{}, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, Pricing: model.Pricing{Currency: "USD", Free: true}, }}, diff --git a/pkg/model/model.go b/pkg/model/model.go index fd69fff..f5a702c 100644 --- a/pkg/model/model.go +++ b/pkg/model/model.go @@ -131,8 +131,8 @@ type Spec struct { // accurately; false means the kernel serializes tool calls for this // model. SupportsParallelToolCalls bool - // Thinking is this model's extended-reasoning capability. Use - // ThinkingSpec{} (Mode left at THINKING_MODE_NONE) when unsupported. + // Thinking is this model's extended-reasoning capability. Use the zero + // ThinkingSpec{} when unsupported. Thinking ThinkingSpec // Caching is this model's prompt-caching capability. Use // CachingSpec{} (Mode left at CACHING_MODE_NONE) when unsupported. @@ -151,32 +151,66 @@ type Spec struct { // ThinkingSpec describes one model's extended-reasoning capability, per // docs/specifications/model/data-types.md#thinkingspec. +// +// These are independent axes, not one-of-N modes. A model may reason +// adaptively AND expose an effort ladder, or accept an effort level AND a +// deprecated token budget. Declare every control the model actually +// accepts; the kernel validates each requested parameter against the +// specific control that governs it. type ThinkingSpec struct { // Supported reports whether this model has any extended-reasoning - // capability at all. + // capability at all. When false, Effort and Budget MUST both be nil, + // AdaptiveByDefault MUST be false, and Disable MUST be + // THINKING_DISABLE_SUPPORT_NEVER. Supported bool - // Mode is which reasoning-control shape this model uses. MUST be - // THINKING_MODE_NONE when Supported is false. - Mode modelv1.ThinkingMode - // EffortLevels are the selectable effort levels, e.g. ["low","medium", - // "high","xhigh","max"]. MUST be non-empty when - // Mode == THINKING_MODE_DISCRETE_EFFORT. - EffortLevels []string - // BudgetRange is the selectable token-budget range. MUST be present - // when Mode == THINKING_MODE_CONTINUOUS_BUDGET. - BudgetRange *ThinkingBudgetRange - // CanDisable reports whether reasoning can be turned off once - // enabled. Some vendors' reasoning cannot be disabled. - CanDisable bool - // Default is the effort level (discrete_effort) or budget-token value - // (continuous_budget), as a string, the vendor applies when a request - // omits thinking config entirely. MUST be non-empty when - // Mode != THINKING_MODE_NONE. + // Effort is the named-effort-level control, non-nil iff this model + // accepts one. Nil means sending a thinking effort to this model is a + // kernel-level reject rather than something forwarded to the vendor. + Effort *EffortControl + // Budget is the explicit-token-budget control, non-nil iff this model + // accepts one. A model that never had one, and a model whose vendor + // removed it, both leave this nil. + Budget *BudgetControl + // AdaptiveByDefault reports whether omitting every thinking control + // still produces reasoning. False means an unconfigured request + // reasons zero tokens. + AdaptiveByDefault bool + // Disable reports whether, and when, reasoning can be turned off. MUST + // be set when Supported is true. + Disable modelv1.ThinkingDisableSupport +} + +// EffortControl declares that a model accepts a named reasoning-effort +// level, and which levels. +type EffortControl struct { + // Levels are the selectable effort levels, e.g. ["low","medium", + // "high","xhigh","max"]. MUST be non-empty — a model with no + // selectable levels leaves ThinkingSpec.Effort nil instead. + Levels []string + // Default is the level the vendor applies when a request omits effort + // entirely. MUST be non-empty and MUST appear in Levels. Default string } -// ThinkingBudgetRange bounds the token budget a caller may request when -// ThinkingSpec.Mode is THINKING_MODE_CONTINUOUS_BUDGET. +// BudgetControl declares that a model accepts an explicit reasoning-token +// budget, and its bounds. +type BudgetControl struct { + // Range is the accepted token-budget range, inclusive on both bounds. + Range ThinkingBudgetRange + // Default is the budget the vendor applies when a request omits one. + // Nil means the vendor reasons zero tokens by default — a pointer + // because a declared budget of 0 and an undeclared default are + // different statements. + Default *int64 + // Deprecated reports that the vendor still honors this control but + // steers callers to effort/adaptive instead, and may remove it in a + // later model. A vendor that has already removed it is declared by + // leaving ThinkingSpec.Budget nil, not by setting this. + Deprecated bool +} + +// ThinkingBudgetRange bounds the token budget a caller may request on a +// model declaring a BudgetControl. Both bounds are inclusive. type ThinkingBudgetRange struct { // Min is the smallest thinking-token budget this model accepts. Min int64 diff --git a/pkg/model/proto/v1/types.pb.go b/pkg/model/proto/v1/types.pb.go index 1a8907e..8374cba 100644 --- a/pkg/model/proto/v1/types.pb.go +++ b/pkg/model/proto/v1/types.pb.go @@ -25,72 +25,72 @@ const ( _ = protoimpl.EnforceVersion(protoimpl.MaxVersion - 20) ) -// ThinkingMode enumerates the shapes of extended-reasoning control found -// across researched vendors (model.md §2) — a plain supports_thinking -// bool would lose information the kernel needs to build a correct -// request, since some vendors (e.g. Anthropic) expose more than one mode -// across their own model lineup. -type ThinkingMode int32 +// ThinkingDisableSupport describes whether a model's reasoning can be +// turned off, per model/data-types.md#thinkingspec. A plain bool cannot +// express the real answer for every model: Anthropic's Opus 5 accepts an +// explicit disable at effort "high" or below and returns a 400 at "xhigh" +// or "max", so both true and false are wrong for it. +type ThinkingDisableSupport int32 const ( // Zero value. Never valid when ThinkingSpec.supported is true; its // presence on the wire means a caller forgot to set the field. - ThinkingMode_THINKING_MODE_UNSPECIFIED ThinkingMode = 0 - // The model has no extended-reasoning capability. Pairs with - // ThinkingSpec.supported == false. - ThinkingMode_THINKING_MODE_NONE ThinkingMode = 1 - // The model always reasons, adaptively, with no caller-selectable - // effort level or budget. - ThinkingMode_THINKING_MODE_ALWAYS_ON_ADAPTIVE ThinkingMode = 2 - // The caller selects one of a fixed set of named effort levels - // (ThinkingSpec.effort_levels). - ThinkingMode_THINKING_MODE_DISCRETE_EFFORT ThinkingMode = 3 - // The caller selects a token budget within ThinkingSpec.budget_range. - ThinkingMode_THINKING_MODE_CONTINUOUS_BUDGET ThinkingMode = 4 + ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_UNSPECIFIED ThinkingDisableSupport = 0 + // Reasoning cannot be turned off in any configuration — a researched + // Grok model defaults reasoning on with no off switch, and Anthropic's + // Fable 5 returns a 400 for an explicit disable. Also the correct value + // when ThinkingSpec.supported is false: there is nothing to disable. + ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER ThinkingDisableSupport = 1 + // An explicit disable is accepted in every configuration. + ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS ThinkingDisableSupport = 2 + // An explicit disable is accepted in some configurations and rejected in + // others. This protocol deliberately does not model WHICH: the condition + // is vendor-specific and expressing it would need a general constraint + // language. What this value buys the kernel is knowing that a failed + // disable attempt is a vendor policy response, not an adapter bug. + ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL ThinkingDisableSupport = 3 ) -// Enum value maps for ThinkingMode. +// Enum value maps for ThinkingDisableSupport. var ( - ThinkingMode_name = map[int32]string{ - 0: "THINKING_MODE_UNSPECIFIED", - 1: "THINKING_MODE_NONE", - 2: "THINKING_MODE_ALWAYS_ON_ADAPTIVE", - 3: "THINKING_MODE_DISCRETE_EFFORT", - 4: "THINKING_MODE_CONTINUOUS_BUDGET", - } - ThinkingMode_value = map[string]int32{ - "THINKING_MODE_UNSPECIFIED": 0, - "THINKING_MODE_NONE": 1, - "THINKING_MODE_ALWAYS_ON_ADAPTIVE": 2, - "THINKING_MODE_DISCRETE_EFFORT": 3, - "THINKING_MODE_CONTINUOUS_BUDGET": 4, + ThinkingDisableSupport_name = map[int32]string{ + 0: "THINKING_DISABLE_SUPPORT_UNSPECIFIED", + 1: "THINKING_DISABLE_SUPPORT_NEVER", + 2: "THINKING_DISABLE_SUPPORT_ALWAYS", + 3: "THINKING_DISABLE_SUPPORT_CONDITIONAL", + } + ThinkingDisableSupport_value = map[string]int32{ + "THINKING_DISABLE_SUPPORT_UNSPECIFIED": 0, + "THINKING_DISABLE_SUPPORT_NEVER": 1, + "THINKING_DISABLE_SUPPORT_ALWAYS": 2, + "THINKING_DISABLE_SUPPORT_CONDITIONAL": 3, } ) -func (x ThinkingMode) Enum() *ThinkingMode { - p := new(ThinkingMode) +func (x ThinkingDisableSupport) Enum() *ThinkingDisableSupport { + p := new(ThinkingDisableSupport) *p = x return p } -func (x ThinkingMode) String() string { +func (x ThinkingDisableSupport) String() string { return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) } -func (ThinkingMode) Descriptor() protoreflect.EnumDescriptor { +func (ThinkingDisableSupport) Descriptor() protoreflect.EnumDescriptor { return file_pluggableharness_model_v1_types_proto_enumTypes[0].Descriptor() } -func (ThinkingMode) Type() protoreflect.EnumType { +func (ThinkingDisableSupport) Type() protoreflect.EnumType { return &file_pluggableharness_model_v1_types_proto_enumTypes[0] } -func (x ThinkingMode) Number() protoreflect.EnumNumber { +func (x ThinkingDisableSupport) Number() protoreflect.EnumNumber { return protoreflect.EnumNumber(x) } -// Deprecated: Use ThinkingMode.Descriptor instead. -func (ThinkingMode) EnumDescriptor() ([]byte, []int) { +// Deprecated: Use ThinkingDisableSupport.Descriptor instead. +func (ThinkingDisableSupport) EnumDescriptor() ([]byte, []int) { return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{0} } @@ -485,8 +485,9 @@ func (x *ModelSpec) GetSupportsDocuments() bool { return false } -// ThinkingBudgetRange bounds the token budget a caller may request when -// ThinkingMode is THINKING_MODE_CONTINUOUS_BUDGET. +// ThinkingBudgetRange bounds the token budget a caller may request on a +// model whose ThinkingSpec declares a BudgetControl. Both bounds are +// inclusive. type ThinkingBudgetRange struct { state protoimpl.MessageState `protogen:"open.v1"` // The smallest thinking-token budget this model accepts. @@ -541,41 +542,179 @@ func (x *ThinkingBudgetRange) GetMax() int64 { return 0 } +// EffortControl declares that a model accepts a named reasoning-effort +// level, and which levels, per model/data-types.md#thinkingspec. +type EffortControl struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The selectable effort levels, e.g. ["low","medium","high","xhigh", + // "max"]. MUST be non-empty — a model with no selectable levels omits + // the whole EffortControl instead. + Levels []string `protobuf:"bytes,1,rep,name=levels,proto3" json:"levels,omitempty"` + // The level the vendor applies when a request omits effort entirely. + // MUST be set, and MUST be one of `levels` — makes the vendor's actual + // default behavior visible/auditable via GetCapabilities rather than + // hidden in adapter code, so a kernel wanting deterministic behavior can + // always send an explicit override. + Default string `protobuf:"bytes,2,opt,name=default,proto3" json:"default,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *EffortControl) Reset() { + *x = EffortControl{} + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *EffortControl) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*EffortControl) ProtoMessage() {} + +func (x *EffortControl) ProtoReflect() protoreflect.Message { + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[3] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use EffortControl.ProtoReflect.Descriptor instead. +func (*EffortControl) Descriptor() ([]byte, []int) { + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{3} +} + +func (x *EffortControl) GetLevels() []string { + if x != nil { + return x.Levels + } + return nil +} + +func (x *EffortControl) GetDefault() string { + if x != nil { + return x.Default + } + return "" +} + +// BudgetControl declares that a model accepts an explicit reasoning-token +// budget, and its bounds, per model/data-types.md#thinkingspec. +type BudgetControl struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The accepted token-budget range. MUST be present. + Range *ThinkingBudgetRange `protobuf:"bytes,1,opt,name=range,proto3" json:"range,omitempty"` + // The budget the vendor applies when a request omits one. MAY be + // omitted, which means the vendor reasons zero tokens by default. + Default *int64 `protobuf:"varint,2,opt,name=default,proto3,oneof" json:"default,omitempty"` + // Whether the vendor still honors this control but steers callers to + // effort/adaptive instead, and MAY remove it in a later model. This + // status is per-model, never per-vendor: Anthropic's Opus 4.6 and + // Sonnet 4.6 accept a deprecated budget, Haiku 4.5 accepts ONLY the + // budget, and Opus 4.7 onward reject it with a 400 (declared by omitting + // BudgetControl entirely, not by setting this flag). + Deprecated bool `protobuf:"varint,3,opt,name=deprecated,proto3" json:"deprecated,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *BudgetControl) Reset() { + *x = BudgetControl{} + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[4] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *BudgetControl) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*BudgetControl) ProtoMessage() {} + +func (x *BudgetControl) ProtoReflect() protoreflect.Message { + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[4] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use BudgetControl.ProtoReflect.Descriptor instead. +func (*BudgetControl) Descriptor() ([]byte, []int) { + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{4} +} + +func (x *BudgetControl) GetRange() *ThinkingBudgetRange { + if x != nil { + return x.Range + } + return nil +} + +func (x *BudgetControl) GetDefault() int64 { + if x != nil && x.Default != nil { + return *x.Default + } + return 0 +} + +func (x *BudgetControl) GetDeprecated() bool { + if x != nil { + return x.Deprecated + } + return false +} + // ThinkingSpec describes one model's extended-reasoning capability, per -// model.md §2. +// model/data-types.md#thinkingspec. +// +// These are independent axes, not one-of-N modes. A model MAY reason +// adaptively AND expose an effort ladder (Anthropic Opus 4.8, Sonnet 5), +// or accept an effort level AND a deprecated token budget (Opus 4.6, +// Sonnet 4.6). An earlier revision modeled this as a single mutually- +// exclusive enum, which forced every such model to declare a half-truth. type ThinkingSpec struct { state protoimpl.MessageState `protogen:"open.v1"` - // Whether this model has any extended-reasoning capability at all. + // Whether this model has any extended-reasoning capability at all. When + // false, effort and budget MUST both be absent and adaptive_by_default + // MUST be false; `disable` is meaningless (there is nothing to disable), + // so UNSPECIFIED and NEVER are equivalent and a reader MUST treat them + // identically. That equivalence is what keeps an all-zero ThinkingSpec a + // valid declaration for a model that does not reason — the common case. + // Only a positive claim that reasoning CAN be turned off (ALWAYS or + // CONDITIONAL) contradicts supported == false. Supported bool `protobuf:"varint,1,opt,name=supported,proto3" json:"supported,omitempty"` - // Which reasoning-control shape this model uses. MUST be - // THINKING_MODE_NONE when supported == false. - Mode ThinkingMode `protobuf:"varint,2,opt,name=mode,proto3,enum=pluggableharness.model.v1.ThinkingMode" json:"mode,omitempty"` - // The selectable effort levels, e.g. ["low","medium","high","xhigh", - // "max"]. MUST be non-empty when mode == THINKING_MODE_DISCRETE_EFFORT; - // meaningless otherwise. - EffortLevels []string `protobuf:"bytes,3,rep,name=effort_levels,json=effortLevels,proto3" json:"effort_levels,omitempty"` - // The selectable token-budget range. MUST be present when mode == - // THINKING_MODE_CONTINUOUS_BUDGET; meaningless otherwise. - BudgetRange *ThinkingBudgetRange `protobuf:"bytes,4,opt,name=budget_range,json=budgetRange,proto3,oneof" json:"budget_range,omitempty"` - // Whether reasoning can be turned off once enabled. MUST be set - // accurately — some vendors' reasoning cannot be disabled (e.g. a - // researched Grok model defaults reasoning on with no off switch). - CanDisable bool `protobuf:"varint,5,opt,name=can_disable,json=canDisable,proto3" json:"can_disable,omitempty"` - // The effort level (discrete_effort) or budget-token value - // (continuous_budget), as a string, the vendor applies when a request - // omits thinking config entirely. MUST be set when mode != - // THINKING_MODE_NONE — makes the vendor's actual default behavior - // visible/auditable via GetCapabilities rather than hidden in adapter - // code, so a kernel wanting deterministic behavior can always send an - // explicit override. - Default *string `protobuf:"bytes,6,opt,name=default,proto3,oneof" json:"default,omitempty"` + // The named-effort-level control, present iff this model accepts one. + // Absent means sending GenerationParams.thinking_effort to this model is + // a kernel-level reject, not something forwarded to the vendor. + Effort *EffortControl `protobuf:"bytes,7,opt,name=effort,proto3,oneof" json:"effort,omitempty"` + // The explicit-token-budget control, present iff this model accepts one. + // A model that never had one, and a model whose vendor removed it, + // both declare it absent. + Budget *BudgetControl `protobuf:"bytes,8,opt,name=budget,proto3,oneof" json:"budget,omitempty"` + // Whether omitting every thinking control still produces reasoning. + // False means an unconfigured request reasons zero tokens. + AdaptiveByDefault bool `protobuf:"varint,9,opt,name=adaptive_by_default,json=adaptiveByDefault,proto3" json:"adaptive_by_default,omitempty"` + // Whether, and when, reasoning can be turned off. MUST be set when + // supported is true. + Disable ThinkingDisableSupport `protobuf:"varint,10,opt,name=disable,proto3,enum=pluggableharness.model.v1.ThinkingDisableSupport" json:"disable,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *ThinkingSpec) Reset() { *x = ThinkingSpec{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[3] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -587,7 +726,7 @@ func (x *ThinkingSpec) String() string { func (*ThinkingSpec) ProtoMessage() {} func (x *ThinkingSpec) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[3] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[5] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -600,7 +739,7 @@ func (x *ThinkingSpec) ProtoReflect() protoreflect.Message { // Deprecated: Use ThinkingSpec.ProtoReflect.Descriptor instead. func (*ThinkingSpec) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{3} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{5} } func (x *ThinkingSpec) GetSupported() bool { @@ -610,39 +749,32 @@ func (x *ThinkingSpec) GetSupported() bool { return false } -func (x *ThinkingSpec) GetMode() ThinkingMode { +func (x *ThinkingSpec) GetEffort() *EffortControl { if x != nil { - return x.Mode - } - return ThinkingMode_THINKING_MODE_UNSPECIFIED -} - -func (x *ThinkingSpec) GetEffortLevels() []string { - if x != nil { - return x.EffortLevels + return x.Effort } return nil } -func (x *ThinkingSpec) GetBudgetRange() *ThinkingBudgetRange { +func (x *ThinkingSpec) GetBudget() *BudgetControl { if x != nil { - return x.BudgetRange + return x.Budget } return nil } -func (x *ThinkingSpec) GetCanDisable() bool { +func (x *ThinkingSpec) GetAdaptiveByDefault() bool { if x != nil { - return x.CanDisable + return x.AdaptiveByDefault } return false } -func (x *ThinkingSpec) GetDefault() string { - if x != nil && x.Default != nil { - return *x.Default +func (x *ThinkingSpec) GetDisable() ThinkingDisableSupport { + if x != nil { + return x.Disable } - return "" + return ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_UNSPECIFIED } // CachingSpec describes one model's prompt-caching capability, per @@ -668,7 +800,7 @@ type CachingSpec struct { func (x *CachingSpec) Reset() { *x = CachingSpec{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[4] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[6] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -680,7 +812,7 @@ func (x *CachingSpec) String() string { func (*CachingSpec) ProtoMessage() {} func (x *CachingSpec) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[4] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[6] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -693,7 +825,7 @@ func (x *CachingSpec) ProtoReflect() protoreflect.Message { // Deprecated: Use CachingSpec.ProtoReflect.Descriptor instead. func (*CachingSpec) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{4} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{6} } func (x *CachingSpec) GetSupported() bool { @@ -772,7 +904,7 @@ type PricingTier struct { func (x *PricingTier) Reset() { *x = PricingTier{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[5] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[7] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -784,7 +916,7 @@ func (x *PricingTier) String() string { func (*PricingTier) ProtoMessage() {} func (x *PricingTier) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[5] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[7] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -797,7 +929,7 @@ func (x *PricingTier) ProtoReflect() protoreflect.Message { // Deprecated: Use PricingTier.ProtoReflect.Descriptor instead. func (*PricingTier) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{5} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{7} } func (x *PricingTier) GetEffectiveFrom() *timestamppb.Timestamp { @@ -892,7 +1024,7 @@ type Pricing struct { func (x *Pricing) Reset() { *x = Pricing{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[6] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -904,7 +1036,7 @@ func (x *Pricing) String() string { func (*Pricing) ProtoMessage() {} func (x *Pricing) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[6] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[8] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -917,7 +1049,7 @@ func (x *Pricing) ProtoReflect() protoreflect.Message { // Deprecated: Use Pricing.ProtoReflect.Descriptor instead. func (*Pricing) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{6} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{8} } func (x *Pricing) GetCurrency() string { @@ -960,7 +1092,7 @@ type CacheBreakpoint struct { func (x *CacheBreakpoint) Reset() { *x = CacheBreakpoint{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[7] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[9] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -972,7 +1104,7 @@ func (x *CacheBreakpoint) String() string { func (*CacheBreakpoint) ProtoMessage() {} func (x *CacheBreakpoint) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[7] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[9] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -985,7 +1117,7 @@ func (x *CacheBreakpoint) ProtoReflect() protoreflect.Message { // Deprecated: Use CacheBreakpoint.ProtoReflect.Descriptor instead. func (*CacheBreakpoint) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{7} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{9} } func (x *CacheBreakpoint) GetPosition() isCacheBreakpoint_Position { @@ -1070,7 +1202,7 @@ type ToolDeclaration struct { func (x *ToolDeclaration) Reset() { *x = ToolDeclaration{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[8] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1082,7 +1214,7 @@ func (x *ToolDeclaration) String() string { func (*ToolDeclaration) ProtoMessage() {} func (x *ToolDeclaration) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[8] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[10] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1095,7 +1227,7 @@ func (x *ToolDeclaration) ProtoReflect() protoreflect.Message { // Deprecated: Use ToolDeclaration.ProtoReflect.Descriptor instead. func (*ToolDeclaration) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{8} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{10} } func (x *ToolDeclaration) GetName() string { @@ -1159,7 +1291,7 @@ type GenerationParams struct { func (x *GenerationParams) Reset() { *x = GenerationParams{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[9] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[11] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1171,7 +1303,7 @@ func (x *GenerationParams) String() string { func (*GenerationParams) ProtoMessage() {} func (x *GenerationParams) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[9] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[11] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1184,7 +1316,7 @@ func (x *GenerationParams) ProtoReflect() protoreflect.Message { // Deprecated: Use GenerationParams.ProtoReflect.Descriptor instead. func (*GenerationParams) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{9} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{11} } func (x *GenerationParams) GetThinkingEffort() string { @@ -1246,7 +1378,7 @@ type ToolChoice struct { func (x *ToolChoice) Reset() { *x = ToolChoice{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[10] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[12] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1258,7 +1390,7 @@ func (x *ToolChoice) String() string { func (*ToolChoice) ProtoMessage() {} func (x *ToolChoice) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[10] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[12] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1271,7 +1403,7 @@ func (x *ToolChoice) ProtoReflect() protoreflect.Message { // Deprecated: Use ToolChoice.ProtoReflect.Descriptor instead. func (*ToolChoice) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{10} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{12} } func (x *ToolChoice) GetMode() ToolChoiceMode { @@ -1321,7 +1453,7 @@ type Usage struct { func (x *Usage) Reset() { *x = Usage{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[11] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[13] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1333,7 +1465,7 @@ func (x *Usage) String() string { func (*Usage) ProtoMessage() {} func (x *Usage) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[11] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[13] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1346,7 +1478,7 @@ func (x *Usage) ProtoReflect() protoreflect.Message { // Deprecated: Use Usage.ProtoReflect.Descriptor instead. func (*Usage) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{11} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{13} } func (x *Usage) GetInputTokens() int64 { @@ -1409,7 +1541,7 @@ type ModelTarget struct { func (x *ModelTarget) Reset() { *x = ModelTarget{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[12] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1421,7 +1553,7 @@ func (x *ModelTarget) String() string { func (*ModelTarget) ProtoMessage() {} func (x *ModelTarget) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[12] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1434,7 +1566,7 @@ func (x *ModelTarget) ProtoReflect() protoreflect.Message { // Deprecated: Use ModelTarget.ProtoReflect.Descriptor instead. func (*ModelTarget) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{12} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{14} } func (x *ModelTarget) GetId() string { @@ -1475,7 +1607,7 @@ type ModelRef struct { func (x *ModelRef) Reset() { *x = ModelRef{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[13] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1487,7 +1619,7 @@ func (x *ModelRef) String() string { func (*ModelRef) ProtoMessage() {} func (x *ModelRef) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[13] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1500,7 +1632,7 @@ func (x *ModelRef) ProtoReflect() protoreflect.Message { // Deprecated: Use ModelRef.ProtoReflect.Descriptor instead. func (*ModelRef) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{13} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{15} } func (x *ModelRef) GetProvider() string { @@ -1527,7 +1659,7 @@ type CacheBreakpoint_AfterAssembledContext struct { func (x *CacheBreakpoint_AfterAssembledContext) Reset() { *x = CacheBreakpoint_AfterAssembledContext{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1539,7 +1671,7 @@ func (x *CacheBreakpoint_AfterAssembledContext) String() string { func (*CacheBreakpoint_AfterAssembledContext) ProtoMessage() {} func (x *CacheBreakpoint_AfterAssembledContext) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1552,7 +1684,7 @@ func (x *CacheBreakpoint_AfterAssembledContext) ProtoReflect() protoreflect.Mess // Deprecated: Use CacheBreakpoint_AfterAssembledContext.ProtoReflect.Descriptor instead. func (*CacheBreakpoint_AfterAssembledContext) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{7, 0} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{9, 0} } // AfterTools is an empty marker message: its presence as the set oneof @@ -1565,7 +1697,7 @@ type CacheBreakpoint_AfterTools struct { func (x *CacheBreakpoint_AfterTools) Reset() { *x = CacheBreakpoint_AfterTools{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1577,7 +1709,7 @@ func (x *CacheBreakpoint_AfterTools) String() string { func (*CacheBreakpoint_AfterTools) ProtoMessage() {} func (x *CacheBreakpoint_AfterTools) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1590,7 +1722,7 @@ func (x *CacheBreakpoint_AfterTools) ProtoReflect() protoreflect.Message { // Deprecated: Use CacheBreakpoint_AfterTools.ProtoReflect.Descriptor instead. func (*CacheBreakpoint_AfterTools) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{7, 1} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{9, 1} } var File_pluggableharness_model_v1_types_proto protoreflect.FileDescriptor @@ -1620,18 +1752,27 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "\x1d_supports_parallel_tool_calls\"9\n" + "\x13ThinkingBudgetRange\x12\x10\n" + "\x03min\x18\x01 \x01(\x03R\x03min\x12\x10\n" + - "\x03max\x18\x02 \x01(\x03R\x03max\"\xc3\x02\n" + - "\fThinkingSpec\x12\x1c\n" + - "\tsupported\x18\x01 \x01(\bR\tsupported\x12;\n" + - "\x04mode\x18\x02 \x01(\x0e2'.pluggableharness.model.v1.ThinkingModeR\x04mode\x12#\n" + - "\reffort_levels\x18\x03 \x03(\tR\feffortLevels\x12V\n" + - "\fbudget_range\x18\x04 \x01(\v2..pluggableharness.model.v1.ThinkingBudgetRangeH\x00R\vbudgetRange\x88\x01\x01\x12\x1f\n" + - "\vcan_disable\x18\x05 \x01(\bR\n" + - "canDisable\x12\x1d\n" + - "\adefault\x18\x06 \x01(\tH\x01R\adefault\x88\x01\x01B\x0f\n" + - "\r_budget_rangeB\n" + + "\x03max\x18\x02 \x01(\x03R\x03max\"A\n" + + "\rEffortControl\x12\x16\n" + + "\x06levels\x18\x01 \x03(\tR\x06levels\x12\x18\n" + + "\adefault\x18\x02 \x01(\tR\adefault\"\xa0\x01\n" + + "\rBudgetControl\x12D\n" + + "\x05range\x18\x01 \x01(\v2..pluggableharness.model.v1.ThinkingBudgetRangeR\x05range\x12\x1d\n" + + "\adefault\x18\x02 \x01(\x03H\x00R\adefault\x88\x01\x01\x12\x1e\n" + "\n" + - "\b_default\"\x98\x01\n" + + "deprecated\x18\x03 \x01(\bR\n" + + "deprecatedB\n" + + "\n" + + "\b_default\"\x8c\x03\n" + + "\fThinkingSpec\x12\x1c\n" + + "\tsupported\x18\x01 \x01(\bR\tsupported\x12E\n" + + "\x06effort\x18\a \x01(\v2(.pluggableharness.model.v1.EffortControlH\x00R\x06effort\x88\x01\x01\x12E\n" + + "\x06budget\x18\b \x01(\v2(.pluggableharness.model.v1.BudgetControlH\x01R\x06budget\x88\x01\x01\x12.\n" + + "\x13adaptive_by_default\x18\t \x01(\bR\x11adaptiveByDefault\x12K\n" + + "\adisable\x18\n" + + " \x01(\x0e21.pluggableharness.model.v1.ThinkingDisableSupportR\adisableB\t\n" + + "\a_effortB\t\n" + + "\a_budgetJ\x04\b\x02\x10\aR\fbudget_rangeR\vcan_disableR\adefaultR\reffort_levelsR\x04mode\"\x98\x01\n" + "\vCachingSpec\x12\x1c\n" + "\tsupported\x18\x01 \x01(\bR\tsupported\x12:\n" + "\x04mode\x18\x02 \x01(\x0e2&.pluggableharness.model.v1.CachingModeR\x04mode\x12/\n" + @@ -1708,13 +1849,12 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "\x11effective_ceiling\x18\x03 \x01(\x03R\x10effectiveCeiling\"6\n" + "\bModelRef\x12\x1a\n" + "\bprovider\x18\x01 \x01(\tR\bprovider\x12\x0e\n" + - "\x02id\x18\x02 \x01(\tR\x02id*\xb3\x01\n" + - "\fThinkingMode\x12\x1d\n" + - "\x19THINKING_MODE_UNSPECIFIED\x10\x00\x12\x16\n" + - "\x12THINKING_MODE_NONE\x10\x01\x12$\n" + - " THINKING_MODE_ALWAYS_ON_ADAPTIVE\x10\x02\x12!\n" + - "\x1dTHINKING_MODE_DISCRETE_EFFORT\x10\x03\x12#\n" + - "\x1fTHINKING_MODE_CONTINUOUS_BUDGET\x10\x04*\x8a\x01\n" + + "\x02id\x18\x02 \x01(\tR\x02id*\xb5\x01\n" + + "\x16ThinkingDisableSupport\x12(\n" + + "$THINKING_DISABLE_SUPPORT_UNSPECIFIED\x10\x00\x12\"\n" + + "\x1eTHINKING_DISABLE_SUPPORT_NEVER\x10\x01\x12#\n" + + "\x1fTHINKING_DISABLE_SUPPORT_ALWAYS\x10\x02\x12(\n" + + "$THINKING_DISABLE_SUPPORT_CONDITIONAL\x10\x03*\x8a\x01\n" + "\vCachingMode\x12\x1c\n" + "\x18CACHING_MODE_UNSPECIFIED\x10\x00\x12\x15\n" + "\x11CACHING_MODE_NONE\x10\x01\x12!\n" + @@ -1740,58 +1880,62 @@ func file_pluggableharness_model_v1_types_proto_rawDescGZIP() []byte { } var file_pluggableharness_model_v1_types_proto_enumTypes = make([]protoimpl.EnumInfo, 3) -var file_pluggableharness_model_v1_types_proto_msgTypes = make([]protoimpl.MessageInfo, 16) +var file_pluggableharness_model_v1_types_proto_msgTypes = make([]protoimpl.MessageInfo, 18) var file_pluggableharness_model_v1_types_proto_goTypes = []any{ - (ThinkingMode)(0), // 0: pluggableharness.model.v1.ThinkingMode + (ThinkingDisableSupport)(0), // 0: pluggableharness.model.v1.ThinkingDisableSupport (CachingMode)(0), // 1: pluggableharness.model.v1.CachingMode (ToolChoiceMode)(0), // 2: pluggableharness.model.v1.ToolChoiceMode (*Capabilities)(nil), // 3: pluggableharness.model.v1.Capabilities (*ModelSpec)(nil), // 4: pluggableharness.model.v1.ModelSpec (*ThinkingBudgetRange)(nil), // 5: pluggableharness.model.v1.ThinkingBudgetRange - (*ThinkingSpec)(nil), // 6: pluggableharness.model.v1.ThinkingSpec - (*CachingSpec)(nil), // 7: pluggableharness.model.v1.CachingSpec - (*PricingTier)(nil), // 8: pluggableharness.model.v1.PricingTier - (*Pricing)(nil), // 9: pluggableharness.model.v1.Pricing - (*CacheBreakpoint)(nil), // 10: pluggableharness.model.v1.CacheBreakpoint - (*ToolDeclaration)(nil), // 11: pluggableharness.model.v1.ToolDeclaration - (*GenerationParams)(nil), // 12: pluggableharness.model.v1.GenerationParams - (*ToolChoice)(nil), // 13: pluggableharness.model.v1.ToolChoice - (*Usage)(nil), // 14: pluggableharness.model.v1.Usage - (*ModelTarget)(nil), // 15: pluggableharness.model.v1.ModelTarget - (*ModelRef)(nil), // 16: pluggableharness.model.v1.ModelRef - (*CacheBreakpoint_AfterAssembledContext)(nil), // 17: pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext - (*CacheBreakpoint_AfterTools)(nil), // 18: pluggableharness.model.v1.CacheBreakpoint.AfterTools - (*v1.PromptExpansionSpec)(nil), // 19: pluggableharness.common.v1.PromptExpansionSpec - (*v11.ConfigSchema)(nil), // 20: pluggableharness.config.v1.ConfigSchema - (v1.HookPoint)(0), // 21: pluggableharness.common.v1.HookPoint - (*timestamppb.Timestamp)(nil), // 22: google.protobuf.Timestamp - (*v12.Schema)(nil), // 23: pluggableharness.schema.v1.Schema + (*EffortControl)(nil), // 6: pluggableharness.model.v1.EffortControl + (*BudgetControl)(nil), // 7: pluggableharness.model.v1.BudgetControl + (*ThinkingSpec)(nil), // 8: pluggableharness.model.v1.ThinkingSpec + (*CachingSpec)(nil), // 9: pluggableharness.model.v1.CachingSpec + (*PricingTier)(nil), // 10: pluggableharness.model.v1.PricingTier + (*Pricing)(nil), // 11: pluggableharness.model.v1.Pricing + (*CacheBreakpoint)(nil), // 12: pluggableharness.model.v1.CacheBreakpoint + (*ToolDeclaration)(nil), // 13: pluggableharness.model.v1.ToolDeclaration + (*GenerationParams)(nil), // 14: pluggableharness.model.v1.GenerationParams + (*ToolChoice)(nil), // 15: pluggableharness.model.v1.ToolChoice + (*Usage)(nil), // 16: pluggableharness.model.v1.Usage + (*ModelTarget)(nil), // 17: pluggableharness.model.v1.ModelTarget + (*ModelRef)(nil), // 18: pluggableharness.model.v1.ModelRef + (*CacheBreakpoint_AfterAssembledContext)(nil), // 19: pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext + (*CacheBreakpoint_AfterTools)(nil), // 20: pluggableharness.model.v1.CacheBreakpoint.AfterTools + (*v1.PromptExpansionSpec)(nil), // 21: pluggableharness.common.v1.PromptExpansionSpec + (*v11.ConfigSchema)(nil), // 22: pluggableharness.config.v1.ConfigSchema + (v1.HookPoint)(0), // 23: pluggableharness.common.v1.HookPoint + (*timestamppb.Timestamp)(nil), // 24: google.protobuf.Timestamp + (*v12.Schema)(nil), // 25: pluggableharness.schema.v1.Schema } var file_pluggableharness_model_v1_types_proto_depIdxs = []int32{ 4, // 0: pluggableharness.model.v1.Capabilities.models:type_name -> pluggableharness.model.v1.ModelSpec - 19, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec - 20, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema - 21, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint - 6, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec - 7, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec - 9, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing + 21, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec + 22, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema + 23, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint + 8, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec + 9, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec + 11, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing 2, // 7: pluggableharness.model.v1.ModelSpec.supported_tool_choice_modes:type_name -> pluggableharness.model.v1.ToolChoiceMode - 0, // 8: pluggableharness.model.v1.ThinkingSpec.mode:type_name -> pluggableharness.model.v1.ThinkingMode - 5, // 9: pluggableharness.model.v1.ThinkingSpec.budget_range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange - 1, // 10: pluggableharness.model.v1.CachingSpec.mode:type_name -> pluggableharness.model.v1.CachingMode - 22, // 11: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp - 22, // 12: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp - 8, // 13: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier - 17, // 14: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext - 18, // 15: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools - 23, // 16: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema - 13, // 17: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice - 2, // 18: pluggableharness.model.v1.ToolChoice.mode:type_name -> pluggableharness.model.v1.ToolChoiceMode - 19, // [19:19] is the sub-list for method output_type - 19, // [19:19] is the sub-list for method input_type - 19, // [19:19] is the sub-list for extension type_name - 19, // [19:19] is the sub-list for extension extendee - 0, // [0:19] is the sub-list for field type_name + 5, // 8: pluggableharness.model.v1.BudgetControl.range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange + 6, // 9: pluggableharness.model.v1.ThinkingSpec.effort:type_name -> pluggableharness.model.v1.EffortControl + 7, // 10: pluggableharness.model.v1.ThinkingSpec.budget:type_name -> pluggableharness.model.v1.BudgetControl + 0, // 11: pluggableharness.model.v1.ThinkingSpec.disable:type_name -> pluggableharness.model.v1.ThinkingDisableSupport + 1, // 12: pluggableharness.model.v1.CachingSpec.mode:type_name -> pluggableharness.model.v1.CachingMode + 24, // 13: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp + 24, // 14: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp + 10, // 15: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier + 19, // 16: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext + 20, // 17: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools + 25, // 18: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema + 15, // 19: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice + 2, // 20: pluggableharness.model.v1.ToolChoice.mode:type_name -> pluggableharness.model.v1.ToolChoiceMode + 21, // [21:21] is the sub-list for method output_type + 21, // [21:21] is the sub-list for method input_type + 21, // [21:21] is the sub-list for extension type_name + 21, // [21:21] is the sub-list for extension extendee + 0, // [0:21] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_types_proto_init() } @@ -1800,23 +1944,24 @@ func file_pluggableharness_model_v1_types_proto_init() { return } file_pluggableharness_model_v1_types_proto_msgTypes[1].OneofWrappers = []any{} - file_pluggableharness_model_v1_types_proto_msgTypes[3].OneofWrappers = []any{} + file_pluggableharness_model_v1_types_proto_msgTypes[4].OneofWrappers = []any{} file_pluggableharness_model_v1_types_proto_msgTypes[5].OneofWrappers = []any{} - file_pluggableharness_model_v1_types_proto_msgTypes[7].OneofWrappers = []any{ + file_pluggableharness_model_v1_types_proto_msgTypes[7].OneofWrappers = []any{} + file_pluggableharness_model_v1_types_proto_msgTypes[9].OneofWrappers = []any{ (*CacheBreakpoint_AfterAssembledContext_)(nil), (*CacheBreakpoint_AfterTools_)(nil), (*CacheBreakpoint_AfterMessageIndex)(nil), } - file_pluggableharness_model_v1_types_proto_msgTypes[9].OneofWrappers = []any{} - file_pluggableharness_model_v1_types_proto_msgTypes[10].OneofWrappers = []any{} file_pluggableharness_model_v1_types_proto_msgTypes[11].OneofWrappers = []any{} + file_pluggableharness_model_v1_types_proto_msgTypes[12].OneofWrappers = []any{} + file_pluggableharness_model_v1_types_proto_msgTypes[13].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_pluggableharness_model_v1_types_proto_rawDesc), len(file_pluggableharness_model_v1_types_proto_rawDesc)), NumEnums: 3, - NumMessages: 16, + NumMessages: 18, NumExtensions: 0, NumServices: 0, }, diff --git a/pkg/model/server_test.go b/pkg/model/server_test.go index 1e9476a..e36e416 100644 --- a/pkg/model/server_test.go +++ b/pkg/model/server_test.go @@ -40,7 +40,7 @@ func (f *fakeProvider) Capabilities(ctx context.Context) (*model.Capabilities, e } return model.NewCapabilities([]model.Spec{{ ID: "fake-model", - Thinking: model.ThinkingSpec{Mode: modelv1.ThinkingMode_THINKING_MODE_NONE}, + Thinking: model.ThinkingSpec{}, Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, Pricing: model.Pricing{Currency: "USD", Free: true}, }}, &configv1.ConfigSchema{}) From 0a79de9189ddb1f21cbd8f0bc811da30a157a0d2 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:08:50 -0400 Subject: [PATCH 03/16] model: make CachingSpec axes independent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CachingMode forced explicit_markers and implicit_automatic to be mutually exclusive, but Gemini 2.5 and later run implicit automatic caching by default AND offer explicit manual declaration concurrently, at a deeper discount. docs/first-party/providers/google.md had already escalated this: "the current CachingSpec shape has no way to declare 'this model supports two caching modes concurrently, with different discount rates', which is precisely the situation Google's own docs describe. This is worth flagging back to the protocol's designers as a real gap." The gap had teeth beyond declaration accuracy. cache_breakpoints were gated on the enum naming EXPLICIT_MARKERS, so such a model — declaring the mode that accurately described its default behavior — was thereby required to discard breakpoints it could in fact have honored, losing a cache discount it was eligible for with no error anywhere. Replace the enum with two independent bools and gate breakpoints on the explicit_markers axis alone. Declaring caching now requires naming at least one mechanism, since declaring neither would read as "no caching" to every caller. Also records the pricing gap this exposes, as an open question rather than a fix: PricingTier carries one cache-rate pair, but a model can bill cached tokens at several rates depending on which mechanism served the request — Anthropic's 5-minute vs 1-hour TTLs, Gemini's implicit vs explicit discounts. That deliberately did not become a provider_options knob, because the kernel computes and persists cost_usd, so a rate- changing value riding in a pass-through field would produce silently wrong ledger rows forever. Fixing it means making the rate a function of the mechanism used, which is a Pricing redesign rather than a field. --- api/pluggableharness/model/v1/types.proto | 58 +++-- docs/specifications/model/conformance.md | 9 +- docs/specifications/model/data-types.md | 22 +- internal/anthropic/catalog/catalog.go | 4 +- internal/anthropic/catalog/catalog_test.go | 4 +- internal/anthropic/messages/request.go | 2 +- internal/anthropic/messages/request_test.go | 8 +- internal/kernel/testdata/plugin/main.go | 11 +- internal/modelrequest/cache.go | 2 +- internal/modelrequest/cache_test.go | 39 +++- pkg/model/capabilities.go | 25 ++ pkg/model/capabilities_test.go | 29 ++- pkg/model/convert.go | 6 +- pkg/model/convert_test.go | 11 +- pkg/model/model.go | 22 +- pkg/model/proto/v1/types.pb.go | 241 ++++++++------------ pkg/model/server_test.go | 2 +- 17 files changed, 282 insertions(+), 213 deletions(-) diff --git a/api/pluggableharness/model/v1/types.proto b/api/pluggableharness/model/v1/types.proto index a0c5a59..a9cb706 100644 --- a/api/pluggableharness/model/v1/types.proto +++ b/api/pluggableharness/model/v1/types.proto @@ -221,32 +221,28 @@ message ThinkingSpec { ThinkingDisableSupport disable = 10; } -// CachingMode enumerates the prompt-caching mechanics found across -// researched vendors (model.md §2). -enum CachingMode { - // Zero value. Never valid when CachingSpec.supported is true; its - // presence on the wire means a caller forgot to set the field. - CACHING_MODE_UNSPECIFIED = 0; - // The model has no prompt-caching capability. Pairs with - // CachingSpec.supported == false. - CACHING_MODE_NONE = 1; - // The caller must place cache breakpoints on content blocks explicitly - // (Anthropic/Mistral-style). - CACHING_MODE_EXPLICIT_MARKERS = 2; - // The vendor applies caching transparently above a token threshold, no - // caller action required. - CACHING_MODE_IMPLICIT_AUTOMATIC = 3; -} - // CachingSpec describes one model's prompt-caching capability, per -// model.md §2. +// model/data-types.md#cachingspec. +// +// These are independent axes, not one-of-N modes, for the same reason +// ThinkingSpec's are: Google's Gemini 2.5 and later run implicit automatic +// caching by default AND offer explicit manual declaration concurrently at +// a deeper discount. An earlier revision modeled this as a single +// mutually-exclusive enum, which forced such a model to under-declare +// itself — and, because cache_breakpoints were gated on that enum naming +// EXPLICIT_MARKERS, required it to discard breakpoints it could in fact +// have honored. message CachingSpec { - // Whether this model has any prompt-caching capability at all. - bool supported = 1; + reserved 2; + + reserved "mode"; - // Which caching mechanic this model uses. MUST be CACHING_MODE_NONE - // when supported == false. - CachingMode mode = 2; + // Whether this model has any prompt-caching capability at all. When + // false, explicit_markers and implicit_automatic MUST both be false; + // when true, at least one of them MUST be true — a model caching by a + // mechanism this protocol cannot name is not declarable, and declaring + // neither reads as "no caching" to every caller. + bool supported = 1; // Whether this provider runs its own cache-keepalive loop (e.g. a // background goroutine re-pinging before a cache TTL expires, so a long @@ -254,8 +250,22 @@ message CachingSpec { // default false. Cache TTL mechanics are vendor-specific, so per // operator decision this is a provider-owned behavior the kernel never // drives — this field only tells the kernel/operator whether a given - // provider implements the optimization (model.md §2). + // provider implements the optimization. bool keepalive_supported = 3; + + // Whether the caller may place cache breakpoints on content blocks and + // have the adapter translate them into vendor-native markers (an + // Anthropic cache_control block, a Mistral prompt_cache_key). This is + // the axis StreamCompletionRequest.cache_breakpoints is gated on: an + // adapter for a model that does not declare it MUST ignore that field + // rather than error on it. + bool explicit_markers = 4; + + // Whether the vendor caches transparently above some token threshold + // with no caller action. Declaring this requires nothing of the kernel; + // it exists so cache-hit and cost behavior are explicable rather than + // surprising. + bool implicit_automatic = 5; } // PricingTier is one time-bounded, input-size-bounded rate within a diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index 3c07506..e75abda 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -34,8 +34,8 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | `ThinkingSpec.effort.default` / `budget.default` | MUST when that control is present | [`data-types.md#thinkingspec`](data-types.md#thinkingspec) — `effort.default` names a level; `budget.default` MAY be omitted, meaning zero reasoning tokens by default | | `ThinkingSpec.adaptive_by_default` / `disable` | MUST | [`data-types.md#thinkingspec`](data-types.md#thinkingspec) — `disable = conditional` tells the kernel a disable attempt MAY legitimately fail, so such a failure is vendor policy, not an adapter bug | | `StreamEvent.redacted_thinking` | MUST, for a vendor that emits vendor-encrypted reasoning blocks | [`data-types.md#streamevent`](data-types.md#streamevent) — a whole block, never fragmented; stored and echoed back verbatim or the vendor rejects the whole conversation on a later turn | -| Prompt caching | MAY, capability-gated via `CachingSpec` | declare `mode` (explicit vs. implicit) | -| Cache breakpoints (`StreamCompletionRequest.cache_breakpoints`) | MUST honor where `CachingSpec.mode = CACHING_MODE_EXPLICIT_MARKERS`; MUST ignore otherwise | [`protocol.md#cache-breakpoint-placement-policy`](protocol.md#cache-breakpoint-placement-policy) — placement is a kernel decision, never the plugin's | +| Prompt caching | MAY, capability-gated via `CachingSpec` | declare each axis it actually has; at least one MUST be true when `supported` | +| Cache breakpoints (`StreamCompletionRequest.cache_breakpoints`) | MUST honor where `CachingSpec.explicit_markers`; MUST ignore otherwise | [`protocol.md#cache-breakpoint-placement-policy`](protocol.md#cache-breakpoint-placement-policy) — placement is a kernel decision, never the plugin's. Gating on the axis rather than a mode is what lets a model declaring both caching axes still honor breakpoints | | `StreamCompletionRequest.provider_options` | MAY consume; kernel MUST pass through untouched | [`data-types.md#provider_options`](data-types.md#provider_options) — vendor knobs the kernel has no semantics for. A value the kernel reads MUST be a typed field instead, never smuggled through here | | Parallel tool calls in one turn | SHOULD declare via `supports_parallel_tool_calls` | kernel serializes calls if absent/false | | Tool-choice constraint (`GenerationParams.tool_choice`) | MAY, capability-gated via `ModelSpec.supported_tool_choice_modes` | kernel MUST NOT send a mode absent from the declared list, mirroring `ThinkingSpec` validation | @@ -53,6 +53,11 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded ## Open questions - Whether `supports_parallel_tool_calls` needs a per-request override (some vendors may allow disabling parallel calls per-call even when generally supported). +- **`PricingTier` assumes one cache rate per model, and real vendors have several.** The tier carries a single `cache_write_per_mtok`/`cache_read_per_mtok` pair, but a model can bill cached tokens at more than one rate depending on *which* caching mechanism serviced the request: Anthropic publishes distinct 5-minute and 1-hour cache-write rates (the 1-hour rate is 2× input against 1.25× for 5-minute), and a Gemini model running both caching axes discounts implicit hits ~75% and explicit hits ~90%. Two consequences follow, and both are currently unresolved rather than solved: + - There is no way to *select* a TTL. `CacheBreakpoint` carries no TTL field, so an adapter can only ever incur its vendor's default, and `internal/anthropic/catalog` correspondingly quotes only the 5-minute rate — quoting the 1-hour rate would overstate every cached turn by 60%. + - There is no way to *price* the non-default path even if one were selectable. + + This deliberately did not become a `provider_options` knob: the kernel computes and persists `cost_usd` from `Pricing`, so a TTL that changed the rate while riding in a pass-through field would silently produce wrong ledger rows forever — exactly the failure [`.claude/rules/determinism.md`](../../../.claude/rules/determinism.md) treats as a correctness bug. Fixing it properly means making the cache rate a function of the mechanism used, which is a `Pricing` redesign, not a field addition. - Whether a model needs to declare that it rejects `GenerationParams.temperature`. Several vendors' reasoning models reject non-default sampling parameters outright (Anthropic's effort-ladder models return a 400; other vendors' reasoning models ignore the value silently). There is no field for this, so an adapter facing such a model can only drop the operator's `temperature` on the floor — which is the right behavior, but it happens invisibly, and the kernel cannot tell a dropped parameter from an honored one. Today adapters infer it from the thinking shape, which is a proxy that will be wrong for the first model that has an effort ladder *and* accepts temperature. The same question applies to `top_p` and any other sampling parameter added later, so the fix is probably a general "declared sampling parameters" list rather than a per-parameter bool. - Retry/backoff policy specifics (exponential backoff parameters) — likely belongs in the kernel's routing logic rather than this protocol, but needs to be decided somewhere; see [`configuration/blocks-reference.md`](../configuration/blocks-reference.md)'s `settings{}` retry defaults for the current kernel-side values. - Whether `content_filtered` needs sub-categories (input filtered vs. output filtered) — there isn't enough vendor detail yet to decide. diff --git a/docs/specifications/model/data-types.md b/docs/specifications/model/data-types.md index 269dffe..9aa9b79 100644 --- a/docs/specifications/model/data-types.md +++ b/docs/specifications/model/data-types.md @@ -75,14 +75,22 @@ BudgetControl { ```protobuf CachingSpec { - supported bool - mode enum { none, explicit_markers, implicit_automatic } - // explicit_markers: caller must place cache breakpoints on content blocks (Anthropic/Mistral-style) - // implicit_automatic: vendor applies caching transparently above a token threshold, no caller action - keepalive_supported bool // MUST, default false — see the keepalive note below + supported bool // MUST — whether this model caches prompts at all + explicit_markers bool // MUST — the caller MAY place cache breakpoints on content blocks + implicit_automatic bool // MUST — the vendor caches transparently above a token threshold + keepalive_supported bool // MUST, default false — see the keepalive note below } ``` +Like [`ThinkingSpec`](#thinkingspec), these are independent axes rather than one-of-N modes, and for the same reason: a real model occupies both positions at once. Google's Gemini 2.5 and later run implicit automatic caching by default *and* offer explicit manual declaration concurrently, at a deeper discount. An enum could name only one, which forced an adapter either to under-declare the model or to route the second mechanism outside this protocol entirely. + +`supported == false` means the model does not cache: `explicit_markers` and `implicit_automatic` MUST both be false. `supported == true` requires at least one of them to be true — a model that caches by some mechanism the protocol cannot name is not declarable, and silently declaring neither would read as "no caching" to every caller. + +| Axis | Meaning when true | +|---|---| +| `explicit_markers` | The caller places cache breakpoints and the adapter translates them into vendor-native markers (Anthropic's `cache_control`, Mistral's `prompt_cache_key`). This is the axis [`cache_breakpoints`](#cache_breakpoints-and-cache-breakpoint-placement-policy) is gated on. | +| `implicit_automatic` | The vendor caches transparently above some token threshold with no caller action. Declaring this does not require the kernel to do anything; it exists so cost and cache-hit behavior are explicable rather than surprising. | + **Cache keepalive is a provider-owned behavior, not a kernel mechanism.** A dedicated keepalive daemon — re-pinging every 5 minutes so a long tool-execution gap doesn't let a prompt-cache TTL expire — is a real-world pattern (as used, for example, by Aider) and a real cost concern given `cache_read_per_mtok` (below) is typically far cheaper than `input_per_mtok`. This is deliberately **not** a kernel-loop responsibility: cache TTL mechanics are vendor-specific (5m/1h TTLs differ per vendor), and the adapter that already understands its own vendor's `CachingSpec.mode` is the natural owner of keeping that cache warm — not a kernel that would need to learn every vendor's TTL semantics to orchestrate a generic loop. A model provider MAY implement its own internal keepalive (e.g. a background goroutine within the plugin subprocess watching elapsed time since the last real call) and declares this via `keepalive_supported` so the kernel/operator can tell whether a given provider implements the optimization, without the kernel ever driving the loop itself. ## `Pricing` @@ -245,7 +253,9 @@ CacheBreakpoint { } ``` -`cache_breakpoints` is meaningful only when the target model's `CachingSpec.mode == CACHING_MODE_EXPLICIT_MARKERS`; a model-provider adapter targeting a model whose `CachingSpec.mode` is `CACHING_MODE_IMPLICIT_AUTOMATIC` or `CACHING_MODE_NONE` MUST ignore this field rather than error on it. The adapter maps each breakpoint to its vendor's own cache-control mechanism (e.g. an Anthropic `cache_control` block on the targeted content). +`cache_breakpoints` is meaningful only when the target model declares `CachingSpec.explicit_markers`; an adapter targeting a model that does not MUST ignore this field rather than error on it. The adapter maps each breakpoint to its vendor's own cache-control mechanism (e.g. an Anthropic `cache_control` block on the targeted content). + +Gating on `explicit_markers` alone — rather than on a mode that could name only one mechanism — is what lets a model declaring *both* caching axes still honor breakpoints. Under the earlier enum, a model like Gemini 2.5 that declared implicit caching (the accurate description of its default behavior) was thereby required to discard breakpoints it could in fact have honored through its explicit pathway. **Breakpoint placement is a kernel decision, not the plugin's.** The kernel knows each `assembled_context` section's `Stability` (`content/v1/types.proto`'s `Stability` enum: `STABILITY_STATIC` vs. `STABILITY_DYNAMIC`) and each message's position in the conversation, so it places breakpoints at natural stable-prefix boundaries — the same tools → system → static-project-context → conversation-tail ordering that governs `assembled_context`'s own chain order. In practice this means: a breakpoint after `after_tools` when the tool declaration list is stable turn to turn, and a breakpoint after `after_assembled_context` when the whole chain's leading sections are `STABILITY_STATIC` — since that's usually the longest stable prefix a vendor's prompt cache can actually reuse. A plugin never invents its own placement; it only translates the breakpoints the kernel already decided into vendor-native markers. diff --git a/internal/anthropic/catalog/catalog.go b/internal/anthropic/catalog/catalog.go index 96c8d07..848238a 100644 --- a/internal/anthropic/catalog/catalog.go +++ b/internal/anthropic/catalog/catalog.go @@ -99,8 +99,8 @@ func base(id string, contextWindow, maxOutput int64) model.Spec { SupportsParallelToolCalls: true, SupportsDocuments: true, Caching: model.CachingSpec{ - Supported: true, - Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS, + Supported: true, + ExplicitMarkers: true, // The plugin runs no background cache-keepalive loop. A // keepalive would mean issuing extra billed requests on the // operator's behalf without them asking, which is not a diff --git a/internal/anthropic/catalog/catalog_test.go b/internal/anthropic/catalog/catalog_test.go index 894e773..8d3cf68 100644 --- a/internal/anthropic/catalog/catalog_test.go +++ b/internal/anthropic/catalog/catalog_test.go @@ -247,8 +247,8 @@ func TestCaching_everyModelDeclaresExplicitMarkers(t *testing.T) { if !m.Caching.Supported { t.Errorf("%s: caching is not declared supported", m.ID) } - if m.Caching.Mode != modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS { - t.Errorf("%s: caching mode = %v, want CACHING_MODE_EXPLICIT_MARKERS", m.ID, m.Caching.Mode) + if !m.Caching.ExplicitMarkers { + t.Errorf("%s: ExplicitMarkers = false, want true", m.ID) } if m.Caching.KeepaliveSupported { t.Errorf("%s: this plugin runs no keepalive loop, so the flag must be false", m.ID) diff --git a/internal/anthropic/messages/request.go b/internal/anthropic/messages/request.go index 1611181..4a8d27e 100644 --- a/internal/anthropic/messages/request.go +++ b/internal/anthropic/messages/request.go @@ -267,7 +267,7 @@ func buildThinking(params *modelv1.GenerationParams, spec model.Spec) (*Thinking // breakpoints beforehand is what keeps a breakpoint's placement correct // regardless of how the messages are later merged. func applyCacheBreakpoints(breakpoints []*modelv1.CacheBreakpoint, spec model.Spec, system []TextBlock, tools []Tool, origMessages []Message) error { - if spec.Caching.Mode != modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS { + if !spec.Caching.ExplicitMarkers { // MUST ignore per StreamCompletionRequest.cache_breakpoints: this // field is meaningful only under explicit-marker caching, and // placement is a kernel decision this adapter only executes. diff --git a/internal/anthropic/messages/request_test.go b/internal/anthropic/messages/request_test.go index ccbc706..1e6fb54 100644 --- a/internal/anthropic/messages/request_test.go +++ b/internal/anthropic/messages/request_test.go @@ -36,8 +36,8 @@ func fullSpec() model.Spec { Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_NEVER, }, Caching: model.CachingSpec{ - Supported: true, - Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS, + Supported: true, + ExplicitMarkers: true, }, } } @@ -61,7 +61,7 @@ func budgetSpec(canDisable bool) model.Spec { }, Disable: disable, }, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, } } @@ -73,7 +73,7 @@ func minimalSpec() model.Spec { ID: "claude-minimal", MaxOutputTokens: 2048, Thinking: model.ThinkingSpec{}, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, } } diff --git a/internal/kernel/testdata/plugin/main.go b/internal/kernel/testdata/plugin/main.go index a5ba4cc..a96c7da 100644 --- a/internal/kernel/testdata/plugin/main.go +++ b/internal/kernel/testdata/plugin/main.go @@ -80,14 +80,11 @@ func (*fixtureProvider) Capabilities(context.Context) (*model.Capabilities, erro MaxOutputTokens: 8192, SupportsToolUse: true, SupportsStreaming: true, - // A model with no thinking capability leaves ThinkingSpec at its - // zero value: Supported false, no effort or budget control, and - // disable NEVER — there is nothing to disable. CachingSpec must - // still name an explicit NONE mode, whose zero value is - // CACHING_MODE_UNSPECIFIED and which NewCapabilities rejects rather - // than guessing at. + // A model with neither capability leaves both specs at their zero + // value: no thinking (no controls, nothing to disable) and no + // caching (neither mechanism declared). Thinking: model.ThinkingSpec{}, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, Pricing: model.Pricing{Currency: "USD", Free: true}, SupportedToolChoiceModes: []modelv1.ToolChoiceMode{ modelv1.ToolChoiceMode_TOOL_CHOICE_MODE_AUTO, diff --git a/internal/modelrequest/cache.go b/internal/modelrequest/cache.go index 046ba54..a7a83d9 100644 --- a/internal/modelrequest/cache.go +++ b/internal/modelrequest/cache.go @@ -63,7 +63,7 @@ type CacheBreakpoint = modelv1.CacheBreakpoint // Stability, which messages carries no equivalent of (content.v1.Message // has no Stability field), so it is currently unused. func PlaceCacheBreakpoints(sections []*contentv1.ContextSection, messages []*contentv1.Message, spec *modelv1.ModelSpec) []*CacheBreakpoint { //nolint:revive // messages reserved for future message-position-aware placement, see doc comment above - if spec.GetCaching().GetMode() != modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS { + if !spec.GetCaching().GetExplicitMarkers() { return nil } diff --git a/internal/modelrequest/cache_test.go b/internal/modelrequest/cache_test.go index 7c2932c..bf75cb0 100644 --- a/internal/modelrequest/cache_test.go +++ b/internal/modelrequest/cache_test.go @@ -8,7 +8,7 @@ import ( ) func explicitMarkersSpec() *modelv1.ModelSpec { - return &modelv1.ModelSpec{Caching: &modelv1.CachingSpec{Supported: true, Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS}} + return &modelv1.ModelSpec{Caching: &modelv1.CachingSpec{Supported: true, ExplicitMarkers: true}} } func staticSection() *contentv1.ContextSection { @@ -48,29 +48,48 @@ func TestPlaceCacheBreakpointsWorkedExample(t *testing.T) { wantAfterAssembledContext(t, got) } -func TestPlaceCacheBreakpointsNonExplicitMarkersModes(t *testing.T) { +func TestPlaceCacheBreakpointsWithoutExplicitMarkers(t *testing.T) { t.Parallel() sections := []*contentv1.ContextSection{staticSection()} - modes := []modelv1.CachingMode{ - modelv1.CachingMode_CACHING_MODE_NONE, - modelv1.CachingMode_CACHING_MODE_IMPLICIT_AUTOMATIC, - modelv1.CachingMode_CACHING_MODE_UNSPECIFIED, + specs := map[string]*modelv1.CachingSpec{ + "no caching at all": {}, + "implicit only": {Supported: true, ImplicitAutomatic: true}, + "supported but bare": {Supported: true}, } - for _, mode := range modes { - t.Run(mode.String(), func(t *testing.T) { + for name, caching := range specs { + t.Run(name, func(t *testing.T) { t.Parallel() - spec := &modelv1.ModelSpec{Caching: &modelv1.CachingSpec{Mode: mode}} + spec := &modelv1.ModelSpec{Caching: caching} got := PlaceCacheBreakpoints(sections, nil, spec) if got != nil { - t.Fatalf("got %+v, want nil for caching mode %v", got, mode) + t.Fatalf("got %+v, want nil when explicit_markers is false", got) } }) } } +func TestPlaceCacheBreakpointsBothCachingAxesStillPlacesBreakpoints(t *testing.T) { + t.Parallel() + + // A model running implicit caching by default AND accepting explicit + // markers must still get breakpoints. Under the earlier single-mode + // enum this model could only declare IMPLICIT_AUTOMATIC, which gated + // breakpoints off entirely and silently discarded a cache discount it + // was eligible for. + sections := []*contentv1.ContextSection{staticSection()} + spec := &modelv1.ModelSpec{Caching: &modelv1.CachingSpec{ + Supported: true, + ExplicitMarkers: true, + ImplicitAutomatic: true, + }} + + got := PlaceCacheBreakpoints(sections, nil, spec) + wantAfterAssembledContext(t, got) +} + func TestPlaceCacheBreakpointsNoStaticLeadingSection(t *testing.T) { t.Parallel() diff --git a/pkg/model/capabilities.go b/pkg/model/capabilities.go index 2d82895..4d9b868 100644 --- a/pkg/model/capabilities.go +++ b/pkg/model/capabilities.go @@ -87,12 +87,37 @@ func validateModelSpec(m Spec) error { if err := validateThinkingSpec(m.Thinking); err != nil { return err } + if err := validateCachingSpec(m.Caching); err != nil { + return err + } if err := validatePricing(m.Pricing, m.Caching.Supported); err != nil { return err } return nil } +// validateCachingSpec checks c against +// docs/specifications/model/data-types.md#cachingspec. +// +// The two axes are independent, so a model may declare either or both — +// the only rules are that declaring caching requires naming at least one +// mechanism, and that a non-caching model names none. A model caching by +// some mechanism this protocol cannot express is not declarable, and +// leaving both axes false would read as "no caching" to every caller, +// which is why the positive case is checked rather than assumed. +func validateCachingSpec(c CachingSpec) error { + if !c.Supported { + if c.ExplicitMarkers || c.ImplicitAutomatic { + return fmt.Errorf("%w: caching mechanism declared on a model with caching unsupported", ErrInvalidCapabilities) + } + return nil + } + if !c.ExplicitMarkers && !c.ImplicitAutomatic { + return fmt.Errorf("%w: caching supported but neither explicit_markers nor implicit_automatic declared", ErrInvalidCapabilities) + } + return nil +} + // validateThinkingSpec checks t against // docs/specifications/model/data-types.md#thinkingspec. // diff --git a/pkg/model/capabilities_test.go b/pkg/model/capabilities_test.go index b70183a..647ca78 100644 --- a/pkg/model/capabilities_test.go +++ b/pkg/model/capabilities_test.go @@ -24,7 +24,7 @@ func validModelSpec() model.Spec { SupportsVision: true, SupportsStreaming: true, Thinking: model.ThinkingSpec{}, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, Pricing: model.Pricing{ Currency: "USD", Tiers: []model.PricingTier{ @@ -192,6 +192,29 @@ func TestNewCapabilities_Invalid(t *testing.T) { schema: &configv1.ConfigSchema{}, wantErr: model.ErrInvalidCapabilities, }, + { + // Declaring caching without naming a mechanism would read as + // "no caching" to every caller, so it is rejected rather than + // silently degraded. + name: "caching supported but neither mechanism declared", + models: func() []model.Spec { + m := validModelSpec() + m.Caching = model.CachingSpec{Supported: true} + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, + { + name: "caching mechanism declared on a model with caching unsupported", + models: func() []model.Spec { + m := validModelSpec() + m.Caching = model.CachingSpec{Supported: false, ImplicitAutomatic: true} + return []model.Spec{m} + }(), + schema: &configv1.ConfigSchema{}, + wantErr: model.ErrInvalidCapabilities, + }, { name: "pricing missing currency", models: func() []model.Spec { @@ -216,7 +239,7 @@ func TestNewCapabilities_Invalid(t *testing.T) { name: "caching supported but tier missing cache pricing", models: func() []model.Spec { m := validModelSpec() - m.Caching = model.CachingSpec{Supported: true, Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS} + m.Caching = model.CachingSpec{Supported: true, ExplicitMarkers: true} return []model.Spec{m} }(), schema: &configv1.ConfigSchema{}, @@ -253,7 +276,7 @@ func TestNewCapabilities_CachingSatisfiedTiersAreValid(t *testing.T) { t.Parallel() m := validModelSpec() - m.Caching = model.CachingSpec{Supported: true, Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS} + m.Caching = model.CachingSpec{Supported: true, ExplicitMarkers: true} m.Pricing.Tiers = []model.PricingTier{ {InputPerMtok: 3, OutputPerMtok: 15, CacheWritePerMtok: mustFloat64(3.75), CacheReadPerMtok: mustFloat64(0.3)}, } diff --git a/pkg/model/convert.go b/pkg/model/convert.go index fce704a..85f9f36 100644 --- a/pkg/model/convert.go +++ b/pkg/model/convert.go @@ -147,7 +147,8 @@ func thinkingSpecFromProto(in *modelv1.ThinkingSpec) ThinkingSpec { func cachingSpecToProto(c CachingSpec) *modelv1.CachingSpec { return &modelv1.CachingSpec{ Supported: c.Supported, - Mode: c.Mode, + ExplicitMarkers: c.ExplicitMarkers, + ImplicitAutomatic: c.ImplicitAutomatic, KeepaliveSupported: c.KeepaliveSupported, } } @@ -159,7 +160,8 @@ func cachingSpecFromProto(in *modelv1.CachingSpec) CachingSpec { } return CachingSpec{ Supported: in.GetSupported(), - Mode: in.GetMode(), + ExplicitMarkers: in.GetExplicitMarkers(), + ImplicitAutomatic: in.GetImplicitAutomatic(), KeepaliveSupported: in.GetKeepaliveSupported(), } } diff --git a/pkg/model/convert_test.go b/pkg/model/convert_test.go index 837703a..f31a2c8 100644 --- a/pkg/model/convert_test.go +++ b/pkg/model/convert_test.go @@ -38,7 +38,8 @@ func TestConvert_ModelSpecRoundTrip(t *testing.T) { }, Caching: model.CachingSpec{ Supported: true, - Mode: modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS, + ExplicitMarkers: true, + ImplicitAutomatic: true, KeepaliveSupported: true, }, Pricing: model.Pricing{ @@ -93,8 +94,10 @@ func TestConvert_ModelSpecRoundTrip(t *testing.T) { if back.Thinking.Disable != modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL { t.Errorf("Thinking.Disable = %v, want CONDITIONAL", back.Thinking.Disable) } - if !back.Caching.Supported || back.Caching.Mode != modelv1.CachingMode_CACHING_MODE_EXPLICIT_MARKERS { - t.Errorf("Caching = %+v, want supported explicit_markers", back.Caching) + // A model declaring BOTH caching axes must round-trip both — the pair + // the earlier single-mode enum could not carry. + if !back.Caching.Supported || !back.Caching.ExplicitMarkers || !back.Caching.ImplicitAutomatic { + t.Errorf("Caching = %+v, want supported with both axes true", back.Caching) } if len(back.Pricing.Tiers) != 1 { t.Fatalf("len(Pricing.Tiers) = %d, want 1", len(back.Pricing.Tiers)) @@ -313,7 +316,7 @@ func TestConvert_CapabilitiesRoundTrip(t *testing.T) { Models: []model.Spec{{ ID: "claude-test", Thinking: model.ThinkingSpec{}, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, Pricing: model.Pricing{Currency: "USD", Free: true}, }}, ConfigSchema: &configv1.ConfigSchema{}, diff --git a/pkg/model/model.go b/pkg/model/model.go index f5a702c..3980238 100644 --- a/pkg/model/model.go +++ b/pkg/model/model.go @@ -220,13 +220,27 @@ type ThinkingBudgetRange struct { // CachingSpec describes one model's prompt-caching capability, per // docs/specifications/model/data-types.md#cachingspec. +// +// ExplicitMarkers and ImplicitAutomatic are independent axes, not +// alternatives: a model may run automatic caching by default and still +// accept explicit breakpoints at a deeper discount. Declare both when +// both are true. type CachingSpec struct { // Supported reports whether this model has any prompt-caching - // capability at all. + // capability at all. When false, both axes below MUST be false; when + // true, at least one MUST be true. Supported bool - // Mode is which caching mechanic this model uses. MUST be - // CACHING_MODE_NONE when Supported is false. - Mode modelv1.CachingMode + // ExplicitMarkers reports whether the caller may place cache + // breakpoints that this adapter translates into vendor-native markers. + // This is the axis StreamCompletionRequest.cache_breakpoints is gated + // on — an adapter for a model without it MUST ignore that field rather + // than error on it. + ExplicitMarkers bool + // ImplicitAutomatic reports whether the vendor caches transparently + // above some token threshold with no caller action. Declaring it + // requires nothing of the kernel; it exists so cache-hit and cost + // behavior are explicable. + ImplicitAutomatic bool // KeepaliveSupported reports whether this provider runs its own // cache-keepalive loop. MUST be set (default false); cache TTL // mechanics are vendor-specific and provider-owned, never a kernel diff --git a/pkg/model/proto/v1/types.pb.go b/pkg/model/proto/v1/types.pb.go index 8374cba..16fbf3e 100644 --- a/pkg/model/proto/v1/types.pb.go +++ b/pkg/model/proto/v1/types.pb.go @@ -94,68 +94,6 @@ func (ThinkingDisableSupport) EnumDescriptor() ([]byte, []int) { return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{0} } -// CachingMode enumerates the prompt-caching mechanics found across -// researched vendors (model.md §2). -type CachingMode int32 - -const ( - // Zero value. Never valid when CachingSpec.supported is true; its - // presence on the wire means a caller forgot to set the field. - CachingMode_CACHING_MODE_UNSPECIFIED CachingMode = 0 - // The model has no prompt-caching capability. Pairs with - // CachingSpec.supported == false. - CachingMode_CACHING_MODE_NONE CachingMode = 1 - // The caller must place cache breakpoints on content blocks explicitly - // (Anthropic/Mistral-style). - CachingMode_CACHING_MODE_EXPLICIT_MARKERS CachingMode = 2 - // The vendor applies caching transparently above a token threshold, no - // caller action required. - CachingMode_CACHING_MODE_IMPLICIT_AUTOMATIC CachingMode = 3 -) - -// Enum value maps for CachingMode. -var ( - CachingMode_name = map[int32]string{ - 0: "CACHING_MODE_UNSPECIFIED", - 1: "CACHING_MODE_NONE", - 2: "CACHING_MODE_EXPLICIT_MARKERS", - 3: "CACHING_MODE_IMPLICIT_AUTOMATIC", - } - CachingMode_value = map[string]int32{ - "CACHING_MODE_UNSPECIFIED": 0, - "CACHING_MODE_NONE": 1, - "CACHING_MODE_EXPLICIT_MARKERS": 2, - "CACHING_MODE_IMPLICIT_AUTOMATIC": 3, - } -) - -func (x CachingMode) Enum() *CachingMode { - p := new(CachingMode) - *p = x - return p -} - -func (x CachingMode) String() string { - return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) -} - -func (CachingMode) Descriptor() protoreflect.EnumDescriptor { - return file_pluggableharness_model_v1_types_proto_enumTypes[1].Descriptor() -} - -func (CachingMode) Type() protoreflect.EnumType { - return &file_pluggableharness_model_v1_types_proto_enumTypes[1] -} - -func (x CachingMode) Number() protoreflect.EnumNumber { - return protoreflect.EnumNumber(x) -} - -// Deprecated: Use CachingMode.Descriptor instead. -func (CachingMode) EnumDescriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{1} -} - // ToolChoiceMode enumerates the tool-invocation constraint shapes found // across researched vendors, per the same "declare precisely, don't // collapse to a bool" reasoning as ThinkingMode/CachingMode above. @@ -206,11 +144,11 @@ func (x ToolChoiceMode) String() string { } func (ToolChoiceMode) Descriptor() protoreflect.EnumDescriptor { - return file_pluggableharness_model_v1_types_proto_enumTypes[2].Descriptor() + return file_pluggableharness_model_v1_types_proto_enumTypes[1].Descriptor() } func (ToolChoiceMode) Type() protoreflect.EnumType { - return &file_pluggableharness_model_v1_types_proto_enumTypes[2] + return &file_pluggableharness_model_v1_types_proto_enumTypes[1] } func (x ToolChoiceMode) Number() protoreflect.EnumNumber { @@ -219,7 +157,7 @@ func (x ToolChoiceMode) Number() protoreflect.EnumNumber { // Deprecated: Use ToolChoiceMode.Descriptor instead. func (ToolChoiceMode) EnumDescriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{2} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{1} } // Capabilities is GetCapabilities' response payload: every model this @@ -778,24 +716,46 @@ func (x *ThinkingSpec) GetDisable() ThinkingDisableSupport { } // CachingSpec describes one model's prompt-caching capability, per -// model.md §2. +// model/data-types.md#cachingspec. +// +// These are independent axes, not one-of-N modes, for the same reason +// ThinkingSpec's are: Google's Gemini 2.5 and later run implicit automatic +// caching by default AND offer explicit manual declaration concurrently at +// a deeper discount. An earlier revision modeled this as a single +// mutually-exclusive enum, which forced such a model to under-declare +// itself — and, because cache_breakpoints were gated on that enum naming +// EXPLICIT_MARKERS, required it to discard breakpoints it could in fact +// have honored. type CachingSpec struct { state protoimpl.MessageState `protogen:"open.v1"` - // Whether this model has any prompt-caching capability at all. + // Whether this model has any prompt-caching capability at all. When + // false, explicit_markers and implicit_automatic MUST both be false; + // when true, at least one of them MUST be true — a model caching by a + // mechanism this protocol cannot name is not declarable, and declaring + // neither reads as "no caching" to every caller. Supported bool `protobuf:"varint,1,opt,name=supported,proto3" json:"supported,omitempty"` - // Which caching mechanic this model uses. MUST be CACHING_MODE_NONE - // when supported == false. - Mode CachingMode `protobuf:"varint,2,opt,name=mode,proto3,enum=pluggableharness.model.v1.CachingMode" json:"mode,omitempty"` // Whether this provider runs its own cache-keepalive loop (e.g. a // background goroutine re-pinging before a cache TTL expires, so a long // tool-execution gap doesn't let the cache go cold). MUST be set, // default false. Cache TTL mechanics are vendor-specific, so per // operator decision this is a provider-owned behavior the kernel never // drives — this field only tells the kernel/operator whether a given - // provider implements the optimization (model.md §2). + // provider implements the optimization. KeepaliveSupported bool `protobuf:"varint,3,opt,name=keepalive_supported,json=keepaliveSupported,proto3" json:"keepalive_supported,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // Whether the caller may place cache breakpoints on content blocks and + // have the adapter translate them into vendor-native markers (an + // Anthropic cache_control block, a Mistral prompt_cache_key). This is + // the axis StreamCompletionRequest.cache_breakpoints is gated on: an + // adapter for a model that does not declare it MUST ignore that field + // rather than error on it. + ExplicitMarkers bool `protobuf:"varint,4,opt,name=explicit_markers,json=explicitMarkers,proto3" json:"explicit_markers,omitempty"` + // Whether the vendor caches transparently above some token threshold + // with no caller action. Declaring this requires nothing of the kernel; + // it exists so cache-hit and cost behavior are explicable rather than + // surprising. + ImplicitAutomatic bool `protobuf:"varint,5,opt,name=implicit_automatic,json=implicitAutomatic,proto3" json:"implicit_automatic,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *CachingSpec) Reset() { @@ -835,16 +795,23 @@ func (x *CachingSpec) GetSupported() bool { return false } -func (x *CachingSpec) GetMode() CachingMode { +func (x *CachingSpec) GetKeepaliveSupported() bool { if x != nil { - return x.Mode + return x.KeepaliveSupported } - return CachingMode_CACHING_MODE_UNSPECIFIED + return false } -func (x *CachingSpec) GetKeepaliveSupported() bool { +func (x *CachingSpec) GetExplicitMarkers() bool { if x != nil { - return x.KeepaliveSupported + return x.ExplicitMarkers + } + return false +} + +func (x *CachingSpec) GetImplicitAutomatic() bool { + if x != nil { + return x.ImplicitAutomatic } return false } @@ -1772,11 +1739,12 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "\adisable\x18\n" + " \x01(\x0e21.pluggableharness.model.v1.ThinkingDisableSupportR\adisableB\t\n" + "\a_effortB\t\n" + - "\a_budgetJ\x04\b\x02\x10\aR\fbudget_rangeR\vcan_disableR\adefaultR\reffort_levelsR\x04mode\"\x98\x01\n" + + "\a_budgetJ\x04\b\x02\x10\aR\fbudget_rangeR\vcan_disableR\adefaultR\reffort_levelsR\x04mode\"\xc2\x01\n" + "\vCachingSpec\x12\x1c\n" + - "\tsupported\x18\x01 \x01(\bR\tsupported\x12:\n" + - "\x04mode\x18\x02 \x01(\x0e2&.pluggableharness.model.v1.CachingModeR\x04mode\x12/\n" + - "\x13keepalive_supported\x18\x03 \x01(\bR\x12keepaliveSupported\"\xe1\x05\n" + + "\tsupported\x18\x01 \x01(\bR\tsupported\x12/\n" + + "\x13keepalive_supported\x18\x03 \x01(\bR\x12keepaliveSupported\x12)\n" + + "\x10explicit_markers\x18\x04 \x01(\bR\x0fexplicitMarkers\x12-\n" + + "\x12implicit_automatic\x18\x05 \x01(\bR\x11implicitAutomaticJ\x04\b\x02\x10\x03R\x04mode\"\xe1\x05\n" + "\vPricingTier\x12F\n" + "\x0eeffective_from\x18\x01 \x01(\v2\x1a.google.protobuf.TimestampH\x00R\reffectiveFrom\x88\x01\x01\x12H\n" + "\x0feffective_until\x18\x02 \x01(\v2\x1a.google.protobuf.TimestampH\x01R\x0eeffectiveUntil\x88\x01\x01\x12$\n" + @@ -1854,12 +1822,7 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "$THINKING_DISABLE_SUPPORT_UNSPECIFIED\x10\x00\x12\"\n" + "\x1eTHINKING_DISABLE_SUPPORT_NEVER\x10\x01\x12#\n" + "\x1fTHINKING_DISABLE_SUPPORT_ALWAYS\x10\x02\x12(\n" + - "$THINKING_DISABLE_SUPPORT_CONDITIONAL\x10\x03*\x8a\x01\n" + - "\vCachingMode\x12\x1c\n" + - "\x18CACHING_MODE_UNSPECIFIED\x10\x00\x12\x15\n" + - "\x11CACHING_MODE_NONE\x10\x01\x12!\n" + - "\x1dCACHING_MODE_EXPLICIT_MARKERS\x10\x02\x12#\n" + - "\x1fCACHING_MODE_IMPLICIT_AUTOMATIC\x10\x03*\xa1\x01\n" + + "$THINKING_DISABLE_SUPPORT_CONDITIONAL\x10\x03*\xa1\x01\n" + "\x0eToolChoiceMode\x12 \n" + "\x1cTOOL_CHOICE_MODE_UNSPECIFIED\x10\x00\x12\x19\n" + "\x15TOOL_CHOICE_MODE_AUTO\x10\x01\x12\x18\n" + @@ -1879,63 +1842,61 @@ func file_pluggableharness_model_v1_types_proto_rawDescGZIP() []byte { return file_pluggableharness_model_v1_types_proto_rawDescData } -var file_pluggableharness_model_v1_types_proto_enumTypes = make([]protoimpl.EnumInfo, 3) +var file_pluggableharness_model_v1_types_proto_enumTypes = make([]protoimpl.EnumInfo, 2) var file_pluggableharness_model_v1_types_proto_msgTypes = make([]protoimpl.MessageInfo, 18) var file_pluggableharness_model_v1_types_proto_goTypes = []any{ (ThinkingDisableSupport)(0), // 0: pluggableharness.model.v1.ThinkingDisableSupport - (CachingMode)(0), // 1: pluggableharness.model.v1.CachingMode - (ToolChoiceMode)(0), // 2: pluggableharness.model.v1.ToolChoiceMode - (*Capabilities)(nil), // 3: pluggableharness.model.v1.Capabilities - (*ModelSpec)(nil), // 4: pluggableharness.model.v1.ModelSpec - (*ThinkingBudgetRange)(nil), // 5: pluggableharness.model.v1.ThinkingBudgetRange - (*EffortControl)(nil), // 6: pluggableharness.model.v1.EffortControl - (*BudgetControl)(nil), // 7: pluggableharness.model.v1.BudgetControl - (*ThinkingSpec)(nil), // 8: pluggableharness.model.v1.ThinkingSpec - (*CachingSpec)(nil), // 9: pluggableharness.model.v1.CachingSpec - (*PricingTier)(nil), // 10: pluggableharness.model.v1.PricingTier - (*Pricing)(nil), // 11: pluggableharness.model.v1.Pricing - (*CacheBreakpoint)(nil), // 12: pluggableharness.model.v1.CacheBreakpoint - (*ToolDeclaration)(nil), // 13: pluggableharness.model.v1.ToolDeclaration - (*GenerationParams)(nil), // 14: pluggableharness.model.v1.GenerationParams - (*ToolChoice)(nil), // 15: pluggableharness.model.v1.ToolChoice - (*Usage)(nil), // 16: pluggableharness.model.v1.Usage - (*ModelTarget)(nil), // 17: pluggableharness.model.v1.ModelTarget - (*ModelRef)(nil), // 18: pluggableharness.model.v1.ModelRef - (*CacheBreakpoint_AfterAssembledContext)(nil), // 19: pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext - (*CacheBreakpoint_AfterTools)(nil), // 20: pluggableharness.model.v1.CacheBreakpoint.AfterTools - (*v1.PromptExpansionSpec)(nil), // 21: pluggableharness.common.v1.PromptExpansionSpec - (*v11.ConfigSchema)(nil), // 22: pluggableharness.config.v1.ConfigSchema - (v1.HookPoint)(0), // 23: pluggableharness.common.v1.HookPoint - (*timestamppb.Timestamp)(nil), // 24: google.protobuf.Timestamp - (*v12.Schema)(nil), // 25: pluggableharness.schema.v1.Schema + (ToolChoiceMode)(0), // 1: pluggableharness.model.v1.ToolChoiceMode + (*Capabilities)(nil), // 2: pluggableharness.model.v1.Capabilities + (*ModelSpec)(nil), // 3: pluggableharness.model.v1.ModelSpec + (*ThinkingBudgetRange)(nil), // 4: pluggableharness.model.v1.ThinkingBudgetRange + (*EffortControl)(nil), // 5: pluggableharness.model.v1.EffortControl + (*BudgetControl)(nil), // 6: pluggableharness.model.v1.BudgetControl + (*ThinkingSpec)(nil), // 7: pluggableharness.model.v1.ThinkingSpec + (*CachingSpec)(nil), // 8: pluggableharness.model.v1.CachingSpec + (*PricingTier)(nil), // 9: pluggableharness.model.v1.PricingTier + (*Pricing)(nil), // 10: pluggableharness.model.v1.Pricing + (*CacheBreakpoint)(nil), // 11: pluggableharness.model.v1.CacheBreakpoint + (*ToolDeclaration)(nil), // 12: pluggableharness.model.v1.ToolDeclaration + (*GenerationParams)(nil), // 13: pluggableharness.model.v1.GenerationParams + (*ToolChoice)(nil), // 14: pluggableharness.model.v1.ToolChoice + (*Usage)(nil), // 15: pluggableharness.model.v1.Usage + (*ModelTarget)(nil), // 16: pluggableharness.model.v1.ModelTarget + (*ModelRef)(nil), // 17: pluggableharness.model.v1.ModelRef + (*CacheBreakpoint_AfterAssembledContext)(nil), // 18: pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext + (*CacheBreakpoint_AfterTools)(nil), // 19: pluggableharness.model.v1.CacheBreakpoint.AfterTools + (*v1.PromptExpansionSpec)(nil), // 20: pluggableharness.common.v1.PromptExpansionSpec + (*v11.ConfigSchema)(nil), // 21: pluggableharness.config.v1.ConfigSchema + (v1.HookPoint)(0), // 22: pluggableharness.common.v1.HookPoint + (*timestamppb.Timestamp)(nil), // 23: google.protobuf.Timestamp + (*v12.Schema)(nil), // 24: pluggableharness.schema.v1.Schema } var file_pluggableharness_model_v1_types_proto_depIdxs = []int32{ - 4, // 0: pluggableharness.model.v1.Capabilities.models:type_name -> pluggableharness.model.v1.ModelSpec - 21, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec - 22, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema - 23, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint - 8, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec - 9, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec - 11, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing - 2, // 7: pluggableharness.model.v1.ModelSpec.supported_tool_choice_modes:type_name -> pluggableharness.model.v1.ToolChoiceMode - 5, // 8: pluggableharness.model.v1.BudgetControl.range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange - 6, // 9: pluggableharness.model.v1.ThinkingSpec.effort:type_name -> pluggableharness.model.v1.EffortControl - 7, // 10: pluggableharness.model.v1.ThinkingSpec.budget:type_name -> pluggableharness.model.v1.BudgetControl + 3, // 0: pluggableharness.model.v1.Capabilities.models:type_name -> pluggableharness.model.v1.ModelSpec + 20, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec + 21, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema + 22, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint + 7, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec + 8, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec + 10, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing + 1, // 7: pluggableharness.model.v1.ModelSpec.supported_tool_choice_modes:type_name -> pluggableharness.model.v1.ToolChoiceMode + 4, // 8: pluggableharness.model.v1.BudgetControl.range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange + 5, // 9: pluggableharness.model.v1.ThinkingSpec.effort:type_name -> pluggableharness.model.v1.EffortControl + 6, // 10: pluggableharness.model.v1.ThinkingSpec.budget:type_name -> pluggableharness.model.v1.BudgetControl 0, // 11: pluggableharness.model.v1.ThinkingSpec.disable:type_name -> pluggableharness.model.v1.ThinkingDisableSupport - 1, // 12: pluggableharness.model.v1.CachingSpec.mode:type_name -> pluggableharness.model.v1.CachingMode - 24, // 13: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp - 24, // 14: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp - 10, // 15: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier - 19, // 16: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext - 20, // 17: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools - 25, // 18: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema - 15, // 19: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice - 2, // 20: pluggableharness.model.v1.ToolChoice.mode:type_name -> pluggableharness.model.v1.ToolChoiceMode - 21, // [21:21] is the sub-list for method output_type - 21, // [21:21] is the sub-list for method input_type - 21, // [21:21] is the sub-list for extension type_name - 21, // [21:21] is the sub-list for extension extendee - 0, // [0:21] is the sub-list for field type_name + 23, // 12: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp + 23, // 13: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp + 9, // 14: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier + 18, // 15: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext + 19, // 16: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools + 24, // 17: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema + 14, // 18: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice + 1, // 19: pluggableharness.model.v1.ToolChoice.mode:type_name -> pluggableharness.model.v1.ToolChoiceMode + 20, // [20:20] is the sub-list for method output_type + 20, // [20:20] is the sub-list for method input_type + 20, // [20:20] is the sub-list for extension type_name + 20, // [20:20] is the sub-list for extension extendee + 0, // [0:20] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_types_proto_init() } @@ -1960,7 +1921,7 @@ func file_pluggableharness_model_v1_types_proto_init() { File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_pluggableharness_model_v1_types_proto_rawDesc), len(file_pluggableharness_model_v1_types_proto_rawDesc)), - NumEnums: 3, + NumEnums: 2, NumMessages: 18, NumExtensions: 0, NumServices: 0, diff --git a/pkg/model/server_test.go b/pkg/model/server_test.go index e36e416..cd25d59 100644 --- a/pkg/model/server_test.go +++ b/pkg/model/server_test.go @@ -41,7 +41,7 @@ func (f *fakeProvider) Capabilities(ctx context.Context) (*model.Capabilities, e return model.NewCapabilities([]model.Spec{{ ID: "fake-model", Thinking: model.ThinkingSpec{}, - Caching: model.CachingSpec{Mode: modelv1.CachingMode_CACHING_MODE_NONE}, + Caching: model.CachingSpec{}, Pricing: model.Pricing{Currency: "USD", Free: true}, }}, &configv1.ConfigSchema{}) } From 01910f68b0a791970e1954eca4799e92dde9c138 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:16:21 -0400 Subject: [PATCH 04/16] model: count a request in CountTokens, not a string MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit CountTokensRequest carried a flat `text` field, but every vendor that exposes exact counting counts a whole request: Anthropic's /v1/messages/count_tokens takes messages plus system plus tools and returns that request's input-token total. The old shape could not express the question the kernel actually asks. Worse, it discarded the two things most likely to dominate the answer — tool schemas and the system preamble — so a count omitted exactly the weight that decides whether a turn fits in the context window. The round trip was lossy twice over: internal/tokencount flattened content blocks into a joined string, and the Anthropic client then re-wrapped that string back into a single user message to satisfy the endpoint. Replace it with messages + assembled_context + tools + model_id, mirroring StreamCompletionRequest's content-bearing fields minus everything that only affects generation. A caller holding loose content passes it as one user message, which is what the adapter had to construct anyway. BuildCountTokensRequest runs the same buildSystem/buildTools/ translateMessage path BuildRequest does, so a count is computed over exactly the content a completion would have carried; any divergence would make the count silently unrepresentative of the request it sizes. One deliberate consequence: internal/tokencount now forwards every block, not just the text ones, so the exact path and the ceil(bytes/4) fallback no longer measure the same input. That asymmetry is the correct one — the vendor charges for an image, so including it makes the exact count more accurate, while the fallback keeps under-estimating non-text content. That package's CLAUDE.md previously required the two to stay symmetric and now records why they no longer are. --- .../model/v1/rpc_request.proto | 36 ++++++++- docs/specifications/model/protocol.md | 10 ++- internal/anthropic/messages/client.go | 25 +++--- internal/anthropic/messages/client_test.go | 29 ++++++- internal/anthropic/messages/request.go | 49 ++++++++++++ internal/anthropic/messages/request_test.go | 62 +++++++++++++++ internal/anthropic/provider.go | 7 +- internal/anthropic/provider_test.go | 22 +++++- internal/tokencount/CLAUDE.md | 2 +- internal/tokencount/tokencount.go | 45 +++++++---- internal/tokencount/tokencount_test.go | 9 ++- pkg/model/model.go | 19 +++-- pkg/model/proto/v1/rpc_request.pb.go | 79 ++++++++++++++----- pkg/model/server.go | 2 +- pkg/model/server_test.go | 35 +++++--- 15 files changed, 353 insertions(+), 78 deletions(-) diff --git a/api/pluggableharness/model/v1/rpc_request.proto b/api/pluggableharness/model/v1/rpc_request.proto index dc30d62..19b15f8 100644 --- a/api/pluggableharness/model/v1/rpc_request.proto +++ b/api/pluggableharness/model/v1/rpc_request.proto @@ -116,16 +116,44 @@ message StreamCompletionRequest { optional google.protobuf.Struct provider_options = 8; } -// CountTokensRequest is CountTokens' request: the raw text to count, per -// model.md §2.1. +// CountTokensRequest is CountTokens' request: the request whose input +// tokens are being counted, per model/protocol.md#counttokens. +// +// This mirrors StreamCompletionRequest's content-bearing fields, minus +// everything that only affects generation, because every vendor exposing +// exact counting counts a whole request rather than a string — Anthropic's +// /v1/messages/count_tokens takes messages plus system plus tools. An +// earlier revision carried a flat `text` field; concatenating text and +// discarding the rest undercounts by the entire tool-schema and +// system-preamble weight, which is exactly the weight that decides whether +// a turn fits in the context window. message CountTokensRequest { - // The text to count tokens for. - string text = 1; + reserved 1; + + reserved "text"; // Selects which of this provider's ModelSpec.id to count against — a // provider serving several models MAY have distinct tokenizers per // model. MUST be set. string model_id = 2; + + // The conversation to count, in emission order. MAY be empty. A caller + // with only loose content to measure (a context provider sizing its own + // contribution, kernel-callbacks.md#counttokens) passes it as a single + // user message — which is what the adapter would have had to construct + // anyway. + repeated pluggableharness.content.v1.Message messages = 3; + + // The kernel-assembled context chain that would accompany these + // messages, counted against the same vendor mechanism + // StreamCompletionRequest.assembled_context maps to. MAY be empty. + repeated pluggableharness.content.v1.ContextSection assembled_context = 4; + + // The tool declarations that would accompany these messages. MAY be + // empty. Tool schemas are frequently the largest single contributor to + // a request's input tokens, so omitting them is the main way a count + // goes badly wrong. + repeated ToolDeclaration tools = 5; } // RenderRequest carries the opaque payload to render, per model.md §7. diff --git a/docs/specifications/model/protocol.md b/docs/specifications/model/protocol.md index 50a86b8..b0b1b63 100644 --- a/docs/specifications/model/protocol.md +++ b/docs/specifications/model/protocol.md @@ -13,9 +13,17 @@ The response also carries `supported_hook_points: []common.v1.HookPoint` ([`data ### `CountTokens` ```text -CountTokens(text: string, model_id: string) -> { count: int } +CountTokens(CountTokensRequest{ + model_id, messages, assembled_context, tools +}) -> { count: int } ``` +`CountTokens` counts **a request**, not a string. Its request mirrors [`StreamCompletionRequest`](data-types.md#streamcompletionrequest)'s content-bearing fields — `messages`, `assembled_context`, and `tools` — minus everything that only affects generation (`params`, `cache_breakpoints`, `call_context`). + +This shape is what the question actually requires. Every vendor that exposes exact counting counts a whole request: Anthropic's `/v1/messages/count_tokens` takes `messages` plus `system` plus `tools` and returns the input-token total for that request. A flat string cannot express the question "how many tokens is this conversation", and answering it by concatenating text and discarding the rest undercounts by the entire tool-schema and system-preamble weight — which is precisely the weight that decides whether a turn fits in the context window. A caller with only loose content to measure (a context provider sizing its own contribution via [`kernel-callbacks.md#counttokens`](../kernel-callbacks.md#counttokens)) passes it as a single user message; that is what the adapter would have had to construct anyway. + +Every field except `model_id` MAY be empty, and an empty request MUST count as whatever that vendor charges for an empty request — usually not zero, since most vendors bill some fixed request overhead. + `model_id` MUST be set on every `CountTokensRequest` — it selects which of this provider's `ModelSpec.id` to count against, since a provider serving several models MAY use a distinct tokenizer per model. SHOULD be implemented per model, using that vendor's real tokenizer: rather than investing in a smarter kernel-side fallback heuristic, the expectation is that providers actually implement this against real vendor tokenizers wherever the vendor makes it available, and the fallback ([`kernel-callbacks.md#the-fallback-heuristic`](../kernel-callbacks.md#the-fallback-heuristic)) stays a genuine last resort, not a normal operating path. This is the model-provider side of [`kernel-callbacks.md`](../kernel-callbacks.md)'s `CountTokens` primitive — a model provider that implements this gets its counts marked `exact: true` when the kernel resolves a `CountTokens` call against it; a model provider that doesn't falls back to the documented heuristic. Still not a MUST, because not every vendor makes exact counting cheap or even possible without a network round-trip — but a provider author should treat skipping it as the exception, not the default. diff --git a/internal/anthropic/messages/client.go b/internal/anthropic/messages/client.go index d636ff3..fd21ebf 100644 --- a/internal/anthropic/messages/client.go +++ b/internal/anthropic/messages/client.go @@ -199,22 +199,26 @@ func (c *Client) drive(ctx context.Context, body io.Reader, sink EventSink) erro return sink.Error(truncated) } -// CountTokens returns an exact count for text against modelID via +// CountTokens returns an exact input-token count for req against // POST /v1/messages/count_tokens. -func (c *Client) CountTokens(ctx context.Context, text, modelID string) (int64, error) { +// +// The endpoint takes the same messages/system/tools triple the completion +// endpoint does, so this translates req through the same builders +// BuildRequest uses rather than flattening it to a string — tool schemas +// in particular are frequently the largest single contributor to a +// request's input tokens, and dropping them was the main way the earlier +// text-only shape produced a badly wrong number. +func (c *Client) CountTokens(ctx context.Context, req *modelv1.CountTokensRequest, spec model.Spec) (int64, error) { // Same pre-flight cancellation check as Stream, for the same reason. if err := ctx.Err(); err != nil { return 0, err } - reqBody := struct { - Model string `json:"model"` - Messages []Message `json:"messages"` - }{ - Model: modelID, - Messages: []Message{{Role: roleUser, Content: []Block{{Type: blockText, Text: text}}}}, + countReq, err := BuildCountTokensRequest(req, spec) + if err != nil { + return 0, err } - body, err := json.Marshal(reqBody) + body, err := json.Marshal(countReq) if err != nil { return 0, fmt.Errorf("anthropic: count tokens: encode request: %w", err) } @@ -225,7 +229,8 @@ func (c *Client) CountTokens(ctx context.Context, text, modelID string) (int64, } c.setHeaders(httpReq) - c.logger.DebugContext(ctx, "anthropic: count tokens: request", "method", httpReq.Method, "path", countTokensPath, "model", modelID) + c.logger.DebugContext(ctx, "anthropic: count tokens: request", "method", httpReq.Method, "path", countTokensPath, "model", countReq.Model, + "messages", len(countReq.Messages), "tools", len(countReq.Tools)) resp, err := c.http.Do(httpReq) if err != nil { diff --git a/internal/anthropic/messages/client_test.go b/internal/anthropic/messages/client_test.go index 07bece0..47ee56c 100644 --- a/internal/anthropic/messages/client_test.go +++ b/internal/anthropic/messages/client_test.go @@ -12,10 +12,31 @@ import ( "testing" "time" + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" "github.com/pluggableharness/agent/pkg/model" modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" ) +// countReq builds the request-shaped CountTokensRequest the RPC now takes, +// carrying loose text as one user message. +func countReq(text, modelID string) *modelv1.CountTokensRequest { + return &modelv1.CountTokensRequest{ + ModelId: modelID, + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{{ + Block: &contentv1.ContentBlock_Text{Text: &contentv1.TextBlock{Text: text}}, + }}, + }}, + } +} + +// specFor is the minimal model.Spec CountTokens needs to translate a +// request: enough to accept text blocks, nothing more. +func specFor(id string) model.Spec { + return model.Spec{ID: id, MaxOutputTokens: 4096} +} + // roundTripFunc adapts a function to http.RoundTripper, the standard // fake-transport seam for testing an *http.Client without a real network // call. @@ -382,7 +403,7 @@ func TestClient_CountTokens_success(t *testing.T) { }) client := newTestClient(transport) - got, err := client.CountTokens(context.Background(), "hello world", "claude-opus-5") + got, err := client.CountTokens(context.Background(), countReq("hello world", "claude-opus-5"), specFor("claude-opus-5")) if err != nil { t.Fatalf("CountTokens: %v", err) } @@ -404,7 +425,7 @@ func TestClient_CountTokens_malformedResponseBody(t *testing.T) { }) client := newTestClient(transport) - got, err := client.CountTokens(context.Background(), "hi", "claude-opus-5") + got, err := client.CountTokens(context.Background(), countReq("hi", "claude-opus-5"), specFor("claude-opus-5")) if err == nil { t.Fatalf("expected a decode error") } @@ -425,7 +446,7 @@ func TestClient_CountTokens_nonRetryableClassification(t *testing.T) { }) client := newTestClient(transport) - got, err := client.CountTokens(context.Background(), "hi", "unknown-model") + got, err := client.CountTokens(context.Background(), countReq("hi", "unknown-model"), specFor("unknown-model")) if got != 0 { t.Errorf("got %d, want 0 on error", got) } @@ -455,7 +476,7 @@ func TestClient_CountTokens_cancellation(t *testing.T) { ctx, cancel := context.WithCancel(context.Background()) cancel() - _, err := client.CountTokens(ctx, "hi", "claude-opus-5") + _, err := client.CountTokens(ctx, countReq("hi", "claude-opus-5"), specFor("claude-opus-5")) if !errors.Is(err, context.Canceled) { t.Fatalf("err = %v, want context.Canceled", err) } diff --git a/internal/anthropic/messages/request.go b/internal/anthropic/messages/request.go index 4a8d27e..c6dfeac 100644 --- a/internal/anthropic/messages/request.go +++ b/internal/anthropic/messages/request.go @@ -137,6 +137,55 @@ func BuildRequest(in *modelv1.StreamCompletionRequest, spec model.Spec) (*Reques }, nil } +// CountTokensRequest is the POST /v1/messages/count_tokens body. +// +// It is deliberately a narrower struct than Request rather than a reuse of +// it: the endpoint accepts only the fields that affect the input-token +// total, and sending generation params it does not expect risks a 400 for +// no benefit. +type CountTokensRequest struct { + Model string `json:"model"` + Messages []Message `json:"messages"` + System []TextBlock `json:"system,omitempty"` + Tools []Tool `json:"tools,omitempty"` +} + +// BuildCountTokensRequest translates in into the Anthropic count-tokens +// body for the model described by spec. +// +// It runs the same buildSystem/buildTools/translateMessage path +// BuildRequest does, so a count is computed over exactly the content a +// completion would have carried. Any divergence between the two would +// make the count silently unrepresentative of the request it is meant to +// size. +func BuildCountTokensRequest(in *modelv1.CountTokensRequest, spec model.Spec) (*CountTokensRequest, error) { + system, err := buildSystem(in.GetAssembledContext()) + if err != nil { + return nil, err + } + + tools, err := buildTools(in.GetTools()) + if err != nil { + return nil, err + } + + messages := make([]Message, len(in.GetMessages())) + for i, m := range in.GetMessages() { + msg, err := translateMessage(m, spec) + if err != nil { + return nil, err + } + messages[i] = msg + } + + return &CountTokensRequest{ + Model: in.GetModelId(), + Messages: coalesceMessages(messages), + System: system, + Tools: tools, + }, nil +} + // buildSystem translates the kernel-assembled context chain into // Anthropic's top-level `system` array, one TextBlock per section. Each // section's concatenated text is wrapped in a delimiter line built from its diff --git a/internal/anthropic/messages/request_test.go b/internal/anthropic/messages/request_test.go index 1e6fb54..31a45cf 100644 --- a/internal/anthropic/messages/request_test.go +++ b/internal/anthropic/messages/request_test.go @@ -4,6 +4,7 @@ import ( "encoding/base64" "encoding/json" "reflect" + "strings" "testing" "google.golang.org/protobuf/types/known/structpb" @@ -886,3 +887,64 @@ func TestCoalesceMessages(t *testing.T) { } }) } + +// TestBuildCountTokensRequest_carriesToolsAndSystem is the regression +// guard for what the request-shaped CountTokens RPC exists to fix. The +// earlier flat-text shape could carry neither tool schemas nor the system +// preamble, so a count omitted both — and tool schemas are frequently the +// single largest contributor to a request's input tokens, which is exactly +// the weight that decides whether a turn fits in the context window. +func TestBuildCountTokensRequest_carriesToolsAndSystem(t *testing.T) { + t.Parallel() + + in := &modelv1.CountTokensRequest{ + ModelId: "claude-opus-5", + Messages: []*contentv1.Message{{Role: contentv1.Role_ROLE_USER, Content: []*contentv1.ContentBlock{content.Text("what changed?")}}}, + AssembledContext: []*contentv1.ContextSection{{ + Provider: "project-context", + Label: "CLAUDE.md", + Content: []*contentv1.ContentBlock{content.Text("house rules")}, + }}, + Tools: []*modelv1.ToolDeclaration{{ + Name: "read", + Description: "read a file", + }}, + } + + got, err := BuildCountTokensRequest(in, fullSpec()) + if err != nil { + t.Fatalf("BuildCountTokensRequest: %v", err) + } + + if got.Model != "claude-opus-5" { + t.Errorf("Model = %q, want claude-opus-5", got.Model) + } + if len(got.Messages) != 1 { + t.Fatalf("len(Messages) = %d, want 1", len(got.Messages)) + } + if len(got.Tools) != 1 || got.Tools[0].Name != "read" { + t.Errorf("Tools = %+v, want the declared read tool", got.Tools) + } + if len(got.System) != 1 || !strings.Contains(got.System[0].Text, "house rules") { + t.Errorf("System = %+v, want the assembled context section", got.System) + } +} + +// TestBuildCountTokensRequest_emptyRequestIsValid proves an empty request +// is a legal thing to count: most vendors bill some fixed request +// overhead, so the answer is not necessarily zero and the adapter must not +// short-circuit it. +func TestBuildCountTokensRequest_emptyRequestIsValid(t *testing.T) { + t.Parallel() + + got, err := BuildCountTokensRequest(&modelv1.CountTokensRequest{ModelId: "claude-opus-5"}, fullSpec()) + if err != nil { + t.Fatalf("BuildCountTokensRequest: %v", err) + } + if got.Model != "claude-opus-5" { + t.Errorf("Model = %q, want claude-opus-5", got.Model) + } + if len(got.Messages) != 0 || len(got.Tools) != 0 || len(got.System) != 0 { + t.Errorf("got %+v, want an empty body apart from the model", got) + } +} diff --git a/internal/anthropic/provider.go b/internal/anthropic/provider.go index 3ddffbf..349ac81 100644 --- a/internal/anthropic/provider.go +++ b/internal/anthropic/provider.go @@ -161,15 +161,16 @@ func (p *Provider) StreamCompletion(ctx context.Context, req *modelv1.StreamComp // this is implemented even though it is only a SHOULD: Anthropic exposes // exact counting over a cheap endpoint, so declining to use it would be // choosing a worse number for no reason. -func (p *Provider) CountTokens(ctx context.Context, text, modelID string) (int64, error) { +func (p *Provider) CountTokens(ctx context.Context, req *modelv1.CountTokensRequest) (int64, error) { client, err := p.readyClient("count tokens") if err != nil { return 0, err } - if _, err := specByID(modelID); err != nil { + spec, err := specByID(req.GetModelId()) + if err != nil { return 0, err } - return client.CountTokens(ctx, text, modelID) + return client.CountTokens(ctx, req, spec) } // readyClient returns the configured vendor client, or the structured diff --git a/internal/anthropic/provider_test.go b/internal/anthropic/provider_test.go index aa51dcf..6b9c93b 100644 --- a/internal/anthropic/provider_test.go +++ b/internal/anthropic/provider_test.go @@ -9,10 +9,26 @@ import ( "google.golang.org/protobuf/types/known/structpb" + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" "github.com/pluggableharness/agent/pkg/model" modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" ) +// countReq builds the CountTokensRequest shape the provider's CountTokens +// RPC now takes: a whole request, with loose text carried as one user +// message (model/protocol.md#counttokens). +func countReq(text, modelID string) *modelv1.CountTokensRequest { + return &modelv1.CountTokensRequest{ + ModelId: modelID, + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{{ + Block: &contentv1.ContentBlock_Text{Text: &contentv1.TextBlock{Text: text}}, + }}, + }}, + } +} + // TestCapabilities_servesTheRosterAndSchema checks the one RPC the kernel // calls before every routing decision. It must not touch the network and // must not need Configure to have run — a provider that could only @@ -84,7 +100,7 @@ func TestRPCs_beforeConfigureAreRejected(t *testing.T) { &modelv1.StreamCompletionRequest{ModelId: "claude-opus-5"}, nil) assertInvalidRequest(t, streamErr, "not configured") - _, countErr := p.CountTokens(context.Background(), "hello", "claude-opus-5") + _, countErr := p.CountTokens(context.Background(), countReq("hello", "claude-opus-5")) assertInvalidRequest(t, countErr, "not configured") } @@ -121,7 +137,7 @@ func TestStreamCompletion_rejectsAnUnknownModel(t *testing.T) { &modelv1.StreamCompletionRequest{ModelId: "claude-does-not-exist"}, nil) assertInvalidRequest(t, err, "unknown model") - _, countErr := p.CountTokens(context.Background(), "hello", "claude-does-not-exist") + _, countErr := p.CountTokens(context.Background(), countReq("hello", "claude-does-not-exist")) assertInvalidRequest(t, countErr, "unknown model") } @@ -145,7 +161,7 @@ func TestConfigure_neverLeaksTheKey(t *testing.T) { // Force a transport-level failure and confirm the key is absent from // whatever comes back. - _, countErr := p.CountTokens(context.Background(), "hello", "claude-opus-5") + _, countErr := p.CountTokens(context.Background(), countReq("hello", "claude-opus-5")) if countErr == nil { t.Fatal("expected the failing transport to produce an error") } diff --git a/internal/tokencount/CLAUDE.md b/internal/tokencount/CLAUDE.md index 92991f7..a1413a5 100644 --- a/internal/tokencount/CLAUDE.md +++ b/internal/tokencount/CLAUDE.md @@ -8,7 +8,7 @@ - **`codes.Canceled`/`codes.DeadlineExceeded` are deliberately not counted against any of the four bounded `TokenCountFallbackReasonKey` values.** They're not a provider-side condition at all — the caller's own context is what ended the call — so `resolveError` returns the fallback without touching `recordFallback`. Don't add a fifth reason value for cancellation; that would need a change to `internal/telemetry` (a different phase) and isn't what the four-value enum in `attributes.go` is for. -- **`joinedText` (the exact-path RPC request text) and `Fallback`'s byte-length sum MUST stay computed over the identical concatenation** — same blocks, same order, same "text-only" filter. If one changes (e.g. adding a separator between blocks), the other must change identically, or a provider's exact count and the fallback estimate stop being comparable for the same input, silently reintroducing the cross-provider inconsistency `kernel-callbacks.md`'s "why a kernel primitive" section exists to prevent. +- **The exact path and `Fallback` deliberately no longer measure the same thing, and that asymmetry is the correct one.** `countingMessages` forwards **every** block to the provider's `CountTokens` RPC — including images and documents — because the vendor genuinely charges for them and `model/protocol.md#counttokens` counts a whole request rather than a string. `Fallback` still sums text bytes only, per its own hard rule, so it under-estimates any non-text content. Do not "fix" this by filtering non-text blocks out of `countingMessages` to restore symmetry: that would make the exact count wrong in order to match an estimate that was never meant to be accurate. The gap is one more reason `kernel-callbacks.md` calls the fallback a genuine last resort rather than a normal operating path. (An earlier revision did keep the two symmetric, because the provider RPC took a flat `text` string and there was nothing else to send.) - **`internal/telemetry.Instruments.TokenCountFallbacks` and `TokenCountFallbackReasonKey` already existed before this package was written** (a prior phase of the same kernel-callbacks effort) — this package only calls `Add`, it never adds a new metric or attribute key to `internal/telemetry` itself. If a fifth fallback scenario is ever needed, that's a change to `internal/telemetry`, done deliberately, not a value invented locally here. diff --git a/internal/tokencount/tokencount.go b/internal/tokencount/tokencount.go index be00f5d..bcc7269 100644 --- a/internal/tokencount/tokencount.go +++ b/internal/tokencount/tokencount.go @@ -3,7 +3,6 @@ package tokencount import ( "context" "log/slog" - "strings" "sync" "google.golang.org/grpc/codes" @@ -38,20 +37,34 @@ func Fallback(blocks []*contentv1.ContentBlock) int64 { return (totalBytes + 3) / 4 } -// joinedText concatenates every text block's content, in order, into the -// single string a model provider's own CountTokensRequest.Text expects — -// the same "text_of(content)" Fallback sums the byte length of, so the two -// counts (exact vendor tokenizer vs. fallback heuristic) are always -// computed over identical text. Non-text blocks contribute nothing, same -// as Fallback. -func joinedText(blocks []*contentv1.ContentBlock) string { - var sb strings.Builder - for _, block := range blocks { - if text := block.GetText(); text != nil { - sb.WriteString(text.GetText()) - } +// countingMessages wraps blocks into the single user message a model +// provider's CountTokensRequest expects, per +// docs/specifications/model/protocol.md#counttokens: that RPC counts a +// request, and a caller holding only loose content passes it as one +// message — which is what every vendor's counting endpoint would have +// required the adapter to construct anyway. +// +// Every block is forwarded, not just the text ones. That is deliberate and +// is where the exact path now diverges from Fallback: the vendor charges +// for an image or a document, so including it makes the exact count more +// accurate, while Fallback still sums text bytes only and therefore +// under-estimates any non-text content. The two were never equal — one is +// a real tokenizer, the other is ceil(bytes/4) — and this widens the gap +// in the direction of the exact path being right. It is one more reason +// the fallback is documented as a genuine last resort rather than a normal +// operating path. +// +// No message id is set: this request is transient, never persisted, and +// ids are kernel-assigned at persist time +// (docs/specifications/model/data-types.md#message-identity-and-model-attribution). +func countingMessages(blocks []*contentv1.ContentBlock) []*contentv1.Message { + if len(blocks) == 0 { + return nil } - return sb.String() + return []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: blocks, + }} } // ModelLookup is how a Counter reaches a model provider's own optional @@ -150,8 +163,8 @@ func (c *Counter) Count(ctx context.Context, blocks []*contentv1.ContentBlock, r } resp, err := client.CountTokens(ctx, &modelv1.CountTokensRequest{ - Text: joinedText(blocks), - ModelId: ref.GetId(), + ModelId: ref.GetId(), + Messages: countingMessages(blocks), }) if err != nil { return c.resolveError(ctx, blocks, provider, err) diff --git a/internal/tokencount/tokencount_test.go b/internal/tokencount/tokencount_test.go index 5e065c4..e9c0c16 100644 --- a/internal/tokencount/tokencount_test.go +++ b/internal/tokencount/tokencount_test.go @@ -208,7 +208,14 @@ func TestCounter_Count_success(t *testing.T) { if len(client.calls) != 1 { t.Fatalf("CountTokens calls = %d, want 1", len(client.calls)) } - if got, want := client.calls[0].GetText(), "hello world"; got != want { + // The blocks reach the provider as one user message, not a flattened + // string — the request-shaped CountTokens RPC is what makes tool + // schemas and non-text content countable at all. + msgs := client.calls[0].GetMessages() + if len(msgs) != 1 || msgs[0].GetRole() != contentv1.Role_ROLE_USER { + t.Fatalf("messages = %+v, want one user message", msgs) + } + if got, want := msgs[0].GetContent()[0].GetText().GetText(), "hello world"; got != want { t.Errorf("request text = %q, want %q", got, want) } if got, want := client.calls[0].GetModelId(), "claude"; got != want { diff --git a/pkg/model/model.go b/pkg/model/model.go index 3980238..bb016de 100644 --- a/pkg/model/model.go +++ b/pkg/model/model.go @@ -61,12 +61,21 @@ type Provider interface { // io.ReaderFrom) rather than adding a boolean capability flag a Provider // must remember to keep in sync with its own method set. type TokenCounter interface { - // CountTokens returns an exact token count for text against modelID's + // CountTokens returns an exact input-token count for req against the // real vendor tokenizer, per - // docs/specifications/model/protocol.md#counttokens. modelID MUST be - // honored — a provider serving several models MAY use a distinct - // tokenizer per model. - CountTokens(ctx context.Context, text, modelID string) (int64, error) + // docs/specifications/model/protocol.md#counttokens. + // + // req names a whole request — messages, assembled context, and tool + // declarations — not a string, because that is what every vendor's + // counting endpoint actually measures, and because tool schemas are + // frequently the largest single contributor to a request's input + // tokens. req.ModelId MUST be honored: a provider serving several + // models MAY use a distinct tokenizer per model. + // + // Takes the generated request type directly, for the same reason + // StreamCompletion does (see this package's doc.go): it is already the + // canonical shape, and mirroring it would be a purely duplicative copy. + CountTokens(ctx context.Context, req *modelv1.CountTokensRequest) (int64, error) } // Renderer is the optional interface behind Render diff --git a/pkg/model/proto/v1/rpc_request.pb.go b/pkg/model/proto/v1/rpc_request.pb.go index 1b3cffb..8b1b995 100644 --- a/pkg/model/proto/v1/rpc_request.pb.go +++ b/pkg/model/proto/v1/rpc_request.pb.go @@ -319,16 +319,38 @@ func (x *StreamCompletionRequest) GetProviderOptions() *structpb.Struct { return nil } -// CountTokensRequest is CountTokens' request: the raw text to count, per -// model.md §2.1. +// CountTokensRequest is CountTokens' request: the request whose input +// tokens are being counted, per model/protocol.md#counttokens. +// +// This mirrors StreamCompletionRequest's content-bearing fields, minus +// everything that only affects generation, because every vendor exposing +// exact counting counts a whole request rather than a string — Anthropic's +// /v1/messages/count_tokens takes messages plus system plus tools. An +// earlier revision carried a flat `text` field; concatenating text and +// discarding the rest undercounts by the entire tool-schema and +// system-preamble weight, which is exactly the weight that decides whether +// a turn fits in the context window. type CountTokensRequest struct { state protoimpl.MessageState `protogen:"open.v1"` - // The text to count tokens for. - Text string `protobuf:"bytes,1,opt,name=text,proto3" json:"text,omitempty"` // Selects which of this provider's ModelSpec.id to count against — a // provider serving several models MAY have distinct tokenizers per // model. MUST be set. - ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` + ModelId string `protobuf:"bytes,2,opt,name=model_id,json=modelId,proto3" json:"model_id,omitempty"` + // The conversation to count, in emission order. MAY be empty. A caller + // with only loose content to measure (a context provider sizing its own + // contribution, kernel-callbacks.md#counttokens) passes it as a single + // user message — which is what the adapter would have had to construct + // anyway. + Messages []*v1.Message `protobuf:"bytes,3,rep,name=messages,proto3" json:"messages,omitempty"` + // The kernel-assembled context chain that would accompany these + // messages, counted against the same vendor mechanism + // StreamCompletionRequest.assembled_context maps to. MAY be empty. + AssembledContext []*v1.ContextSection `protobuf:"bytes,4,rep,name=assembled_context,json=assembledContext,proto3" json:"assembled_context,omitempty"` + // The tool declarations that would accompany these messages. MAY be + // empty. Tool schemas are frequently the largest single contributor to + // a request's input tokens, so omitting them is the main way a count + // goes badly wrong. + Tools []*ToolDeclaration `protobuf:"bytes,5,rep,name=tools,proto3" json:"tools,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } @@ -363,18 +385,32 @@ func (*CountTokensRequest) Descriptor() ([]byte, []int) { return file_pluggableharness_model_v1_rpc_request_proto_rawDescGZIP(), []int{4} } -func (x *CountTokensRequest) GetText() string { +func (x *CountTokensRequest) GetModelId() string { if x != nil { - return x.Text + return x.ModelId } return "" } -func (x *CountTokensRequest) GetModelId() string { +func (x *CountTokensRequest) GetMessages() []*v1.Message { if x != nil { - return x.ModelId + return x.Messages } - return "" + return nil +} + +func (x *CountTokensRequest) GetAssembledContext() []*v1.ContextSection { + if x != nil { + return x.AssembledContext + } + return nil +} + +func (x *CountTokensRequest) GetTools() []*ToolDeclaration { + if x != nil { + return x.Tools + } + return nil } // RenderRequest carries the opaque payload to render, per model.md §7. @@ -458,10 +494,12 @@ const file_pluggableharness_model_v1_rpc_request_proto_rawDesc = "" + "\x11cache_breakpoints\x18\a \x03(\v2*.pluggableharness.model.v1.CacheBreakpointR\x10cacheBreakpoints\x12G\n" + "\x10provider_options\x18\b \x01(\v2\x17.google.protobuf.StructH\x01R\x0fproviderOptions\x88\x01\x01B\t\n" + "\a_paramsB\x13\n" + - "\x11_provider_options\"C\n" + - "\x12CountTokensRequest\x12\x12\n" + - "\x04text\x18\x01 \x01(\tR\x04text\x12\x19\n" + - "\bmodel_id\x18\x02 \x01(\tR\amodelId\"P\n" + + "\x11_provider_options\"\x99\x02\n" + + "\x12CountTokensRequest\x12\x19\n" + + "\bmodel_id\x18\x02 \x01(\tR\amodelId\x12@\n" + + "\bmessages\x18\x03 \x03(\v2$.pluggableharness.content.v1.MessageR\bmessages\x12X\n" + + "\x11assembled_context\x18\x04 \x03(\v2+.pluggableharness.content.v1.ContextSectionR\x10assembledContext\x12@\n" + + "\x05tools\x18\x05 \x03(\v2*.pluggableharness.model.v1.ToolDeclarationR\x05toolsJ\x04\b\x01\x10\x02R\x04text\"P\n" + "\rRenderRequest\x12\x18\n" + "\apayload\x18\x01 \x01(\fR\apayload\x12%\n" + "\x0eschema_version\x18\x02 \x01(\tR\rschemaVersionB>Z pluggableharness.common.v1.CallContext 12, // 6: pluggableharness.model.v1.StreamCompletionRequest.cache_breakpoints:type_name -> pluggableharness.model.v1.CacheBreakpoint 6, // 7: pluggableharness.model.v1.StreamCompletionRequest.provider_options:type_name -> google.protobuf.Struct - 8, // [8:8] is the sub-list for method output_type - 8, // [8:8] is the sub-list for method input_type - 8, // [8:8] is the sub-list for extension type_name - 8, // [8:8] is the sub-list for extension extendee - 0, // [0:8] is the sub-list for field type_name + 7, // 8: pluggableharness.model.v1.CountTokensRequest.messages:type_name -> pluggableharness.content.v1.Message + 10, // 9: pluggableharness.model.v1.CountTokensRequest.assembled_context:type_name -> pluggableharness.content.v1.ContextSection + 8, // 10: pluggableharness.model.v1.CountTokensRequest.tools:type_name -> pluggableharness.model.v1.ToolDeclaration + 11, // [11:11] is the sub-list for method output_type + 11, // [11:11] is the sub-list for method input_type + 11, // [11:11] is the sub-list for extension type_name + 11, // [11:11] is the sub-list for extension extendee + 0, // [0:11] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_rpc_request_proto_init() } diff --git a/pkg/model/server.go b/pkg/model/server.go index e667361..b58d12c 100644 --- a/pkg/model/server.go +++ b/pkg/model/server.go @@ -106,7 +106,7 @@ func (svc *Service) CountTokens(ctx context.Context, req *modelv1.CountTokensReq if !ok { return nil, status.Error(codes.Unimplemented, "model: CountTokens not implemented by this provider") } - count, err := tc.CountTokens(ctx, req.GetText(), req.GetModelId()) + count, err := tc.CountTokens(ctx, req) if err != nil { return nil, statusFromErr(err) } diff --git a/pkg/model/server_test.go b/pkg/model/server_test.go index cd25d59..545817e 100644 --- a/pkg/model/server_test.go +++ b/pkg/model/server_test.go @@ -71,14 +71,16 @@ var _ model.Provider = (*fakeProvider)(nil) // unrelated fake types. type fakeTokenCounterProvider struct { fakeProvider - countTokensFunc func(ctx context.Context, text, modelID string) (int64, error) + countTokensFunc func(ctx context.Context, req *modelv1.CountTokensRequest) (int64, error) } -func (f *fakeTokenCounterProvider) CountTokens(ctx context.Context, text, modelID string) (int64, error) { +func (f *fakeTokenCounterProvider) CountTokens(ctx context.Context, req *modelv1.CountTokensRequest) (int64, error) { if f.countTokensFunc != nil { - return f.countTokensFunc(ctx, text, modelID) + return f.countTokensFunc(ctx, req) } - return int64(len(text)), nil + // A stand-in count that varies with the request, so a test asserting a + // number is asserting the request actually reached the provider. + return int64(len(req.GetMessages()) + len(req.GetTools())), nil } var _ model.TokenCounter = (*fakeTokenCounterProvider)(nil) @@ -405,12 +407,25 @@ func TestService_CountTokens_Implemented(t *testing.T) { t.Parallel() client := newTestClient(t, &fakeTokenCounterProvider{}) - resp, err := client.CountTokens(t.Context(), &modelv1.CountTokensRequest{Text: "hello world", ModelId: "fake-model"}) + + // Two messages and one tool: the fake counts both, so a wrong answer + // means the request's content did not survive the round trip. Counting + // tools at all is the point of the request-shaped RPC — the earlier + // text-only shape could not carry them. + req := &modelv1.CountTokensRequest{ + ModelId: "fake-model", + Messages: []*contentv1.Message{ + {Role: contentv1.Role_ROLE_USER}, + {Role: contentv1.Role_ROLE_ASSISTANT}, + }, + Tools: []*modelv1.ToolDeclaration{{Name: "read"}}, + } + resp, err := client.CountTokens(t.Context(), req) if err != nil { t.Fatalf("CountTokens() = %v, want nil error", err) } - if resp.GetCount() != int64(len("hello world")) { - t.Errorf("Count = %d, want %d", resp.GetCount(), len("hello world")) + if resp.GetCount() != 3 { + t.Errorf("Count = %d, want 3 (2 messages + 1 tool)", resp.GetCount()) } } @@ -418,7 +433,7 @@ func TestService_CountTokens_NotImplemented(t *testing.T) { t.Parallel() client := newTestClient(t, &fakeProvider{}) - _, err := client.CountTokens(t.Context(), &modelv1.CountTokensRequest{Text: "hi", ModelId: "fake-model"}) + _, err := client.CountTokens(t.Context(), &modelv1.CountTokensRequest{ModelId: "fake-model"}) if grpcstatus.Code(err) != codes.Unimplemented { t.Errorf("code = %v, want codes.Unimplemented", grpcstatus.Code(err)) } @@ -428,12 +443,12 @@ func TestService_CountTokens_ProviderError(t *testing.T) { t.Parallel() p := &fakeTokenCounterProvider{ - countTokensFunc: func(context.Context, string, string) (int64, error) { + countTokensFunc: func(context.Context, *modelv1.CountTokensRequest) (int64, error) { return 0, &model.Error{Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, Message: "unknown model"} }, } client := newTestClient(t, p) - _, err := client.CountTokens(t.Context(), &modelv1.CountTokensRequest{Text: "hi", ModelId: "nope"}) + _, err := client.CountTokens(t.Context(), &modelv1.CountTokensRequest{ModelId: "nope"}) if grpcstatus.Code(err) != codes.InvalidArgument { t.Errorf("code = %v, want codes.InvalidArgument", grpcstatus.Code(err)) } From 2b44a6bff1813ccc17506214f0ee87077144287e Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:24:18 -0400 Subject: [PATCH 05/16] model: add rate-limit snapshots and stream identity MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two things the kernel needs to read but had no field for. Usage.rate_limits reports the vendor's own rate-limit budgets. Without it the kernel cannot tell a user which ceiling stopped their session, and "you have 2% left" is unactionable without saying 2% of what. It is repeated because vendors publish several budgets at once and they exhaust independently — OpenAI and xAI return separate request and token headers, Anthropic reports input and output separately. Every numeric field is optional so an adapter reports the subset its vendor actually returned; synthesizing a snapshot from the adapter's own bookkeeping is forbidden, since a guessed budget is worse than none. StreamEvent.stream_start carries the vendor's request id. It is a separate early event rather than a field on stop because an id that arrives only on successful completion is absent in exactly the case it is needed — correlating a failure with the vendor's own logs. The Anthropic client emits stream_start from response headers, before any content streams. Reading a header rather than the body is what makes it safe to be best-effort: the spec permits omitting the event, so a header name that stops matching costs a correlation id, never a request. The e2e tier asserts the id is actually present, which is the only tier that can prove the vendor publishes it at all. A third field from this group is deliberately absent. GenerationParams gained no response_format: nothing in the kernel produces or consumes structured output today, so a typed field would be dead protocol surface and go-style.md forbids speculative generality. Until a caller exists it is provider_options territory, and promoting it to a typed field at that point is exactly the path proto.md's pass-through rule describes. --- api/pluggableharness/model/v1/events.proto | 18 ++ api/pluggableharness/model/v1/types.proto | 48 ++++ docs/specifications/model/conformance.md | 2 + docs/specifications/model/data-types.md | 15 +- internal/anthropic/messages/client.go | 31 +- internal/anthropic/messages/events.go | 2 + internal/anthropic/messages/events_test.go | 4 + internal/anthropic/provider_e2e_test.go | 46 ++- pkg/model/convert.go | 43 +++ pkg/model/convert_test.go | 64 +++++ pkg/model/model.go | 21 ++ pkg/model/proto/v1/events.pb.go | 199 +++++++++---- pkg/model/proto/v1/types.pb.go | 311 ++++++++++++++++----- pkg/model/stream.go | 17 ++ 14 files changed, 688 insertions(+), 133 deletions(-) diff --git a/api/pluggableharness/model/v1/events.proto b/api/pluggableharness/model/v1/events.proto index b0fb77f..c4ec378 100644 --- a/api/pluggableharness/model/v1/events.proto +++ b/api/pluggableharness/model/v1/events.proto @@ -34,6 +34,24 @@ message StreamEvent { // A complete, vendor-encrypted reasoning block the kernel cannot // interpret. RedactedThinking redacted_thinking = 10; + // The vendor has accepted the request and named it. + StreamStart stream_start = 11; + } + + // StreamStart carries the vendor's own identifier for this request, as + // soon as the adapter learns it — normally from response headers, + // before any content streams. + // + // It exists so a failure can be correlated with the vendor's own logs + // when asking them what went wrong. That is why it is a separate event + // emitted early rather than a field on Stop: an id that only arrives on + // successful completion is absent in exactly the case it is needed. + // MAY be omitted entirely by a vendor that publishes no such id. + message StreamStart { + // The vendor's request identifier, verbatim (an Anthropic + // `request-id` header, an OpenAI `x-request-id`). Opaque to the + // kernel: logged and surfaced, never parsed. + string provider_request_id = 1; } // TextDelta carries one incremental fragment of assistant text output. diff --git a/api/pluggableharness/model/v1/types.proto b/api/pluggableharness/model/v1/types.proto index a9cb706..3d1e000 100644 --- a/api/pluggableharness/model/v1/types.proto +++ b/api/pluggableharness/model/v1/types.proto @@ -489,11 +489,59 @@ message Usage { // future Pricing revision declares a distinct reasoning rate — there is // none as of this revision. optional int64 reasoning_tokens = 5; + + // The vendor's own rate-limit state as of this completion, when it + // reports any. MAY be empty — a vendor that publishes nothing has + // nothing to declare, and an adapter MUST NOT synthesize a snapshot. + // + // Repeated because vendors publish several budgets at once and they + // exhaust independently: OpenAI and xAI return separate request and + // token headers, Anthropic reports input and output separately. Naming + // which budget is close to empty is the whole point — "you have 2% + // left" is unactionable without saying 2% of what. + repeated RateLimitSnapshot rate_limits = 6; + // Deliberately no cost field: the kernel computes and persists // cost_usd from these token counts plus the matching PricingTier, per // model.md §4.1 — the provider never computes cost itself. } +// RateLimitKind names which vendor budget a RateLimitSnapshot describes. +enum RateLimitKind { + // Zero value. Never valid on a real snapshot; its presence on the wire + // means an adapter forgot to set the field. + RATE_LIMIT_KIND_UNSPECIFIED = 0; + // Requests per window. + RATE_LIMIT_KIND_REQUESTS = 1; + // Tokens per window, undifferentiated by direction. + RATE_LIMIT_KIND_TOKENS = 2; + // Input tokens per window, where the vendor meters them separately. + RATE_LIMIT_KIND_INPUT_TOKENS = 3; + // Output tokens per window, where the vendor meters them separately. + RATE_LIMIT_KIND_OUTPUT_TOKENS = 4; +} + +// RateLimitSnapshot is one of the vendor's rate-limit budgets as of one +// completion, per model/data-types.md#streamevent. +// +// Every numeric field is optional because vendors publish different +// subsets: an adapter reports what its vendor actually returned and omits +// the rest rather than inventing a value. A snapshot with only `kind` set +// is still useful — it says the budget exists. +message RateLimitSnapshot { + // Which budget this describes. MUST be set. + RateLimitKind kind = 1; + + // How much of this budget remains. + optional int64 remaining = 2; + + // This budget's ceiling for the current window. + optional int64 limit = 3; + + // When this budget next resets. + optional google.protobuf.Timestamp reset_at = 4; +} + // ModelTarget describes the model a context or memory contribution is // being assembled for, derived from that model's ModelSpec // (model.md §2). Carried on context.md's ContextRequest and memory.md's diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index e75abda..eaec450 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -45,6 +45,8 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | `Pricing.tiers`, time-bounded/tiered/input-size-bounded rates | MUST | [`data-types.md`](data-types.md#pricing) — exactly one tier MUST match any given `(timestamp, input_token_count)` pair | | `Pricing` on every `ModelSpec` | MUST | required even for `free: true` models | | Kernel computes + persists `cost_usd` at usage-event time, not lazily at query time | MUST | [`protocol.md`](protocol.md#cost-computation) — includes `reasoning_tokens` billed at the output rate | +| `StreamEvent.stream_start` | SHOULD, when the vendor publishes a request id | [`data-types.md#stream_start-and-vendor-request-correlation`](data-types.md#stream_start-and-vendor-request-correlation) — emitted early, so a failed stream is still correlatable to the vendor's logs | +| `Usage.rate_limits` | SHOULD, when the vendor publishes rate-limit state | [`data-types.md#usagerate_limits`](data-types.md#usagerate_limits) — MUST NOT be synthesized from the adapter's own bookkeeping | | `Usage.reasoning_tokens` | SHOULD, when the vendor reports it distinctly | [`data-types.md#streamevent`](data-types.md#streamevent) — never double-counted in `output_tokens` | | `supported_hook_points` | MUST (field, MAY be empty) | [`data-types.md#capabilitiessupported_hook_points`](data-types.md#capabilitiessupported_hook_points) — kernel rejects an unsupported `hook{}` block at config-load time | | Realtime/voice (WebSocket-style APIs) | MUST NOT — out of scope for v1 | likely a distinct wire protocol per vendor; treat as a future, separate plugin surface, not a mode of this one | diff --git a/docs/specifications/model/data-types.md b/docs/specifications/model/data-types.md index 9aa9b79..ae85f48 100644 --- a/docs/specifications/model/data-types.md +++ b/docs/specifications/model/data-types.md @@ -140,6 +140,7 @@ The full shape of what `StreamCompletion` streams back — see [`protocol.md#str ```protobuf StreamEvent = oneof { + stream_start { provider_request_id: string } // MAY — see below text_delta { text: string } thinking_delta { text: string } // only when ThinkingSpec.supported thinking_signature { signature: bytes } // see "Canonical message" below — @@ -152,7 +153,7 @@ StreamEvent = oneof { tool_call_start { id: string, name: string } tool_call_delta { id: string, arguments_fragment: string } // partial-JSON accumulation tool_call_done { id: string } - usage { input_tokens, output_tokens, cache_read_tokens?, cache_write_tokens?, reasoning_tokens? } + usage { input_tokens, output_tokens, cache_read_tokens?, cache_write_tokens?, reasoning_tokens?, rate_limits[] } stop { reason: StopReason, matched_stop_sequence?: string } error ModelError // see conformance.md#error-taxonomy } @@ -177,6 +178,18 @@ StopReason = enum { `usage.reasoning_tokens` is set only when the vendor reports thinking/reasoning tokens as a distinct count (`ThinkingSpec.supported` models only) and is never also counted in `output_tokens` — a vendor that folds reasoning tokens into its reported `output_tokens` has no separate figure to report, so this stays unset rather than being derived or subtracted. It's billed at `PricingTier.output_per_mtok` unless a future `Pricing` revision declares a distinct reasoning rate; there is none as of this revision. +### `stream_start` and vendor request correlation + +`stream_start` carries the vendor's own identifier for this request — an Anthropic `request-id` header, an OpenAI `x-request-id` — as soon as the adapter learns it, normally from response headers before any content streams. It exists so a failure can be correlated with the vendor's own logs when asking them what went wrong. A plugin whose vendor publishes no such id MAY omit the event entirely. + +It is a separate early event rather than a field on `stop` deliberately: an id that arrives only on successful completion is absent in exactly the case it is needed. The kernel treats the value as opaque — logged and surfaced, never parsed. + +### `usage.rate_limits` + +`rate_limits` reports the vendor's own rate-limit budgets as of this completion. It MAY be empty; a vendor that publishes nothing has nothing to declare, and an adapter MUST NOT synthesize a snapshot from its own bookkeeping. + +It is repeated because vendors publish several budgets at once and they exhaust independently — OpenAI and xAI return separate request and token headers, Anthropic reports input and output separately. Naming *which* budget is close to empty is the entire value: "you have 2% left" is unactionable without saying 2% of what, and a user whose session stops mid-task without being told which ceiling they hit cannot act on it. Every numeric field on a snapshot is optional, so an adapter reports the subset its vendor actually returned rather than inventing the rest. + A plugin MUST classify every terminal failure via a `stop` event's `content_filtered` reason or an `error` event carrying a `ModelError` ([`conformance.md#error-taxonomy`](conformance.md#error-taxonomy)) — the in-band `error` variant is how a plugin reports a classified failure *within* an otherwise-open stream, distinct from the stream simply being torn down at the transport level (a gRPC-level status, or the kernel closing the stream on cancellation). A plugin whose backend fails outright before producing any events MAY end the stream with just an `error` event and no preceding `stop`. ## Canonical message & content-block schema diff --git a/internal/anthropic/messages/client.go b/internal/anthropic/messages/client.go index fd21ebf..77cb682 100644 --- a/internal/anthropic/messages/client.go +++ b/internal/anthropic/messages/client.go @@ -150,7 +150,8 @@ func (c *Client) Stream(ctx context.Context, req *Request, sink EventSink) error } defer func() { _ = resp.Body.Close() }() - c.logger.DebugContext(ctx, "anthropic: stream: response", "status", resp.StatusCode) + requestID := providerRequestID(resp.Header) + c.logger.DebugContext(ctx, "anthropic: stream: response", "status", resp.StatusCode, "provider_request_id", requestID) if resp.StatusCode < http.StatusOK || resp.StatusCode >= http.StatusMultipleChoices { respBody, _ := io.ReadAll(io.LimitReader(resp.Body, maxErrorResponseBytes)) @@ -159,6 +160,16 @@ func (c *Client) Stream(ctx context.Context, req *Request, sink EventSink) error return modelErr } + // Emitted before any content so a stream that fails midway is still + // correlatable to the vendor's own logs — the whole reason stream_start + // is an early event rather than a field on stop + // (model/data-types.md#stream_start-and-vendor-request-correlation). + if requestID != "" { + if err := sink.StreamStart(requestID); err != nil { + return err + } + } + return c.drive(ctx, resp.Body, sink) } @@ -199,6 +210,24 @@ func (c *Client) drive(ctx context.Context, body io.Reader, sink EventSink) erro return sink.Error(truncated) } +// providerRequestID extracts the vendor's request identifier from a +// response's headers, or "" when the vendor published none. +// +// Two header names are tried because Anthropic has used both spellings +// across its API surface, and this is best-effort by design: the spec +// permits omitting stream_start entirely, so a header name that stops +// matching costs a correlation id, never a failed request. That is why +// this reads headers rather than parsing the response body — a missing +// header must be a non-event. +func providerRequestID(h http.Header) string { + for _, name := range []string{"request-id", "x-request-id"} { + if v := h.Get(name); v != "" { + return v + } + } + return "" +} + // CountTokens returns an exact input-token count for req against // POST /v1/messages/count_tokens. // diff --git a/internal/anthropic/messages/events.go b/internal/anthropic/messages/events.go index 10d60fe..20b2ab6 100644 --- a/internal/anthropic/messages/events.go +++ b/internal/anthropic/messages/events.go @@ -9,6 +9,8 @@ import ( // so Translator can be tested against a recording fake without a live gRPC // stream; *model.Sink satisfies it structurally. type EventSink interface { + // StreamStart sends the vendor's own identifier for this request. + StreamStart(providerRequestID string) error // TextDelta sends an incremental fragment of assistant text output. TextDelta(text string) error // ThinkingDelta sends an incremental fragment of the model's reasoning diff --git a/internal/anthropic/messages/events_test.go b/internal/anthropic/messages/events_test.go index 6fa6d85..888de0a 100644 --- a/internal/anthropic/messages/events_test.go +++ b/internal/anthropic/messages/events_test.go @@ -43,6 +43,10 @@ func (f *fakeSink) record(method string, args ...any) error { return f.failAt[method] } +func (f *fakeSink) StreamStart(providerRequestID string) error { + return f.record("StreamStart", providerRequestID) +} + func (f *fakeSink) TextDelta(text string) error { return f.record("TextDelta", text) } func (f *fakeSink) ThinkingDelta(text string) error { return f.record("ThinkingDelta", text) } diff --git a/internal/anthropic/provider_e2e_test.go b/internal/anthropic/provider_e2e_test.go index e96c33b..5fed6a2 100644 --- a/internal/anthropic/provider_e2e_test.go +++ b/internal/anthropic/provider_e2e_test.go @@ -130,6 +130,12 @@ func TestLive_streamCompletion(t *testing.T) { if err := sink.streamError(); err != nil { t.Errorf("the live stream reported an in-band error: %v", err) } + // Only the live tier can prove the vendor actually publishes the + // header stream_start is built on — a recorded transcript would only + // confirm we agree with our own past reading of the docs. + if sink.providerRequestID() == "" { + t.Error("the live stream produced no provider request id — nothing to correlate a failure against") + } } // TestLive_countTokens proves the tokenizer endpoint still answers in the @@ -142,7 +148,19 @@ func TestLive_countTokens(t *testing.T) { ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second) defer cancel() - count, err := client.CountTokens(ctx, "The quick brown fox jumps over the lazy dog.", liveModelID) + req := &modelv1.CountTokensRequest{ + ModelId: liveModelID, + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{{ + Block: &contentv1.ContentBlock_Text{ + Text: &contentv1.TextBlock{Text: "The quick brown fox jumps over the lazy dog."}, + }, + }}, + }}, + } + + count, err := client.CountTokens(ctx, req, liveSpec(t)) if err != nil { t.Fatalf("CountTokens against the live API: %v", err) } @@ -155,15 +173,25 @@ func TestLive_countTokens(t *testing.T) { // than generated, and mutex-guarded because nothing promises the client // drives it from the calling goroutine. type liveSink struct { - mu sync.Mutex - textBuf strings.Builder - usage *model.Usage - stop bool - err *model.Error + mu sync.Mutex + textBuf strings.Builder + usage *model.Usage + stop bool + err *model.Error + requestID string } var _ messages.EventSink = (*liveSink)(nil) +// StreamStart records the vendor's request id, which a live-run failure +// report can quote when asking Anthropic about a specific request. +func (s *liveSink) StreamStart(providerRequestID string) error { + s.mu.Lock() + defer s.mu.Unlock() + s.requestID = providerRequestID + return nil +} + func (s *liveSink) TextDelta(text string) error { s.mu.Lock() defer s.mu.Unlock() @@ -199,6 +227,12 @@ func (s *liveSink) Error(modelErr *model.Error) error { return nil } +func (s *liveSink) providerRequestID() string { + s.mu.Lock() + defer s.mu.Unlock() + return s.requestID +} + func (s *liveSink) text() string { s.mu.Lock() defer s.mu.Unlock() diff --git a/pkg/model/convert.go b/pkg/model/convert.go index 85f9f36..9daab40 100644 --- a/pkg/model/convert.go +++ b/pkg/model/convert.go @@ -251,9 +251,51 @@ func usageToProto(u Usage) *modelv1.Usage { CacheReadTokens: u.CacheReadTokens, CacheWriteTokens: u.CacheWriteTokens, ReasoningTokens: u.ReasoningTokens, + RateLimits: rateLimitsToProto(u.RateLimits), } } +// rateLimitsToProto converts each snapshot into the generated wire type. +func rateLimitsToProto(in []RateLimitSnapshot) []*modelv1.RateLimitSnapshot { + if len(in) == 0 { + return nil + } + out := make([]*modelv1.RateLimitSnapshot, len(in)) + for i, r := range in { + snap := &modelv1.RateLimitSnapshot{ + Kind: r.Kind, + Remaining: r.Remaining, + Limit: r.Limit, + } + if r.ResetAt != nil { + snap.ResetAt = timestamppb.New(*r.ResetAt) + } + out[i] = snap + } + return out +} + +// rateLimitsFromProto is rateLimitsToProto's inverse. +func rateLimitsFromProto(in []*modelv1.RateLimitSnapshot) []RateLimitSnapshot { + if len(in) == 0 { + return nil + } + out := make([]RateLimitSnapshot, len(in)) + for i, r := range in { + snap := RateLimitSnapshot{ + Kind: r.GetKind(), + Remaining: r.Remaining, + Limit: r.Limit, + } + if ts := r.GetResetAt(); ts != nil { + at := ts.AsTime() + snap.ResetAt = &at + } + out[i] = snap + } + return out +} + // usageFromProto is usageToProto's inverse. func usageFromProto(in *modelv1.Usage) Usage { if in == nil { @@ -265,5 +307,6 @@ func usageFromProto(in *modelv1.Usage) Usage { CacheReadTokens: in.CacheReadTokens, CacheWriteTokens: in.CacheWriteTokens, ReasoningTokens: in.ReasoningTokens, + RateLimits: rateLimitsFromProto(in.GetRateLimits()), } } diff --git a/pkg/model/convert_test.go b/pkg/model/convert_test.go index f31a2c8..768faf3 100644 --- a/pkg/model/convert_test.go +++ b/pkg/model/convert_test.go @@ -399,3 +399,67 @@ func TestConvert_ModelErrorFromProtoNil(t *testing.T) { t.Errorf("ModelErrorFromProtoForTest(nil) = %+v, want nil", got) } } + +func TestConvert_UsageRateLimitsRoundTrip(t *testing.T) { + t.Parallel() + + reset := time.Date(2026, 7, 27, 12, 0, 0, 0, time.UTC) + remaining := int64(2) + limit := int64(1000) + + // Two budgets at once, which is the normal case rather than an edge: + // vendors meter requests and tokens separately and they exhaust + // independently, so a single snapshot could not say which one stopped + // a session. + in := model.Usage{ + InputTokens: 10, + OutputTokens: 5, + RateLimits: []model.RateLimitSnapshot{ + { + Kind: modelv1.RateLimitKind_RATE_LIMIT_KIND_REQUESTS, + Remaining: &remaining, + Limit: &limit, + ResetAt: &reset, + }, + // Only Kind set: a vendor that publishes the budget's existence + // but no numbers still produces a useful snapshot, and the + // adapter must not invent the missing values. + {Kind: modelv1.RateLimitKind_RATE_LIMIT_KIND_OUTPUT_TOKENS}, + }, + } + + back := model.UsageFromProtoForTest(model.UsageToProtoForTest(in)) + + if len(back.RateLimits) != 2 { + t.Fatalf("len(RateLimits) = %d, want 2", len(back.RateLimits)) + } + first := back.RateLimits[0] + if first.Kind != modelv1.RateLimitKind_RATE_LIMIT_KIND_REQUESTS { + t.Errorf("RateLimits[0].Kind = %v, want REQUESTS", first.Kind) + } + if first.Remaining == nil || *first.Remaining != remaining { + t.Errorf("RateLimits[0].Remaining = %v, want %d", first.Remaining, remaining) + } + if first.ResetAt == nil || !first.ResetAt.Equal(reset) { + t.Errorf("RateLimits[0].ResetAt = %v, want %v", first.ResetAt, reset) + } + + second := back.RateLimits[1] + if second.Kind != modelv1.RateLimitKind_RATE_LIMIT_KIND_OUTPUT_TOKENS { + t.Errorf("RateLimits[1].Kind = %v, want OUTPUT_TOKENS", second.Kind) + } + if second.Remaining != nil || second.Limit != nil || second.ResetAt != nil { + t.Errorf("RateLimits[1] = %+v, want every numeric field absent", second) + } +} + +func TestConvert_UsageWithoutRateLimitsStaysNil(t *testing.T) { + t.Parallel() + + // A vendor publishing nothing must produce no snapshots at all, rather + // than an empty-but-present one that would read as "the budget exists". + back := model.UsageFromProtoForTest(model.UsageToProtoForTest(model.Usage{InputTokens: 1})) + if back.RateLimits != nil { + t.Errorf("RateLimits = %+v, want nil", back.RateLimits) + } +} diff --git a/pkg/model/model.go b/pkg/model/model.go index bb016de..9dd5f0f 100644 --- a/pkg/model/model.go +++ b/pkg/model/model.go @@ -335,4 +335,25 @@ type Usage struct { // reports them as a distinct count. Never also counted in // OutputTokens; billed at PricingTier.OutputPerMtok. ReasoningTokens *int64 + // RateLimits is the vendor's own rate-limit state as of this + // completion. MAY be empty; a Provider MUST NOT synthesize a snapshot + // from its own bookkeeping — only report what the vendor published. + RateLimits []RateLimitSnapshot +} + +// RateLimitSnapshot is one of the vendor's rate-limit budgets as of one +// completion, per docs/specifications/model/data-types.md#usagerate_limits. +// +// Every field but Kind is a pointer because vendors publish different +// subsets, and "the vendor did not say" is meaningfully different from +// "the vendor said zero" — the second means the budget is exhausted. +type RateLimitSnapshot struct { + // Kind names which budget this describes. MUST be set. + Kind modelv1.RateLimitKind + // Remaining is how much of this budget is left. + Remaining *int64 + // Limit is this budget's ceiling for the current window. + Limit *int64 + // ResetAt is when this budget next resets. + ResetAt *time.Time } diff --git a/pkg/model/proto/v1/events.pb.go b/pkg/model/proto/v1/events.pb.go index 74c126f..fd49255 100644 --- a/pkg/model/proto/v1/events.pb.go +++ b/pkg/model/proto/v1/events.pb.go @@ -119,6 +119,7 @@ type StreamEvent struct { // *StreamEvent_Stop_ // *StreamEvent_Error_ // *StreamEvent_RedactedThinking_ + // *StreamEvent_StreamStart_ Event isStreamEvent_Event `protobuf_oneof:"event"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache @@ -251,6 +252,15 @@ func (x *StreamEvent) GetRedactedThinking() *StreamEvent_RedactedThinking { return nil } +func (x *StreamEvent) GetStreamStart() *StreamEvent_StreamStart { + if x != nil { + if x, ok := x.Event.(*StreamEvent_StreamStart_); ok { + return x.StreamStart + } + } + return nil +} + type isStreamEvent_Event interface { isStreamEvent_Event() } @@ -306,6 +316,11 @@ type StreamEvent_RedactedThinking_ struct { RedactedThinking *StreamEvent_RedactedThinking `protobuf:"bytes,10,opt,name=redacted_thinking,json=redactedThinking,proto3,oneof"` } +type StreamEvent_StreamStart_ struct { + // The vendor has accepted the request and named it. + StreamStart *StreamEvent_StreamStart `protobuf:"bytes,11,opt,name=stream_start,json=streamStart,proto3,oneof"` +} + func (*StreamEvent_TextDelta_) isStreamEvent_Event() {} func (*StreamEvent_ThinkingDelta_) isStreamEvent_Event() {} @@ -326,6 +341,64 @@ func (*StreamEvent_Error_) isStreamEvent_Event() {} func (*StreamEvent_RedactedThinking_) isStreamEvent_Event() {} +func (*StreamEvent_StreamStart_) isStreamEvent_Event() {} + +// StreamStart carries the vendor's own identifier for this request, as +// soon as the adapter learns it — normally from response headers, +// before any content streams. +// +// It exists so a failure can be correlated with the vendor's own logs +// when asking them what went wrong. That is why it is a separate event +// emitted early rather than a field on Stop: an id that only arrives on +// successful completion is absent in exactly the case it is needed. +// MAY be omitted entirely by a vendor that publishes no such id. +type StreamEvent_StreamStart struct { + state protoimpl.MessageState `protogen:"open.v1"` + // The vendor's request identifier, verbatim (an Anthropic + // `request-id` header, an OpenAI `x-request-id`). Opaque to the + // kernel: logged and surfaced, never parsed. + ProviderRequestId string `protobuf:"bytes,1,opt,name=provider_request_id,json=providerRequestId,proto3" json:"provider_request_id,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *StreamEvent_StreamStart) Reset() { + *x = StreamEvent_StreamStart{} + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[1] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *StreamEvent_StreamStart) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*StreamEvent_StreamStart) ProtoMessage() {} + +func (x *StreamEvent_StreamStart) ProtoReflect() protoreflect.Message { + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[1] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use StreamEvent_StreamStart.ProtoReflect.Descriptor instead. +func (*StreamEvent_StreamStart) Descriptor() ([]byte, []int) { + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 0} +} + +func (x *StreamEvent_StreamStart) GetProviderRequestId() string { + if x != nil { + return x.ProviderRequestId + } + return "" +} + // TextDelta carries one incremental fragment of assistant text output. // MUST be supported by every plugin, both directions (model.md §5). type StreamEvent_TextDelta struct { @@ -338,7 +411,7 @@ type StreamEvent_TextDelta struct { func (x *StreamEvent_TextDelta) Reset() { *x = StreamEvent_TextDelta{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[1] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[2] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -350,7 +423,7 @@ func (x *StreamEvent_TextDelta) String() string { func (*StreamEvent_TextDelta) ProtoMessage() {} func (x *StreamEvent_TextDelta) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[1] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[2] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -363,7 +436,7 @@ func (x *StreamEvent_TextDelta) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_TextDelta.ProtoReflect.Descriptor instead. func (*StreamEvent_TextDelta) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 0} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 1} } func (x *StreamEvent_TextDelta) GetText() string { @@ -386,7 +459,7 @@ type StreamEvent_ThinkingDelta struct { func (x *StreamEvent_ThinkingDelta) Reset() { *x = StreamEvent_ThinkingDelta{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[2] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[3] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -398,7 +471,7 @@ func (x *StreamEvent_ThinkingDelta) String() string { func (*StreamEvent_ThinkingDelta) ProtoMessage() {} func (x *StreamEvent_ThinkingDelta) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[2] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[3] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -411,7 +484,7 @@ func (x *StreamEvent_ThinkingDelta) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_ThinkingDelta.ProtoReflect.Descriptor instead. func (*StreamEvent_ThinkingDelta) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 1} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 2} } func (x *StreamEvent_ThinkingDelta) GetText() string { @@ -436,7 +509,7 @@ type StreamEvent_ThinkingSignature struct { func (x *StreamEvent_ThinkingSignature) Reset() { *x = StreamEvent_ThinkingSignature{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[3] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[4] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -448,7 +521,7 @@ func (x *StreamEvent_ThinkingSignature) String() string { func (*StreamEvent_ThinkingSignature) ProtoMessage() {} func (x *StreamEvent_ThinkingSignature) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[3] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[4] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -461,7 +534,7 @@ func (x *StreamEvent_ThinkingSignature) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_ThinkingSignature.ProtoReflect.Descriptor instead. func (*StreamEvent_ThinkingSignature) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 2} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 3} } func (x *StreamEvent_ThinkingSignature) GetSignature() []byte { @@ -486,7 +559,7 @@ type StreamEvent_ToolCallStart struct { func (x *StreamEvent_ToolCallStart) Reset() { *x = StreamEvent_ToolCallStart{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[4] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -498,7 +571,7 @@ func (x *StreamEvent_ToolCallStart) String() string { func (*StreamEvent_ToolCallStart) ProtoMessage() {} func (x *StreamEvent_ToolCallStart) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[4] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[5] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -511,7 +584,7 @@ func (x *StreamEvent_ToolCallStart) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_ToolCallStart.ProtoReflect.Descriptor instead. func (*StreamEvent_ToolCallStart) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 3} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 4} } func (x *StreamEvent_ToolCallStart) GetId() string { @@ -543,7 +616,7 @@ type StreamEvent_ToolCallDelta struct { func (x *StreamEvent_ToolCallDelta) Reset() { *x = StreamEvent_ToolCallDelta{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[5] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[6] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -555,7 +628,7 @@ func (x *StreamEvent_ToolCallDelta) String() string { func (*StreamEvent_ToolCallDelta) ProtoMessage() {} func (x *StreamEvent_ToolCallDelta) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[5] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[6] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -568,7 +641,7 @@ func (x *StreamEvent_ToolCallDelta) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_ToolCallDelta.ProtoReflect.Descriptor instead. func (*StreamEvent_ToolCallDelta) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 4} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 5} } func (x *StreamEvent_ToolCallDelta) GetId() string { @@ -597,7 +670,7 @@ type StreamEvent_ToolCallDone struct { func (x *StreamEvent_ToolCallDone) Reset() { *x = StreamEvent_ToolCallDone{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[6] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[7] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -609,7 +682,7 @@ func (x *StreamEvent_ToolCallDone) String() string { func (*StreamEvent_ToolCallDone) ProtoMessage() {} func (x *StreamEvent_ToolCallDone) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[6] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[7] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -622,7 +695,7 @@ func (x *StreamEvent_ToolCallDone) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_ToolCallDone.ProtoReflect.Descriptor instead. func (*StreamEvent_ToolCallDone) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 5} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 6} } func (x *StreamEvent_ToolCallDone) GetId() string { @@ -647,7 +720,7 @@ type StreamEvent_Stop struct { func (x *StreamEvent_Stop) Reset() { *x = StreamEvent_Stop{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[7] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -659,7 +732,7 @@ func (x *StreamEvent_Stop) String() string { func (*StreamEvent_Stop) ProtoMessage() {} func (x *StreamEvent_Stop) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[7] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[8] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -672,7 +745,7 @@ func (x *StreamEvent_Stop) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_Stop.ProtoReflect.Descriptor instead. func (*StreamEvent_Stop) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 6} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 7} } func (x *StreamEvent_Stop) GetReason() StopReason { @@ -700,7 +773,7 @@ type StreamEvent_Error struct { func (x *StreamEvent_Error) Reset() { *x = StreamEvent_Error{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[8] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[9] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -712,7 +785,7 @@ func (x *StreamEvent_Error) String() string { func (*StreamEvent_Error) ProtoMessage() {} func (x *StreamEvent_Error) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[8] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[9] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -725,7 +798,7 @@ func (x *StreamEvent_Error) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_Error.ProtoReflect.Descriptor instead. func (*StreamEvent_Error) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 7} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 8} } func (x *StreamEvent_Error) GetError() *ModelError { @@ -754,7 +827,7 @@ type StreamEvent_RedactedThinking struct { func (x *StreamEvent_RedactedThinking) Reset() { *x = StreamEvent_RedactedThinking{} - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[9] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[10] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -766,7 +839,7 @@ func (x *StreamEvent_RedactedThinking) String() string { func (*StreamEvent_RedactedThinking) ProtoMessage() {} func (x *StreamEvent_RedactedThinking) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_events_proto_msgTypes[9] + mi := &file_pluggableharness_model_v1_events_proto_msgTypes[10] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -779,7 +852,7 @@ func (x *StreamEvent_RedactedThinking) ProtoReflect() protoreflect.Message { // Deprecated: Use StreamEvent_RedactedThinking.ProtoReflect.Descriptor instead. func (*StreamEvent_RedactedThinking) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 8} + return file_pluggableharness_model_v1_events_proto_rawDescGZIP(), []int{0, 9} } func (x *StreamEvent_RedactedThinking) GetData() []byte { @@ -793,7 +866,7 @@ var File_pluggableharness_model_v1_events_proto protoreflect.FileDescriptor const file_pluggableharness_model_v1_events_proto_rawDesc = "" + "\n" + - "&pluggableharness/model/v1/events.proto\x12\x19pluggableharness.model.v1\x1a&pluggableharness/model/v1/errors.proto\x1a%pluggableharness/model/v1/types.proto\"\xa2\v\n" + + "&pluggableharness/model/v1/events.proto\x12\x19pluggableharness.model.v1\x1a&pluggableharness/model/v1/errors.proto\x1a%pluggableharness/model/v1/types.proto\"\xba\f\n" + "\vStreamEvent\x12Q\n" + "\n" + "text_delta\x18\x01 \x01(\v20.pluggableharness.model.v1.StreamEvent.TextDeltaH\x00R\ttextDelta\x12]\n" + @@ -806,7 +879,10 @@ const file_pluggableharness_model_v1_events_proto_rawDesc = "" + "\x04stop\x18\b \x01(\v2+.pluggableharness.model.v1.StreamEvent.StopH\x00R\x04stop\x12D\n" + "\x05error\x18\t \x01(\v2,.pluggableharness.model.v1.StreamEvent.ErrorH\x00R\x05error\x12f\n" + "\x11redacted_thinking\x18\n" + - " \x01(\v27.pluggableharness.model.v1.StreamEvent.RedactedThinkingH\x00R\x10redactedThinking\x1a\x1f\n" + + " \x01(\v27.pluggableharness.model.v1.StreamEvent.RedactedThinkingH\x00R\x10redactedThinking\x12W\n" + + "\fstream_start\x18\v \x01(\v22.pluggableharness.model.v1.StreamEvent.StreamStartH\x00R\vstreamStart\x1a=\n" + + "\vStreamStart\x12.\n" + + "\x13provider_request_id\x18\x01 \x01(\tR\x11providerRequestId\x1a\x1f\n" + "\tTextDelta\x12\x12\n" + "\x04text\x18\x01 \x01(\tR\x04text\x1a#\n" + "\rThinkingDelta\x12\x12\n" + @@ -854,40 +930,42 @@ func file_pluggableharness_model_v1_events_proto_rawDescGZIP() []byte { } var file_pluggableharness_model_v1_events_proto_enumTypes = make([]protoimpl.EnumInfo, 1) -var file_pluggableharness_model_v1_events_proto_msgTypes = make([]protoimpl.MessageInfo, 10) +var file_pluggableharness_model_v1_events_proto_msgTypes = make([]protoimpl.MessageInfo, 11) var file_pluggableharness_model_v1_events_proto_goTypes = []any{ (StopReason)(0), // 0: pluggableharness.model.v1.StopReason (*StreamEvent)(nil), // 1: pluggableharness.model.v1.StreamEvent - (*StreamEvent_TextDelta)(nil), // 2: pluggableharness.model.v1.StreamEvent.TextDelta - (*StreamEvent_ThinkingDelta)(nil), // 3: pluggableharness.model.v1.StreamEvent.ThinkingDelta - (*StreamEvent_ThinkingSignature)(nil), // 4: pluggableharness.model.v1.StreamEvent.ThinkingSignature - (*StreamEvent_ToolCallStart)(nil), // 5: pluggableharness.model.v1.StreamEvent.ToolCallStart - (*StreamEvent_ToolCallDelta)(nil), // 6: pluggableharness.model.v1.StreamEvent.ToolCallDelta - (*StreamEvent_ToolCallDone)(nil), // 7: pluggableharness.model.v1.StreamEvent.ToolCallDone - (*StreamEvent_Stop)(nil), // 8: pluggableharness.model.v1.StreamEvent.Stop - (*StreamEvent_Error)(nil), // 9: pluggableharness.model.v1.StreamEvent.Error - (*StreamEvent_RedactedThinking)(nil), // 10: pluggableharness.model.v1.StreamEvent.RedactedThinking - (*Usage)(nil), // 11: pluggableharness.model.v1.Usage - (*ModelError)(nil), // 12: pluggableharness.model.v1.ModelError + (*StreamEvent_StreamStart)(nil), // 2: pluggableharness.model.v1.StreamEvent.StreamStart + (*StreamEvent_TextDelta)(nil), // 3: pluggableharness.model.v1.StreamEvent.TextDelta + (*StreamEvent_ThinkingDelta)(nil), // 4: pluggableharness.model.v1.StreamEvent.ThinkingDelta + (*StreamEvent_ThinkingSignature)(nil), // 5: pluggableharness.model.v1.StreamEvent.ThinkingSignature + (*StreamEvent_ToolCallStart)(nil), // 6: pluggableharness.model.v1.StreamEvent.ToolCallStart + (*StreamEvent_ToolCallDelta)(nil), // 7: pluggableharness.model.v1.StreamEvent.ToolCallDelta + (*StreamEvent_ToolCallDone)(nil), // 8: pluggableharness.model.v1.StreamEvent.ToolCallDone + (*StreamEvent_Stop)(nil), // 9: pluggableharness.model.v1.StreamEvent.Stop + (*StreamEvent_Error)(nil), // 10: pluggableharness.model.v1.StreamEvent.Error + (*StreamEvent_RedactedThinking)(nil), // 11: pluggableharness.model.v1.StreamEvent.RedactedThinking + (*Usage)(nil), // 12: pluggableharness.model.v1.Usage + (*ModelError)(nil), // 13: pluggableharness.model.v1.ModelError } var file_pluggableharness_model_v1_events_proto_depIdxs = []int32{ - 2, // 0: pluggableharness.model.v1.StreamEvent.text_delta:type_name -> pluggableharness.model.v1.StreamEvent.TextDelta - 3, // 1: pluggableharness.model.v1.StreamEvent.thinking_delta:type_name -> pluggableharness.model.v1.StreamEvent.ThinkingDelta - 4, // 2: pluggableharness.model.v1.StreamEvent.thinking_signature:type_name -> pluggableharness.model.v1.StreamEvent.ThinkingSignature - 5, // 3: pluggableharness.model.v1.StreamEvent.tool_call_start:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallStart - 6, // 4: pluggableharness.model.v1.StreamEvent.tool_call_delta:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallDelta - 7, // 5: pluggableharness.model.v1.StreamEvent.tool_call_done:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallDone - 11, // 6: pluggableharness.model.v1.StreamEvent.usage:type_name -> pluggableharness.model.v1.Usage - 8, // 7: pluggableharness.model.v1.StreamEvent.stop:type_name -> pluggableharness.model.v1.StreamEvent.Stop - 9, // 8: pluggableharness.model.v1.StreamEvent.error:type_name -> pluggableharness.model.v1.StreamEvent.Error - 10, // 9: pluggableharness.model.v1.StreamEvent.redacted_thinking:type_name -> pluggableharness.model.v1.StreamEvent.RedactedThinking - 0, // 10: pluggableharness.model.v1.StreamEvent.Stop.reason:type_name -> pluggableharness.model.v1.StopReason - 12, // 11: pluggableharness.model.v1.StreamEvent.Error.error:type_name -> pluggableharness.model.v1.ModelError - 12, // [12:12] is the sub-list for method output_type - 12, // [12:12] is the sub-list for method input_type - 12, // [12:12] is the sub-list for extension type_name - 12, // [12:12] is the sub-list for extension extendee - 0, // [0:12] is the sub-list for field type_name + 3, // 0: pluggableharness.model.v1.StreamEvent.text_delta:type_name -> pluggableharness.model.v1.StreamEvent.TextDelta + 4, // 1: pluggableharness.model.v1.StreamEvent.thinking_delta:type_name -> pluggableharness.model.v1.StreamEvent.ThinkingDelta + 5, // 2: pluggableharness.model.v1.StreamEvent.thinking_signature:type_name -> pluggableharness.model.v1.StreamEvent.ThinkingSignature + 6, // 3: pluggableharness.model.v1.StreamEvent.tool_call_start:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallStart + 7, // 4: pluggableharness.model.v1.StreamEvent.tool_call_delta:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallDelta + 8, // 5: pluggableharness.model.v1.StreamEvent.tool_call_done:type_name -> pluggableharness.model.v1.StreamEvent.ToolCallDone + 12, // 6: pluggableharness.model.v1.StreamEvent.usage:type_name -> pluggableharness.model.v1.Usage + 9, // 7: pluggableharness.model.v1.StreamEvent.stop:type_name -> pluggableharness.model.v1.StreamEvent.Stop + 10, // 8: pluggableharness.model.v1.StreamEvent.error:type_name -> pluggableharness.model.v1.StreamEvent.Error + 11, // 9: pluggableharness.model.v1.StreamEvent.redacted_thinking:type_name -> pluggableharness.model.v1.StreamEvent.RedactedThinking + 2, // 10: pluggableharness.model.v1.StreamEvent.stream_start:type_name -> pluggableharness.model.v1.StreamEvent.StreamStart + 0, // 11: pluggableharness.model.v1.StreamEvent.Stop.reason:type_name -> pluggableharness.model.v1.StopReason + 13, // 12: pluggableharness.model.v1.StreamEvent.Error.error:type_name -> pluggableharness.model.v1.ModelError + 13, // [13:13] is the sub-list for method output_type + 13, // [13:13] is the sub-list for method input_type + 13, // [13:13] is the sub-list for extension type_name + 13, // [13:13] is the sub-list for extension extendee + 0, // [0:13] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_events_proto_init() } @@ -908,15 +986,16 @@ func file_pluggableharness_model_v1_events_proto_init() { (*StreamEvent_Stop_)(nil), (*StreamEvent_Error_)(nil), (*StreamEvent_RedactedThinking_)(nil), + (*StreamEvent_StreamStart_)(nil), } - file_pluggableharness_model_v1_events_proto_msgTypes[7].OneofWrappers = []any{} + file_pluggableharness_model_v1_events_proto_msgTypes[8].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_pluggableharness_model_v1_events_proto_rawDesc), len(file_pluggableharness_model_v1_events_proto_rawDesc)), NumEnums: 1, - NumMessages: 10, + NumMessages: 11, NumExtensions: 0, NumServices: 0, }, diff --git a/pkg/model/proto/v1/types.pb.go b/pkg/model/proto/v1/types.pb.go index 16fbf3e..218da7e 100644 --- a/pkg/model/proto/v1/types.pb.go +++ b/pkg/model/proto/v1/types.pb.go @@ -160,6 +160,68 @@ func (ToolChoiceMode) EnumDescriptor() ([]byte, []int) { return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{1} } +// RateLimitKind names which vendor budget a RateLimitSnapshot describes. +type RateLimitKind int32 + +const ( + // Zero value. Never valid on a real snapshot; its presence on the wire + // means an adapter forgot to set the field. + RateLimitKind_RATE_LIMIT_KIND_UNSPECIFIED RateLimitKind = 0 + // Requests per window. + RateLimitKind_RATE_LIMIT_KIND_REQUESTS RateLimitKind = 1 + // Tokens per window, undifferentiated by direction. + RateLimitKind_RATE_LIMIT_KIND_TOKENS RateLimitKind = 2 + // Input tokens per window, where the vendor meters them separately. + RateLimitKind_RATE_LIMIT_KIND_INPUT_TOKENS RateLimitKind = 3 + // Output tokens per window, where the vendor meters them separately. + RateLimitKind_RATE_LIMIT_KIND_OUTPUT_TOKENS RateLimitKind = 4 +) + +// Enum value maps for RateLimitKind. +var ( + RateLimitKind_name = map[int32]string{ + 0: "RATE_LIMIT_KIND_UNSPECIFIED", + 1: "RATE_LIMIT_KIND_REQUESTS", + 2: "RATE_LIMIT_KIND_TOKENS", + 3: "RATE_LIMIT_KIND_INPUT_TOKENS", + 4: "RATE_LIMIT_KIND_OUTPUT_TOKENS", + } + RateLimitKind_value = map[string]int32{ + "RATE_LIMIT_KIND_UNSPECIFIED": 0, + "RATE_LIMIT_KIND_REQUESTS": 1, + "RATE_LIMIT_KIND_TOKENS": 2, + "RATE_LIMIT_KIND_INPUT_TOKENS": 3, + "RATE_LIMIT_KIND_OUTPUT_TOKENS": 4, + } +) + +func (x RateLimitKind) Enum() *RateLimitKind { + p := new(RateLimitKind) + *p = x + return p +} + +func (x RateLimitKind) String() string { + return protoimpl.X.EnumStringOf(x.Descriptor(), protoreflect.EnumNumber(x)) +} + +func (RateLimitKind) Descriptor() protoreflect.EnumDescriptor { + return file_pluggableharness_model_v1_types_proto_enumTypes[2].Descriptor() +} + +func (RateLimitKind) Type() protoreflect.EnumType { + return &file_pluggableharness_model_v1_types_proto_enumTypes[2] +} + +func (x RateLimitKind) Number() protoreflect.EnumNumber { + return protoreflect.EnumNumber(x) +} + +// Deprecated: Use RateLimitKind.Descriptor instead. +func (RateLimitKind) EnumDescriptor() ([]byte, []int) { + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{2} +} + // Capabilities is GetCapabilities' response payload: every model this // plugin can serve, plus provider-wide declarations that apply once, not // per model. @@ -1414,8 +1476,18 @@ type Usage struct { // future Pricing revision declares a distinct reasoning rate — there is // none as of this revision. ReasoningTokens *int64 `protobuf:"varint,5,opt,name=reasoning_tokens,json=reasoningTokens,proto3,oneof" json:"reasoning_tokens,omitempty"` - unknownFields protoimpl.UnknownFields - sizeCache protoimpl.SizeCache + // The vendor's own rate-limit state as of this completion, when it + // reports any. MAY be empty — a vendor that publishes nothing has + // nothing to declare, and an adapter MUST NOT synthesize a snapshot. + // + // Repeated because vendors publish several budgets at once and they + // exhaust independently: OpenAI and xAI return separate request and + // token headers, Anthropic reports input and output separately. Naming + // which budget is close to empty is the whole point — "you have 2% + // left" is unactionable without saying 2% of what. + RateLimits []*RateLimitSnapshot `protobuf:"bytes,6,rep,name=rate_limits,json=rateLimits,proto3" json:"rate_limits,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache } func (x *Usage) Reset() { @@ -1483,6 +1555,92 @@ func (x *Usage) GetReasoningTokens() int64 { return 0 } +func (x *Usage) GetRateLimits() []*RateLimitSnapshot { + if x != nil { + return x.RateLimits + } + return nil +} + +// RateLimitSnapshot is one of the vendor's rate-limit budgets as of one +// completion, per model/data-types.md#streamevent. +// +// Every numeric field is optional because vendors publish different +// subsets: an adapter reports what its vendor actually returned and omits +// the rest rather than inventing a value. A snapshot with only `kind` set +// is still useful — it says the budget exists. +type RateLimitSnapshot struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Which budget this describes. MUST be set. + Kind RateLimitKind `protobuf:"varint,1,opt,name=kind,proto3,enum=pluggableharness.model.v1.RateLimitKind" json:"kind,omitempty"` + // How much of this budget remains. + Remaining *int64 `protobuf:"varint,2,opt,name=remaining,proto3,oneof" json:"remaining,omitempty"` + // This budget's ceiling for the current window. + Limit *int64 `protobuf:"varint,3,opt,name=limit,proto3,oneof" json:"limit,omitempty"` + // When this budget next resets. + ResetAt *timestamppb.Timestamp `protobuf:"bytes,4,opt,name=reset_at,json=resetAt,proto3,oneof" json:"reset_at,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *RateLimitSnapshot) Reset() { + *x = RateLimitSnapshot{} + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *RateLimitSnapshot) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*RateLimitSnapshot) ProtoMessage() {} + +func (x *RateLimitSnapshot) ProtoReflect() protoreflect.Message { + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use RateLimitSnapshot.ProtoReflect.Descriptor instead. +func (*RateLimitSnapshot) Descriptor() ([]byte, []int) { + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{14} +} + +func (x *RateLimitSnapshot) GetKind() RateLimitKind { + if x != nil { + return x.Kind + } + return RateLimitKind_RATE_LIMIT_KIND_UNSPECIFIED +} + +func (x *RateLimitSnapshot) GetRemaining() int64 { + if x != nil && x.Remaining != nil { + return *x.Remaining + } + return 0 +} + +func (x *RateLimitSnapshot) GetLimit() int64 { + if x != nil && x.Limit != nil { + return *x.Limit + } + return 0 +} + +func (x *RateLimitSnapshot) GetResetAt() *timestamppb.Timestamp { + if x != nil { + return x.ResetAt + } + return nil +} + // ModelTarget describes the model a context or memory contribution is // being assembled for, derived from that model's ModelSpec // (model.md §2). Carried on context.md's ContextRequest and memory.md's @@ -1508,7 +1666,7 @@ type ModelTarget struct { func (x *ModelTarget) Reset() { *x = ModelTarget{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1520,7 +1678,7 @@ func (x *ModelTarget) String() string { func (*ModelTarget) ProtoMessage() {} func (x *ModelTarget) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[14] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1533,7 +1691,7 @@ func (x *ModelTarget) ProtoReflect() protoreflect.Message { // Deprecated: Use ModelTarget.ProtoReflect.Descriptor instead. func (*ModelTarget) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{14} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{15} } func (x *ModelTarget) GetId() string { @@ -1574,7 +1732,7 @@ type ModelRef struct { func (x *ModelRef) Reset() { *x = ModelRef{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1586,7 +1744,7 @@ func (x *ModelRef) String() string { func (*ModelRef) ProtoMessage() {} func (x *ModelRef) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[15] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1599,7 +1757,7 @@ func (x *ModelRef) ProtoReflect() protoreflect.Message { // Deprecated: Use ModelRef.ProtoReflect.Descriptor instead. func (*ModelRef) Descriptor() ([]byte, []int) { - return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{15} + return file_pluggableharness_model_v1_types_proto_rawDescGZIP(), []int{16} } func (x *ModelRef) GetProvider() string { @@ -1626,7 +1784,7 @@ type CacheBreakpoint_AfterAssembledContext struct { func (x *CacheBreakpoint_AfterAssembledContext) Reset() { *x = CacheBreakpoint_AfterAssembledContext{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1638,7 +1796,7 @@ func (x *CacheBreakpoint_AfterAssembledContext) String() string { func (*CacheBreakpoint_AfterAssembledContext) ProtoMessage() {} func (x *CacheBreakpoint_AfterAssembledContext) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[16] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1664,7 +1822,7 @@ type CacheBreakpoint_AfterTools struct { func (x *CacheBreakpoint_AfterTools) Reset() { *x = CacheBreakpoint_AfterTools{} - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[18] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -1676,7 +1834,7 @@ func (x *CacheBreakpoint_AfterTools) String() string { func (*CacheBreakpoint_AfterTools) ProtoMessage() {} func (x *CacheBreakpoint_AfterTools) ProtoReflect() protoreflect.Message { - mi := &file_pluggableharness_model_v1_types_proto_msgTypes[17] + mi := &file_pluggableharness_model_v1_types_proto_msgTypes[18] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -1801,16 +1959,27 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "\x04mode\x18\x01 \x01(\x0e2).pluggableharness.model.v1.ToolChoiceModeR\x04mode\x12 \n" + "\ttool_name\x18\x02 \x01(\tH\x00R\btoolName\x88\x01\x01B\f\n" + "\n" + - "_tool_name\"\xa5\x02\n" + + "_tool_name\"\xf4\x02\n" + "\x05Usage\x12!\n" + "\finput_tokens\x18\x01 \x01(\x03R\vinputTokens\x12#\n" + "\routput_tokens\x18\x02 \x01(\x03R\foutputTokens\x12/\n" + "\x11cache_read_tokens\x18\x03 \x01(\x03H\x00R\x0fcacheReadTokens\x88\x01\x01\x121\n" + "\x12cache_write_tokens\x18\x04 \x01(\x03H\x01R\x10cacheWriteTokens\x88\x01\x01\x12.\n" + - "\x10reasoning_tokens\x18\x05 \x01(\x03H\x02R\x0freasoningTokens\x88\x01\x01B\x14\n" + + "\x10reasoning_tokens\x18\x05 \x01(\x03H\x02R\x0freasoningTokens\x88\x01\x01\x12M\n" + + "\vrate_limits\x18\x06 \x03(\v2,.pluggableharness.model.v1.RateLimitSnapshotR\n" + + "rateLimitsB\x14\n" + "\x12_cache_read_tokensB\x15\n" + "\x13_cache_write_tokensB\x13\n" + - "\x11_reasoning_tokens\"q\n" + + "\x11_reasoning_tokens\"\xf0\x01\n" + + "\x11RateLimitSnapshot\x12<\n" + + "\x04kind\x18\x01 \x01(\x0e2(.pluggableharness.model.v1.RateLimitKindR\x04kind\x12!\n" + + "\tremaining\x18\x02 \x01(\x03H\x00R\tremaining\x88\x01\x01\x12\x19\n" + + "\x05limit\x18\x03 \x01(\x03H\x01R\x05limit\x88\x01\x01\x12:\n" + + "\breset_at\x18\x04 \x01(\v2\x1a.google.protobuf.TimestampH\x02R\aresetAt\x88\x01\x01B\f\n" + + "\n" + + "_remainingB\b\n" + + "\x06_limitB\v\n" + + "\t_reset_at\"q\n" + "\vModelTarget\x12\x0e\n" + "\x02id\x18\x01 \x01(\tR\x02id\x12%\n" + "\x0econtext_window\x18\x02 \x01(\x03R\rcontextWindow\x12+\n" + @@ -1828,7 +1997,13 @@ const file_pluggableharness_model_v1_types_proto_rawDesc = "" + "\x15TOOL_CHOICE_MODE_AUTO\x10\x01\x12\x18\n" + "\x14TOOL_CHOICE_MODE_ANY\x10\x02\x12\x19\n" + "\x15TOOL_CHOICE_MODE_NONE\x10\x03\x12\x1d\n" + - "\x19TOOL_CHOICE_MODE_SPECIFIC\x10\x04B>ZZ pluggableharness.model.v1.ModelSpec - 20, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec - 21, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema - 22, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint - 7, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec - 8, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec - 10, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing + 4, // 0: pluggableharness.model.v1.Capabilities.models:type_name -> pluggableharness.model.v1.ModelSpec + 22, // 1: pluggableharness.model.v1.Capabilities.slash_commands:type_name -> pluggableharness.common.v1.PromptExpansionSpec + 23, // 2: pluggableharness.model.v1.Capabilities.config_schema:type_name -> pluggableharness.config.v1.ConfigSchema + 24, // 3: pluggableharness.model.v1.Capabilities.supported_hook_points:type_name -> pluggableharness.common.v1.HookPoint + 8, // 4: pluggableharness.model.v1.ModelSpec.thinking:type_name -> pluggableharness.model.v1.ThinkingSpec + 9, // 5: pluggableharness.model.v1.ModelSpec.caching:type_name -> pluggableharness.model.v1.CachingSpec + 11, // 6: pluggableharness.model.v1.ModelSpec.pricing:type_name -> pluggableharness.model.v1.Pricing 1, // 7: pluggableharness.model.v1.ModelSpec.supported_tool_choice_modes:type_name -> pluggableharness.model.v1.ToolChoiceMode - 4, // 8: pluggableharness.model.v1.BudgetControl.range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange - 5, // 9: pluggableharness.model.v1.ThinkingSpec.effort:type_name -> pluggableharness.model.v1.EffortControl - 6, // 10: pluggableharness.model.v1.ThinkingSpec.budget:type_name -> pluggableharness.model.v1.BudgetControl + 5, // 8: pluggableharness.model.v1.BudgetControl.range:type_name -> pluggableharness.model.v1.ThinkingBudgetRange + 6, // 9: pluggableharness.model.v1.ThinkingSpec.effort:type_name -> pluggableharness.model.v1.EffortControl + 7, // 10: pluggableharness.model.v1.ThinkingSpec.budget:type_name -> pluggableharness.model.v1.BudgetControl 0, // 11: pluggableharness.model.v1.ThinkingSpec.disable:type_name -> pluggableharness.model.v1.ThinkingDisableSupport - 23, // 12: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp - 23, // 13: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp - 9, // 14: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier - 18, // 15: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext - 19, // 16: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools - 24, // 17: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema - 14, // 18: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice + 25, // 12: pluggableharness.model.v1.PricingTier.effective_from:type_name -> google.protobuf.Timestamp + 25, // 13: pluggableharness.model.v1.PricingTier.effective_until:type_name -> google.protobuf.Timestamp + 10, // 14: pluggableharness.model.v1.Pricing.tiers:type_name -> pluggableharness.model.v1.PricingTier + 20, // 15: pluggableharness.model.v1.CacheBreakpoint.after_assembled_context:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterAssembledContext + 21, // 16: pluggableharness.model.v1.CacheBreakpoint.after_tools:type_name -> pluggableharness.model.v1.CacheBreakpoint.AfterTools + 26, // 17: pluggableharness.model.v1.ToolDeclaration.input_schema:type_name -> pluggableharness.schema.v1.Schema + 15, // 18: pluggableharness.model.v1.GenerationParams.tool_choice:type_name -> pluggableharness.model.v1.ToolChoice 1, // 19: pluggableharness.model.v1.ToolChoice.mode:type_name -> pluggableharness.model.v1.ToolChoiceMode - 20, // [20:20] is the sub-list for method output_type - 20, // [20:20] is the sub-list for method input_type - 20, // [20:20] is the sub-list for extension type_name - 20, // [20:20] is the sub-list for extension extendee - 0, // [0:20] is the sub-list for field type_name + 17, // 20: pluggableharness.model.v1.Usage.rate_limits:type_name -> pluggableharness.model.v1.RateLimitSnapshot + 2, // 21: pluggableharness.model.v1.RateLimitSnapshot.kind:type_name -> pluggableharness.model.v1.RateLimitKind + 25, // 22: pluggableharness.model.v1.RateLimitSnapshot.reset_at:type_name -> google.protobuf.Timestamp + 23, // [23:23] is the sub-list for method output_type + 23, // [23:23] is the sub-list for method input_type + 23, // [23:23] is the sub-list for extension type_name + 23, // [23:23] is the sub-list for extension extendee + 0, // [0:23] is the sub-list for field type_name } func init() { file_pluggableharness_model_v1_types_proto_init() } @@ -1916,13 +2096,14 @@ func file_pluggableharness_model_v1_types_proto_init() { file_pluggableharness_model_v1_types_proto_msgTypes[11].OneofWrappers = []any{} file_pluggableharness_model_v1_types_proto_msgTypes[12].OneofWrappers = []any{} file_pluggableharness_model_v1_types_proto_msgTypes[13].OneofWrappers = []any{} + file_pluggableharness_model_v1_types_proto_msgTypes[14].OneofWrappers = []any{} type x struct{} out := protoimpl.TypeBuilder{ File: protoimpl.DescBuilder{ GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_pluggableharness_model_v1_types_proto_rawDesc), len(file_pluggableharness_model_v1_types_proto_rawDesc)), - NumEnums: 2, - NumMessages: 18, + NumEnums: 3, + NumMessages: 19, NumExtensions: 0, NumServices: 0, }, diff --git a/pkg/model/stream.go b/pkg/model/stream.go index a01578c..b7b7ed7 100644 --- a/pkg/model/stream.go +++ b/pkg/model/stream.go @@ -63,6 +63,23 @@ func (s *Sink) send(ev *modelv1.StreamEvent, terminal bool) error { return s.stream.Send(ev) } +// StreamStart sends the vendor's own identifier for this request, so a +// later failure can be correlated with the vendor's logs. +// +// SHOULD be called as soon as the id is known — normally from response +// headers, before any content streams — and before any other event. An id +// that only arrives on successful completion is absent in exactly the case +// it is needed, which is why this is a separate early event rather than a +// field on Stop. A Provider whose vendor publishes no such id simply never +// calls this. +func (s *Sink) StreamStart(providerRequestID string) error { + return s.send(&modelv1.StreamEvent{ + Event: &modelv1.StreamEvent_StreamStart_{ + StreamStart: &modelv1.StreamEvent_StreamStart{ProviderRequestId: providerRequestID}, + }, + }, false) +} + // TextDelta sends an incremental fragment of assistant text output. MUST // be supported by every plugin. func (s *Sink) TextDelta(text string) error { From 97d7a72db57394f11a7405a92e02fa944647b3bf Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:26:39 -0400 Subject: [PATCH 06/16] model: spec how a gateway satisfies GetCapabilities MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit GetCapabilities required a built-in model list and no network call, which no gateway or locally-served provider can satisfy literally: an aggregator's roster is genuinely dynamic across many upstream vendors, and a local runtime's roster is whatever the operator has pulled. Separate the two things the old wording conflated. The real requirement is per-invocation cost — the kernel may call this before every routing decision — not where the roster originates. So: resolve once in Configure, which already does real work and is already where a bad configuration must fail, then serve every call from memory. A background refresh is allowed but must never block the call, since a stale roster served instantly beats a fresh one that stalls routing. Also records that a credential attribute may only be declared required when every supported deployment needs one. A locally-served runtime on loopback typically has no auth while the same plugin pointed at that vendor's hosted tier does, and a required api_key makes the former impossible to configure at all. --- docs/specifications/model/conformance.md | 2 ++ docs/specifications/model/protocol.md | 14 +++++++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index eaec450..f952cee 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -25,6 +25,8 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | `text` content, both directions | MUST | baseline | | Streaming RPC shape | MUST | see [`README.md`](README.md#transport--lifecycle) / [`protocol.md`](protocol.md#streamcompletion) — applies even to non-streaming backends | | `GetCapabilities` / `Configure` / `StreamCompletion` RPCs | MUST | the whole protocol surface | +| `GetCapabilities` makes no per-call network request | MUST | [`protocol.md#getcapabilities`](protocol.md#getcapabilities) — a gateway or locally-served provider resolves its roster once in `Configure` and serves it from memory; a background refresh MUST NOT block the call | +| Credential attribute declared `required` only when every supported deployment needs one | MUST | [`protocol.md#gateway-and-locally-served-providers`](protocol.md#gateway-and-locally-served-providers) — a loopback-served runtime typically has no auth; validate the combination in `Configure` instead | | `Describe` RPC | MUST | [`protocol.md#describe`](protocol.md#describe) — identity for `dev_overrides` binaries with no lock-file entry | | Structured error taxonomy (above) | MUST | | | `tool_use` / `tool_result` | MUST, if any served model has `supports_tool_use = true` | | diff --git a/docs/specifications/model/protocol.md b/docs/specifications/model/protocol.md index b0b1b63..bdd7b7d 100644 --- a/docs/specifications/model/protocol.md +++ b/docs/specifications/model/protocol.md @@ -4,7 +4,19 @@ The six RPCs a model provider plugin exposes. See [`README.md`](README.md#transp ## `GetCapabilities` -Returns a `Capabilities` value with one `ModelSpec` per model the plugin can serve. This MUST be re-queryable cheaply (the kernel may call it often — e.g. before every routing decision) and MUST NOT require network calls to the vendor if avoidable; a plugin SHOULD ship its model list built in and refresh it lazily/periodically rather than blocking on a live API call per invocation. +Returns a `Capabilities` value with one `ModelSpec` per model the plugin can serve. This MUST be re-queryable cheaply — the kernel may call it often, e.g. before every routing decision — and **an individual `GetCapabilities` call MUST NOT make a network call to the vendor**. A plugin serving a fixed roster SHOULD ship its model list built in. + +### Gateway and locally-served providers + +The rule above is about *per-invocation cost*, not about where the roster originates. A provider fronting a gateway or a local runtime cannot ship a meaningful built-in list — an aggregator's roster is genuinely dynamic and spans many upstream vendors with differing capabilities, and a locally-served runtime's roster is whatever the operator has pulled onto that machine. Such a provider satisfies this RPC by **resolving its roster once, out of band, and serving the resolved result from memory**: + +- Resolve during `Configure`, which is called once at bring-up, is already permitted to do real work, and is already the place a bad configuration MUST fail. A roster fetch that fails there is a configuration failure with a clear cause, not a routing decision that mysteriously finds no models. +- Serve every `GetCapabilities` call from the in-process cache. The call stays cheap and non-blocking, which is the guarantee the requirement exists to protect. +- A provider MAY refresh that cache in the background on its own schedule. It MUST NOT make `GetCapabilities` block on the refresh; a stale roster served instantly is strictly better than a fresh one that stalls every routing decision. + +A provider whose upstream roster genuinely cannot be resolved at `Configure` time SHOULD serve a conservative built-in subset rather than an empty list, because an empty `Capabilities` is indistinguishable from "this provider serves nothing" and makes the plugin unroutable. + +**Credentials are not universally required.** A provider MUST NOT declare an API-key attribute `required` unless every deployment it supports needs one. A locally-served runtime reached over loopback typically has no authentication at all, and the same plugin pointed at that vendor's hosted tier does. Such a provider declares the credential optional and validates the actual combination in `Configure` — where a missing key for a remote endpoint is a clear, immediate configuration error — rather than making an unauthenticated local deployment impossible to configure. The response MAY additionally include `slash_commands: []common.v1.PromptExpansionSpec` (declared once for the provider as a whole, not per model) and MUST include the provider's `ConfigSchema`, so the kernel knows what fields `Configure` expects before ever calling it. Each `PromptExpansionSpec` is a static template-expansion command only — the kernel expands `template` with the user's arguments and submits the result as an ordinary user message, never executing anything. A direct-invoke command that runs one of this provider's own operations is declared by a `slashcommand.v1` provider instead ([`../slashcommand/protocol.md`](../slashcommand/protocol.md)), never here. See [`data-types.md`](data-types.md#modelspec) for the full `ModelSpec` shape. From 18000ecbed1d336a67452e725a6323eff2f03372 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:31:26 -0400 Subject: [PATCH 07/16] pkg: promote SSE framing and HTTP error classification MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit internal/anthropic/CLAUDE.md states the rule this applies: wanting something from internal/ means it belongs in pkg/, so every plugin author gets it. Two pieces of that package were never Anthropic-specific, and all four planned providers would otherwise have rewritten them. pkg/sse decodes SSE frames. The framing is the wire format, not any vendor's dialect, and the one subtlety worth centralizing is that bufio.Scanner's 64 KiB default silently TRUNCATES an over-long line rather than erroring — a corrupted frame that still parses is far worse than a failed read, and real vendor frames routinely exceed it. Writing that bound correctly turned out to need its own care: bufio's effective limit is max(limit, cap(initial)), so a larger initial buffer silently raises the ceiling, which a test now pins. The package yields frames and never interprets one. Vendors disagree about every interpretation — Anthropic treats the payload's own type field as authoritative and ignores event: entirely, while OpenAI-compatible vendors dispatch on event: and end with a literal [DONE] — so a framing package taking a position on either would be wrong for somebody. Both fields are surfaced; the caller decides. pkg/model.ClassifyHTTPStatus maps an HTTP status to the conformance taxonomy. Those are HTTP semantics, and centralizing them makes the two mistakes that actually hurt once instead of per vendor: a retryable 403 burns quota against a request that can never succeed, and a 413 read as a generic invalid request loses the kernel's chance to shrink context and retry. Its 5xx fallback is also what makes a vendor-specific overload code — Anthropic's 529 — classify correctly with no entry. Anthropic keeps only what is genuinely its own: which field decides an event's type, and its error.type vocabulary. Its existing tests pass unchanged, which is the evidence that both extractions preserve behavior. --- .claude/rules/go-layout.md | 6 + internal/anthropic/messages/classify.go | 39 +---- internal/anthropic/messages/sse.go | 77 +++------ pkg/model/classify.go | 62 +++++++ pkg/model/classify_test.go | 57 +++++++ pkg/sse/doc.go | 37 +++++ pkg/sse/scanner.go | 199 ++++++++++++++++++++++ pkg/sse/scanner_test.go | 212 ++++++++++++++++++++++++ 8 files changed, 606 insertions(+), 83 deletions(-) create mode 100644 pkg/model/classify.go create mode 100644 pkg/model/classify_test.go create mode 100644 pkg/sse/doc.go create mode 100644 pkg/sse/scanner.go create mode 100644 pkg/sse/scanner_test.go diff --git a/.claude/rules/go-layout.md b/.claude/rules/go-layout.md index c8a208d..b887bd0 100644 --- a/.claude/rules/go-layout.md +++ b/.claude/rules/go-layout.md @@ -39,6 +39,12 @@ pkg/ first-class, third-party-consumable Go integration — the tool/model JSON-Schema subset, ContentBlock) that every category SDK composes rather than reimplementing per category. + sse/ shared vendor-neutral plumbing: SSE frame + decoding, which every model provider needs and + none of them should rewrite. Not a builder and + not tied to one category — the test for belonging + here is that a third-party plugin author would + otherwise copy it out of another provider. docs/specifications/ protocol contracts (already exists, authoritative) ``` diff --git a/internal/anthropic/messages/classify.go b/internal/anthropic/messages/classify.go index 12de320..4a92414 100644 --- a/internal/anthropic/messages/classify.go +++ b/internal/anthropic/messages/classify.go @@ -52,41 +52,20 @@ var errorTypeTable = map[string]errorClassification{ errOverloaded: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, } -// httpStatusTable is the fallback used when the response body is missing -// or unparseable (e.g. an HTML proxy error page carries no error.type at -// all) — keyed by the HTTP status each error.type row above documents. -var httpStatusTable = map[int]errorClassification{ - 400: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, - 401: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, - 402: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, - 403: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, - 404: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, - 409: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, - 413: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_CONTEXT_LENGTH_EXCEEDED, false}, - 429: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_RATE_LIMITED, true}, - 500: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, - 504: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, - 529: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, -} - // classify resolves a category and retryability from errType first, -// falling back to status when errType is empty or unrecognized (an -// unparseable body). A 5xx status with no table entry of its own (502, -// 503, ...) still reads as OVERLOADED/retryable: any 5xx-equivalent -// response is, by definition, the vendor's own transient failure rather -// than an unclassifiable one, even though Anthropic's error.type -// vocabulary doesn't name it individually. +// falling back to the HTTP status when errType is empty or unrecognized +// (an unparseable body — an HTML proxy error page carries no error.type at +// all). +// +// The status fallback is model.ClassifyHTTPStatus, shared with every other +// provider: those are HTTP semantics rather than Anthropic's, and its 5xx +// rule is what makes Anthropic's own 529 overload code classify correctly +// without an entry anywhere. Only errorTypeTable above is vendor-specific. func classify(errType string, status int) (modelv1.ModelErrorCategory, bool) { if c, ok := errorTypeTable[errType]; ok { return c.category, c.retryable } - if c, ok := httpStatusTable[status]; ok { - return c.category, c.retryable - } - if status >= 500 { - return modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true - } - return modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNKNOWN, false + return model.ClassifyHTTPStatus(status) } // looksLikeContextLength reports whether message reads as Anthropic's diff --git a/internal/anthropic/messages/sse.go b/internal/anthropic/messages/sse.go index 97ced51..e170804 100644 --- a/internal/anthropic/messages/sse.go +++ b/internal/anthropic/messages/sse.go @@ -1,44 +1,37 @@ package messages import ( - "bufio" "encoding/json" "fmt" "io" - "strings" -) -// maxSSELineBytes bounds a single SSE line's buffer. bufio.Scanner's -// default 64 KiB cap silently truncates a line rather than erroring, and -// Anthropic can emit a single data: line well past that — a -// redacted_thinking block's base64 payload or a large input_json_delta -// fragment can run into the hundreds of KB. 10 MiB is a generous ceiling -// that costs nothing at rest (bufio.Scanner grows its buffer lazily up to -// this max) while still bounding worst-case memory per line. -const maxSSELineBytes = 10 << 20 + "github.com/pluggableharness/agent/pkg/sse" +) // Scanner reads Anthropic's server-sent event stream, decoding one // StreamEvent per call to Next. // -// Anthropic's wire format is one "name: value" field per line, blank-line -// separated events, and ":"-prefixed comment lines to ignore. Scanner -// decodes every event it sees — including ping — from the data: line's own -// JSON payload rather than the event: line, per this package's CLAUDE.md: -// the JSON's "type" field is authoritative even where it disagrees with, -// or the wire omits, a matching event: line. Handle (events.go) is where -// ping is filtered out; Scanner itself surfaces it so that choice lives in -// exactly one place. +// The SSE framing itself lives in pkg/sse, where every plugin author can +// reach it — nothing about reading blank-line-separated data: frames is +// Anthropic-specific. What stays here is the part that genuinely is: which +// field decides an event's type. +// +// Anthropic sends both an event: line and a JSON payload carrying its own +// "type", and this decodes from the payload alone. The JSON is +// authoritative even where it disagrees with, or the wire omits, a +// matching event: line — which is exactly why the shared scanner surfaces +// both fields and takes no position on either. Every event is decoded +// here, including ping; Handle (events.go) is where ping is filtered, so +// that choice lives in exactly one place. type Scanner struct { - scan *bufio.Scanner + scan *sse.Scanner cur StreamEvent err error } // NewScanner returns a Scanner reading Anthropic's SSE stream from r. func NewScanner(r io.Reader) *Scanner { - scan := bufio.NewScanner(r) - scan.Buffer(make([]byte, 0, 64*1024), maxSSELineBytes) - return &Scanner{scan: scan} + return &Scanner{scan: sse.NewScanner(r)} } // Next advances the Scanner to the next decoded event. It returns false at @@ -48,41 +41,19 @@ func (s *Scanner) Next() bool { if s.err != nil { return false } - - var dataLines []string - for s.scan.Scan() { - line := s.scan.Text() - switch { - case line == "": - if len(dataLines) > 0 { - return s.decode(dataLines) - } - case strings.HasPrefix(line, ":"): - // Comment line, ignored per the SSE spec. - case strings.HasPrefix(line, "data:"): - dataLines = append(dataLines, strings.TrimPrefix(strings.TrimPrefix(line, "data:"), " ")) + if !s.scan.Next() { + if err := s.scan.Err(); err != nil { + s.err = fmt.Errorf("anthropic: sse: %w", err) } - // Any other field (event:, id:, retry:) is ignored — see the - // package-level comment on why decoding relies on data: alone. - } - if err := s.scan.Err(); err != nil { - s.err = fmt.Errorf("anthropic: sse: read: %w", err) return false } - if len(dataLines) > 0 { - // The stream ended without a trailing blank line after the final - // event's data — decode what arrived rather than silently dropping it. - return s.decode(dataLines) - } - return false + return s.decode(s.scan.Data()) } -// decode joins dataLines per the SSE multi-line-data rule (concatenated -// with "\n", even though Anthropic only ever sends one data: line per -// event) and unmarshals the result into s.cur. A parse failure is an -// error, not a skip: a data: line that doesn't parse means something is +// decode unmarshals one frame's payload into s.cur. A parse failure is an +// error, not a skip: a data payload that doesn't parse means something is // wrong with the vendor's wire, not with one ignorable event. -func (s *Scanner) decode(dataLines []string) bool { +func (s *Scanner) decode(data []byte) bool { // Reset before unmarshaling. Both halves of this matter and both are // silent corruption if skipped: // @@ -98,7 +69,7 @@ func (s *Scanner) decode(dataLines []string) bool { // Zeroing costs one small struct assignment per event and removes // both. s.cur = StreamEvent{} - if err := json.Unmarshal([]byte(strings.Join(dataLines, "\n")), &s.cur); err != nil { + if err := json.Unmarshal(data, &s.cur); err != nil { s.err = fmt.Errorf("anthropic: sse: decode event: %w", err) return false } diff --git a/pkg/model/classify.go b/pkg/model/classify.go new file mode 100644 index 0000000..1ee722b --- /dev/null +++ b/pkg/model/classify.go @@ -0,0 +1,62 @@ +package model + +import ( + "net/http" + + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// httpStatusCategories maps an HTTP status to the error category +// docs/specifications/model/conformance.md's taxonomy assigns it, and +// whether the kernel may retry it. +// +// These are HTTP semantics, not any one vendor's: a 401 means the same +// thing everywhere. Only statuses whose meaning is unambiguous appear +// here; anything else falls through to ClassifyHTTPStatus's 5xx rule or +// to UNKNOWN. +var httpStatusCategories = map[int]struct { + category modelv1.ModelErrorCategory + retryable bool +}{ + http.StatusBadRequest: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, + http.StatusUnauthorized: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, + http.StatusPaymentRequired: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, + http.StatusForbidden: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, + http.StatusNotFound: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, + http.StatusConflict: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, + http.StatusRequestEntityTooLarge: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_CONTEXT_LENGTH_EXCEEDED, false}, + http.StatusUnprocessableEntity: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, + http.StatusTooManyRequests: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_RATE_LIMITED, true}, +} + +// ClassifyHTTPStatus maps an HTTP status to its model error category and +// whether the kernel may retry it, per +// docs/specifications/model/conformance.md#error-taxonomy. +// +// It exists so every provider does not re-derive the same table, and so +// the two mistakes that actually hurt are made once rather than per +// vendor: treating a 403 as retryable (it is a policy refusal, and +// retrying it burns quota against a request that can never succeed), and +// treating a 413 as a generic invalid request (it is a context overflow, +// and only the CONTEXT_LENGTH_EXCEEDED category tells the kernel to shrink +// the conversation and try again). +// +// Any 5xx without an entry of its own reads as OVERLOADED and retryable: +// a 5xx-equivalent response is by definition the vendor's own transient +// failure, whatever number it carries. This is what makes a vendor- +// specific overload code — Anthropic's 529, say — classify correctly with +// no table entry. +// +// A provider whose vendor publishes a structured error vocabulary SHOULD +// consult that first and use this only as the fallback for a body that is +// missing or unparseable, which is exactly the case an HTML error page +// from an intermediate proxy produces. +func ClassifyHTTPStatus(status int) (category modelv1.ModelErrorCategory, retryable bool) { + if c, ok := httpStatusCategories[status]; ok { + return c.category, c.retryable + } + if status >= http.StatusInternalServerError { + return modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true + } + return modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNKNOWN, false +} diff --git a/pkg/model/classify_test.go b/pkg/model/classify_test.go new file mode 100644 index 0000000..1ba87ec --- /dev/null +++ b/pkg/model/classify_test.go @@ -0,0 +1,57 @@ +package model_test + +import ( + "net/http" + "testing" + + "github.com/pluggableharness/agent/pkg/model" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +func TestClassifyHTTPStatus(t *testing.T) { + t.Parallel() + + tests := map[int]struct { + category modelv1.ModelErrorCategory + retryable bool + }{ + http.StatusBadRequest: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, false}, + http.StatusUnauthorized: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR, false}, + http.StatusTooManyRequests: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_RATE_LIMITED, true}, + // 413 must not degrade to a generic invalid request: only the + // context-length category tells the kernel to shrink and retry. + http.StatusRequestEntityTooLarge: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_CONTEXT_LENGTH_EXCEEDED, false}, + // A 5xx with no entry of its own still reads as a transient vendor + // failure — this is what makes a vendor-specific overload code + // classify correctly with no table entry. + http.StatusBadGateway: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, + http.StatusServiceUnavailable: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, + 529: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, true}, + // An unmapped 4xx is genuinely unclassifiable, and unknown is + // non-retryable by default per the taxonomy. + http.StatusTeapot: {modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNKNOWN, false}, + } + + for status, want := range tests { + gotCategory, gotRetryable := model.ClassifyHTTPStatus(status) + if gotCategory != want.category || gotRetryable != want.retryable { + t.Errorf("ClassifyHTTPStatus(%d) = (%v, %v), want (%v, %v)", + status, gotCategory, gotRetryable, want.category, want.retryable) + } + } +} + +func TestClassifyHTTPStatus_forbiddenIsNeverRetryable(t *testing.T) { + t.Parallel() + + // Called out on its own because it is the mistake that costs money: a + // 403 is a policy refusal, so retrying it burns quota against a request + // that can never succeed. + category, retryable := model.ClassifyHTTPStatus(http.StatusForbidden) + if retryable { + t.Error("ClassifyHTTPStatus(403) is retryable, want false") + } + if category != modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_AUTH_ERROR { + t.Errorf("ClassifyHTTPStatus(403) category = %v, want AUTH_ERROR", category) + } +} diff --git a/pkg/sse/doc.go b/pkg/sse/doc.go new file mode 100644 index 0000000..6125eee --- /dev/null +++ b/pkg/sse/doc.go @@ -0,0 +1,37 @@ +// Package sse decodes a server-sent event stream into its raw frames. +// +// Every major LLM vendor streams completions over SSE, so a model-provider +// plugin author would otherwise write this same framing loop once per +// vendor. It lives here rather than inside any one provider because +// nothing about it is vendor-specific: the framing is the SSE wire format +// (one "name: value" field per line, blank-line-separated events, +// ":"-prefixed comments ignored, multi-line data concatenated with "\n"). +// +// # What this package does and does not do +// +// A Scanner yields frames. It never interprets one: decoding a frame's +// Data into a vendor's own event type, deciding which event names are +// terminal, and recognizing a vendor's end-of-stream sentinel are all +// caller concerns, because vendors disagree about every one of them. +// Anthropic treats the JSON payload's own "type" field as authoritative +// and ignores the event: line entirely; OpenAI uses the event: line and +// ends a stream with a literal "[DONE]" data payload. A framing package +// that took a position on either would be wrong for somebody. +// +// # Usage +// +// scan := sse.NewScanner(resp.Body) +// for scan.Next() { +// if scan.IsDone() { +// break +// } +// var ev vendorEvent +// if err := json.Unmarshal(scan.Data(), &ev); err != nil { +// return err +// } +// // ... +// } +// if err := scan.Err(); err != nil { +// return err +// } +package sse diff --git a/pkg/sse/scanner.go b/pkg/sse/scanner.go new file mode 100644 index 0000000..01eda07 --- /dev/null +++ b/pkg/sse/scanner.go @@ -0,0 +1,199 @@ +package sse + +import ( + "bufio" + "bytes" + "fmt" + "io" +) + +// DefaultMaxFrameBytes bounds a single SSE frame's buffer. +// +// bufio.Scanner's own 64 KiB default is unusable here because it silently +// *truncates* an over-long line rather than erroring — a corrupted frame +// that still parses is far worse than a failed read. Real vendor frames +// routinely exceed 64 KiB: an encrypted-reasoning block's base64 payload +// or a large tool-argument fragment can run into the hundreds of KB. +// +// 10 MiB is a generous ceiling that costs nothing at rest, since +// bufio.Scanner grows its buffer lazily up to the maximum, while still +// bounding worst-case memory per frame. +const DefaultMaxFrameBytes = 10 << 20 + +// initialFrameBytes is the buffer a Scanner starts with, grown lazily as +// needed. Sized for the common case — most frames are a few hundred bytes +// — so a stream of small frames never allocates past it. +const initialFrameBytes = 64 << 10 + +// doneSentinel is the data payload OpenAI-compatible vendors send to mark +// the end of a stream. Recognized by IsDone, never acted on by Scanner +// itself. +const doneSentinel = "[DONE]" + +// Option configures a Scanner built by NewScanner. +type Option func(*Scanner) + +// WithMaxFrameBytes overrides DefaultMaxFrameBytes. A value <= 0 is +// ignored, so a caller passing an unset config field gets the default +// rather than a Scanner that fails on the first frame. +func WithMaxFrameBytes(n int) Option { + return func(s *Scanner) { + if n > 0 { + s.maxFrame = n + } + } +} + +// Scanner reads a server-sent event stream one frame at a time. +// +// The zero value is not usable; construct one with NewScanner. A Scanner +// is not safe for concurrent use — it is driven by whichever goroutine +// owns the response body. +type Scanner struct { + scan *bufio.Scanner + maxFrame int + + event string + data []byte + err error +} + +// NewScanner returns a Scanner reading an SSE stream from r. +func NewScanner(r io.Reader, opts ...Option) *Scanner { + s := &Scanner{maxFrame: DefaultMaxFrameBytes} + for _, opt := range opts { + opt(s) + } + // The initial buffer is capped at maxFrame because bufio.Scanner's own + // limit is effectively max(maxFrame, cap(initial)) — a larger initial + // buffer silently raises the ceiling, which would make + // WithMaxFrameBytes a no-op for any value below initialFrameBytes. + initial := initialFrameBytes + if s.maxFrame < initial { + initial = s.maxFrame + } + s.scan = bufio.NewScanner(r) + s.scan.Buffer(make([]byte, 0, initial), s.maxFrame) + return s +} + +// Next advances to the next frame carrying data. +// +// It returns false at end of stream or once Err reports an error, and true +// when Data and Event have a new frame ready. A frame with no data: line +// at all (a bare comment, or an event: with no payload) is skipped rather +// than surfaced, since there is nothing for a caller to decode. +func (s *Scanner) Next() bool { + if s.err != nil { + return false + } + + s.event = "" + s.data = nil + var payload [][]byte + + for s.scan.Scan() { + line := s.scan.Bytes() + switch { + case len(line) == 0: + // Blank line terminates a frame. A frame with no data is not + // surfaced, but its fields are still discarded before the next. + if len(payload) > 0 { + s.finish(payload) + return true + } + s.event = "" + case line[0] == ':': + // Comment, ignored per the SSE spec. Vendors use these as + // keepalives. + default: + name, value, ok := splitField(line) + if !ok { + continue + } + switch string(name) { + case "data": + // Copied, not aliased: bufio.Scanner reuses its buffer + // across Scan calls, so retaining the slice would let the + // next line overwrite this frame's payload in place. + payload = append(payload, bytes.Clone(value)) + case "event": + s.event = string(value) + } + // id: and retry: are part of SSE's reconnection model, which + // no vendor here uses for completions — ignored rather than + // surfaced, so the API stays the two fields callers need. + } + } + + if err := s.scan.Err(); err != nil { + s.err = fmt.Errorf("sse: read: %w", err) + return false + } + if len(payload) > 0 { + // The stream ended without a trailing blank line after the final + // frame. Decode what arrived rather than silently dropping it — a + // truncated-looking stream that still carried a complete terminal + // event is a real case, and dropping it would turn a good stream + // into a hang. + s.finish(payload) + return true + } + return false +} + +// finish joins payload per SSE's multi-line-data rule and records it as +// the current frame. +func (s *Scanner) finish(payload [][]byte) { + s.data = bytes.Join(payload, []byte("\n")) +} + +// splitField splits an SSE line into its field name and value, applying +// the spec's rule that a single space after the colon is part of the +// delimiter rather than the value. A line with no colon is not a field. +func splitField(line []byte) (name, value []byte, ok bool) { + name, value, ok = bytes.Cut(line, []byte(":")) + if !ok { + return nil, nil, false + } + if len(value) > 0 && value[0] == ' ' { + value = value[1:] + } + return name, value, true +} + +// Data returns the current frame's payload, with multi-line data already +// joined by "\n". +// +// The returned slice is owned by the caller and is not reused by a later +// Next, so it is safe to retain or unmarshal from directly. +func (s *Scanner) Data() []byte { + return s.data +} + +// IsDone reports whether the current frame is the "[DONE]" sentinel +// OpenAI-compatible vendors send to mark the end of a stream. +// +// Scanner never acts on it: a sentinel is still surfaced as an ordinary +// frame, and a caller that does not use this vendor convention can ignore +// this method entirely. Anthropic, for one, has no such sentinel and ends +// its stream with a real terminal event instead. +func (s *Scanner) IsDone() bool { + return string(s.data) == doneSentinel +} + +// Event returns the current frame's event: field, or "" when the frame +// carried none. +// +// Whether this matters is vendor-specific and deliberately left to the +// caller: some vendors dispatch on it, others treat the data payload's own +// type field as authoritative and ignore this entirely. +func (s *Scanner) Event() string { + return s.event +} + +// Err returns the error that stopped iteration, or nil at a clean end of +// stream. +func (s *Scanner) Err() error { + return s.err +} diff --git a/pkg/sse/scanner_test.go b/pkg/sse/scanner_test.go new file mode 100644 index 0000000..a40f2cd --- /dev/null +++ b/pkg/sse/scanner_test.go @@ -0,0 +1,212 @@ +package sse_test + +import ( + "errors" + "io" + "strings" + "testing" + + "github.com/pluggableharness/agent/pkg/sse" +) + +// frames drains a Scanner into (event, data) pairs, failing on a read +// error so a test asserting frames never also has to assert Err. +func frames(t *testing.T, s *sse.Scanner) [][2]string { + t.Helper() + + var got [][2]string + for s.Next() { + got = append(got, [2]string{s.Event(), string(s.Data())}) + } + if err := s.Err(); err != nil { + t.Fatalf("Err() = %v, want nil", err) + } + return got +} + +func TestScanner_framesAndFields(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + in string + want [][2]string + }{ + "one frame": { + in: "data: {\"a\":1}\n\n", + want: [][2]string{{"", `{"a":1}`}}, + }, + "event field is surfaced": { + in: "event: delta\ndata: {\"a\":1}\n\n", + want: [][2]string{{"delta", `{"a":1}`}}, + }, + "comments are ignored": { + in: ": keepalive\ndata: x\n\n", + want: [][2]string{{"", "x"}}, + }, + "multi-line data joins with newline": { + in: "data: one\ndata: two\n\n", + want: [][2]string{{"", "one\ntwo"}}, + }, + "only one leading space is stripped": { + in: "data: padded\n\n", + want: [][2]string{{"", " padded"}}, + }, + "no space after colon still parses": { + in: "data:tight\n\n", + want: [][2]string{{"", "tight"}}, + }, + "unterminated final frame is still yielded": { + in: "data: a\n\ndata: b", + want: [][2]string{{"", "a"}, {"", "b"}}, + }, + "frames with no data are skipped": { + in: "event: ping\n\ndata: real\n\n", + want: [][2]string{{"", "real"}}, + }, + "id and retry are ignored": { + in: "id: 7\nretry: 100\ndata: x\n\n", + want: [][2]string{{"", "x"}}, + }, + "empty stream yields nothing": { + in: "", + want: nil, + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + got := frames(t, sse.NewScanner(strings.NewReader(tt.in))) + if len(got) != len(tt.want) { + t.Fatalf("got %d frames %v, want %d %v", len(got), got, len(tt.want), tt.want) + } + for i := range got { + if got[i] != tt.want[i] { + t.Errorf("frame %d = %q, want %q", i, got[i], tt.want[i]) + } + } + }) + } +} + +func TestScanner_eventFieldDoesNotLeakAcrossFrames(t *testing.T) { + t.Parallel() + + // A frame carrying only an event: name must not have that name stick + // to the next frame — that would make a caller dispatching on Event() + // handle a payload as the wrong kind. + got := frames(t, sse.NewScanner(strings.NewReader("event: ping\n\ndata: x\n\n"))) + + if len(got) != 1 { + t.Fatalf("got %v, want one frame", got) + } + if got[0][0] != "" { + t.Errorf("Event() = %q, want empty — the preceding frame's name leaked", got[0][0]) + } +} + +func TestScanner_dataIsNotAliasedAcrossFrames(t *testing.T) { + t.Parallel() + + // bufio.Scanner reuses its buffer, so a Scanner retaining the raw slice + // would let a later line overwrite an earlier frame's payload in place + // — silent corruption a caller holding the slice cannot detect. + s := sse.NewScanner(strings.NewReader("data: first\n\ndata: second\n\n")) + + if !s.Next() { + t.Fatal("Next() = false, want the first frame") + } + held := s.Data() + + if !s.Next() { + t.Fatal("Next() = false, want the second frame") + } + if string(held) != "first" { + t.Errorf("the first frame's data became %q after advancing, want %q", held, "first") + } +} + +func TestScanner_isDone(t *testing.T) { + t.Parallel() + + // The sentinel is surfaced as an ordinary frame, not swallowed: a + // vendor that does not use this convention must see every frame, and + // one that does decides for itself when to stop. + s := sse.NewScanner(strings.NewReader("data: {\"a\":1}\n\ndata: [DONE]\n\n")) + + if !s.Next() || s.IsDone() { + t.Fatalf("first frame: IsDone() = true, want false") + } + if !s.Next() { + t.Fatal("Next() = false, want the sentinel frame") + } + if !s.IsDone() { + t.Errorf("IsDone() = false for %q, want true", s.Data()) + } +} + +func TestScanner_overlongFrameErrorsRatherThanTruncating(t *testing.T) { + t.Parallel() + + // The whole reason DefaultMaxFrameBytes exists: bufio.Scanner's own + // default silently truncates, and a corrupted frame that still parses + // is far worse than a failed read. + s := sse.NewScanner(strings.NewReader("data: "+strings.Repeat("x", 200)+"\n\n"), sse.WithMaxFrameBytes(64)) + + if s.Next() { + t.Fatalf("Next() = true with data %q, want false on an over-long frame", s.Data()) + } + if err := s.Err(); err == nil { + t.Fatal("Err() = nil, want an error rather than a silently truncated frame") + } +} + +func TestScanner_maxFrameBytesIgnoresNonPositive(t *testing.T) { + t.Parallel() + + // A caller passing an unset config field gets the default rather than + // a Scanner that fails on the first frame. + got := frames(t, sse.NewScanner(strings.NewReader("data: x\n\n"), sse.WithMaxFrameBytes(0))) + if len(got) != 1 { + t.Fatalf("got %v, want one frame", got) + } +} + +// errReader fails after yielding its payload, so the scanner's read-error +// path is exercised without a real network. +type errReader struct { + payload string + read bool +} + +func (e *errReader) Read(p []byte) (int, error) { + if e.read { + return 0, errors.New("boom") + } + e.read = true + return copy(p, e.payload), nil +} + +func TestScanner_readErrorSurfacesAndStops(t *testing.T) { + t.Parallel() + + s := sse.NewScanner(&errReader{payload: "data: a\n\n"}) + + if !s.Next() { + t.Fatal("Next() = false, want the frame that arrived before the error") + } + if s.Next() { + t.Error("Next() = true after a read error, want false") + } + if err := s.Err(); err == nil { + t.Fatal("Err() = nil, want the read error") + } + // Err is sticky: a caller looping on Next must not be able to resume + // past a failed read. + if s.Next() { + t.Error("Next() = true on a second call after an error, want false") + } +} + +var _ io.Reader = (*errReader)(nil) From 138a4f604d244501bfbecffc36909e17ac5631ff Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:34:06 -0400 Subject: [PATCH 08/16] examples: add a standalone provider module proving pkg/ suffices MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Nothing in this repository proved that pkg/ is usable from outside this module. internal/anthropic's depguard rule forbids it importing other internal/ packages, which simulates that isolation — but a simulation cannot catch an unexported type leaking through an exported signature, or a pkg/ package that only compiles because something else in the main module already resolved a dependency for it. examples/provider is its own Go module with a replace directive back to the working tree, so CI builds it against pkg/ as it exists in the commit under review rather than the last published release. Go excludes a nested module from the parent's ./... automatically, so it costs the main module's build and test nothing. It is also the reference an out-of-band session starts from, so it demonstrates the things that are easy to get wrong rather than the minimum that compiles: the zero ThinkingSpec/CachingSpec as the valid declaration for a model that does neither, a Configure written to be safely re-callable, cancellation returned unwrapped as normal control flow, provider_options read as a pass-through vendor knob, and a CountTokens that counts tool declarations rather than only message text. Adds the companion check the example cannot make on its own: pkg/ must not import internal/, since that compiles here and fails for every downstream author. pkg/telemetry stays the one sanctioned exception, already documented in its own source. The tidied go.mod is worth reading as output: it shows the real dependency tax a third-party plugin author pays for pkg/ — grpc, go-plugin, hclog, yamux, and the otel SDK. --- .github/workflows/ci.yml | 30 +++++ examples/provider/go.mod | 40 +++++++ examples/provider/go.sum | 80 +++++++++++++ examples/provider/main.go | 235 ++++++++++++++++++++++++++++++++++++++ 4 files changed, 385 insertions(+) create mode 100644 examples/provider/go.mod create mode 100644 examples/provider/go.sum create mode 100644 examples/provider/main.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f28edd8..04a5503 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,6 +80,36 @@ jobs: - name: Vet run: go vet ./... + # pkg/ is the third-party plugin-author surface, consumed from + # outside this module where internal/ is unreachable by definition. + # An internal/ import there compiles fine here and fails for every + # downstream author, so it has to be caught on this side. + # + # pkg/telemetry is the one sanctioned exception, documented in its + # own source: it wraps internal/telemetry but keeps its exported + # surface expressible outside this module. + - name: Verify pkg/ does not import internal/ + run: | + if grep -rn --include='*.go' 'pluggableharness/agent/internal' pkg/ \ + | grep -v '^pkg/telemetry/'; then + echo "::error::pkg/ must not import internal/ — see the exception note in pkg/telemetry" + exit 1 + fi + + # Builds the example provider from its own module, against this + # commit via a replace directive. This is the only check that proves + # pkg/ is genuinely usable from outside the main module: the + # depguard rule on internal/anthropic only simulates that isolation, + # and a simulation cannot catch an unexported type leaking through + # an exported signature. + - name: Build the standalone example provider + working-directory: examples/provider + run: | + go mod tidy + git diff --exit-code -- go.mod go.sum + go build ./... + go vet ./... + # --------------------------------------------------------------------------- # test — race-enabled tests on every platform we release for. # diff --git a/examples/provider/go.mod b/examples/provider/go.mod new file mode 100644 index 0000000..956e65d --- /dev/null +++ b/examples/provider/go.mod @@ -0,0 +1,40 @@ +// This is a SEPARATE module on purpose. See main.go's package comment. +module github.com/pluggableharness/agent-example-provider + +go 1.26 + +require ( + github.com/pluggableharness/agent v0.0.0 + google.golang.org/protobuf v1.36.11 +) + +require ( + github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/fatih/color v1.13.0 // indirect + github.com/go-logr/logr v1.4.3 // indirect + github.com/go-logr/stdr v1.2.2 // indirect + github.com/golang/protobuf v1.5.4 // indirect + github.com/google/uuid v1.6.0 // indirect + github.com/hashicorp/go-hclog v1.6.3 // indirect + github.com/hashicorp/go-plugin v1.8.0 // indirect + github.com/hashicorp/yamux v0.1.2 // indirect + github.com/mattn/go-colorable v0.1.12 // indirect + github.com/mattn/go-isatty v0.0.20 // indirect + github.com/oklog/run v1.1.0 // indirect + go.opentelemetry.io/auto/sdk v1.2.1 // indirect + go.opentelemetry.io/otel v1.44.0 // indirect + go.opentelemetry.io/otel/metric v1.44.0 // indirect + go.opentelemetry.io/otel/sdk v1.44.0 // indirect + go.opentelemetry.io/otel/trace v1.44.0 // indirect + golang.org/x/net v0.56.0 // indirect + golang.org/x/sys v0.46.0 // indirect + golang.org/x/text v0.40.0 // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/grpc v1.82.1 // indirect +) + +// Resolved from the working tree rather than the module proxy so this +// example is built against the pkg/ surface as it exists in this commit, +// not against the last published release. Without it, CI would prove +// nothing about the change under review. +replace github.com/pluggableharness/agent => ../.. diff --git a/examples/provider/go.sum b/examples/provider/go.sum new file mode 100644 index 0000000..7bea7cc --- /dev/null +++ b/examples/provider/go.sum @@ -0,0 +1,80 @@ +github.com/bufbuild/protocompile v0.14.1 h1:iA73zAf/fyljNjQKwYzUHD6AD4R8KMasmwa/FBatYVw= +github.com/bufbuild/protocompile v0.14.1/go.mod h1:ppVdAIhbr2H8asPk6k4pY7t9zB1OU5DoEw9xY/FUi1c= +github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs= +github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs= +github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/fatih/color v1.13.0 h1:8LOYc1KYPPmyKMuN8QV2DNRWNbLo6LZ0iLs8+mlH53w= +github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYFFOfk= +github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A= +github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI= +github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY= +github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag= +github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE= +github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek= +github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0= +github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo= +github.com/hashicorp/go-hclog v1.6.3 h1:Qr2kF+eVWjTiYmU7Y31tYlP1h0q/X3Nl3tPGdaB11/k= +github.com/hashicorp/go-hclog v1.6.3/go.mod h1:W4Qnvbt70Wk/zYJryRzDRU/4r0kIg0PVHBcfoyhpF5M= +github.com/hashicorp/go-plugin v1.8.0 h1:ie8S6RRY8RvB2usYZv+AAZ/wBvx2AU5p5QeP5j/FORs= +github.com/hashicorp/go-plugin v1.8.0/go.mod h1:BExt6KEaIYx804z8k4gRzRLEvxKVb+kn0NMcihqOqb8= +github.com/hashicorp/yamux v0.1.2 h1:XtB8kyFOyHXYVFnwT5C3+Bdo8gArse7j2AQ0DA0Uey8= +github.com/hashicorp/yamux v0.1.2/go.mod h1:C+zze2n6e/7wshOZep2A70/aQU6QBRWJO/G6FT1wIns= +github.com/jhump/protoreflect v1.17.0 h1:qOEr613fac2lOuTgWN4tPAtLL7fUSbuJL5X5XumQh94= +github.com/jhump/protoreflect v1.17.0/go.mod h1:h9+vUUL38jiBzck8ck+6G/aeMX8Z4QUY/NiJPwPNi+8= +github.com/mattn/go-colorable v0.1.9/go.mod h1:u6P/XSegPjTcexA+o6vUJrdnUu04hMope9wVRipJSqc= +github.com/mattn/go-colorable v0.1.12 h1:jF+Du6AlPIjs2BiUiQlKOX0rt3SujHxPnksPKZbaA40= +github.com/mattn/go-colorable v0.1.12/go.mod h1:u5H1YNBxpqRaxsYJYSkiCWKzEfiAb1Gb520KVy5xxl4= +github.com/mattn/go-isatty v0.0.12/go.mod h1:cbi8OIDigv2wuxKPP5vlRcQ1OAZbq2CE4Kysco4FUpU= +github.com/mattn/go-isatty v0.0.14/go.mod h1:7GGIvUiUoEMVVmxf/4nioHXj79iQHKdU27kJ6hsGG94= +github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= +github.com/mattn/go-isatty v0.0.20/go.mod h1:W+V8PltTTMOvKvAeJH7IuucS94S2C6jfK/D7dTCTo3Y= +github.com/oklog/run v1.1.0 h1:GEenZ1cK0+q0+wsJew9qUg/DyD8k3JzYsZAi5gYi2mA= +github.com/oklog/run v1.1.0/go.mod h1:sVPdnTZT1zYwAJeCMu2Th4T21pA3FPOQRfWjQlk7DVU= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME= +github.com/stretchr/testify v1.7.2/go.mod h1:R6va5+xMeoiuVRoj+gSkQ7d3FALtqAAGI1FQKckRals= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64= +go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y= +go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU= +go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc= +go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc= +go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo= +go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58= +go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0= +go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI= +go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA= +go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk= +go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o= +golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec= +golang.org/x/sys v0.0.0-20200116001909-b77594299b42/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20200223170610-d5e6a3e2c0ae/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs= +golang.org/x/sys v0.0.0-20210630005230-0f9fa26af87c/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20210927094055-39ccf1dd6fa6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.0.0-20220503163025-988cb79eb6c6/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.6.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg= +golang.org/x/sys v0.46.0 h1:noSf2Fq6F8DBgS+LysIkx7rIExoNHJsxOAtPp4rthXw= +golang.org/x/sys v0.46.0/go.mod h1:4GL1E5IUh+htKOUEOaiffhrAeqysfVGipDYzABqnCmw= +golang.org/x/text v0.40.0 h1:Ub2Z6/xjgF1WrYQz2nuITOEegKFtiIy+rieRJ5lHZKs= +golang.org/x/text v0.40.0/go.mod h1:hpnzDAfGV753zIKo+wk3u1bVKCGPbrnF7+7LBF/UHVY= +gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4= +gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa h1:mZHHdPZl0dbGHCflZgAq/Q468DWVFcU2whhB2KAo8fk= +google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8= +google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE= +google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA= +google.golang.org/protobuf v1.36.11 h1:fV6ZwhNocDyBLK0dj+fg8ektcVegBBuEolpbTQyBNVE= +google.golang.org/protobuf v1.36.11/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco= +gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/examples/provider/main.go b/examples/provider/main.go new file mode 100644 index 0000000..21cf405 --- /dev/null +++ b/examples/provider/main.go @@ -0,0 +1,235 @@ +// Command provider is a minimal, complete model-provider plugin, and the +// reference an author starts from when writing a real one. +// +// # Why this is a separate Go module +// +// It exists to prove a property nothing else in this repository can: +// that `pkg/` is sufficient to write a plugin against, from outside this +// module. `internal/anthropic` is held to a depguard rule that forbids it +// importing any other `internal/` package, which *simulates* that +// isolation — but a simulation cannot catch an unexported type leaking +// through an exported signature, or a `pkg/` package that only compiles +// because something in the main module already resolved a dependency for +// it. Building this module in CI does. +// +// Its own go.mod carries a replace directive back to the working tree, so +// it is built against `pkg/` as it exists in the commit under review +// rather than the last published release. +// +// # What it demonstrates +// +// The three MUST RPCs (docs/specifications/model/conformance.md's summary +// matrix), the optional TokenCounter, and the plugin.Serve wiring. It +// invents no vendor: StreamCompletion echoes a canned completion, so the +// example stays about the SDK surface rather than about HTTP. +// +// A real provider replaces echoProvider's bodies with vendor calls and +// its catalog with a real roster. Everything else here — the identity +// stamping, the Serve call, the capability declaration shape — is what +// that provider would also do. +package main + +import ( + "context" + "fmt" + "strings" + + "google.golang.org/protobuf/types/known/structpb" + + commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" + "github.com/pluggableharness/agent/pkg/config" + configv1 "github.com/pluggableharness/agent/pkg/config/proto/v1" + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" + "github.com/pluggableharness/agent/pkg/model" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" + "github.com/pluggableharness/agent/pkg/plugin" +) + +// modelID is the single model this example serves. +const modelID = "example-echo-1" + +// attrGreeting is the one config attribute, present so the example shows +// a real ConfigSchema round trip rather than an empty one. +const attrGreeting = "greeting" + +// Identity this build reports through Describe. Variables rather than +// constants so a release build can stamp them with -ldflags, matching how +// cmd/anthropic does it. +var ( + pluginName = "example-echo" + pluginVersion = "0.0.0" + pluginSource = "github.com/pluggableharness/agent/examples/provider" +) + +// echoProvider implements model.Provider without a vendor behind it. +type echoProvider struct { + greeting string +} + +// Compile-time proof this type serves the three MUST RPCs and the SHOULD +// one. Render (MAY) is deliberately absent — the kernel's generic +// fallback renders plain text fine, which is what a provider with nothing +// unusual to show should do. +var ( + _ model.Provider = (*echoProvider)(nil) + _ model.TokenCounter = (*echoProvider)(nil) +) + +// Capabilities declares the roster and config schema. +// +// No network call and no caching needed: the roster is a package +// constant. A provider fronting a gateway, whose roster is genuinely +// dynamic, resolves it once in Configure and serves it from memory here +// instead — see docs/specifications/model/protocol.md#getcapabilities. +func (p *echoProvider) Capabilities(context.Context) (*model.Capabilities, error) { + greeting, err := config.Attribute(attrGreeting, configv1.AttrType_ATTR_TYPE_STRING, + config.WithDefault(`"hello"`), + config.WithDescription("Prefix this provider prepends to every echoed completion."), + ) + if err != nil { + return nil, fmt.Errorf("example: config schema: %w", err) + } + schema, err := config.Schema(greeting) + if err != nil { + return nil, fmt.Errorf("example: config schema: %w", err) + } + + return model.NewCapabilities([]model.Spec{{ + ID: modelID, + ContextWindow: 200_000, + MaxOutputTokens: 4096, + SupportsToolUse: true, + // Thinking and Caching left at their zero values: this model does + // neither, and the zero value is the valid declaration for that. + Thinking: model.ThinkingSpec{}, + Caching: model.CachingSpec{}, + // Free, so the kernel's cost ledger stays at zero. A real provider + // declares at least one PricingTier here, and exactly one tier must + // match any (timestamp, input_token_count) pair. + Pricing: model.Pricing{Currency: "USD", Free: true}, + SupportedToolChoiceModes: []modelv1.ToolChoiceMode{ + modelv1.ToolChoiceMode_TOOL_CHOICE_MODE_AUTO, + }, + }}, schema) +} + +// Configure decodes the provider's agent.hcl block. +// +// It is written to be safely re-callable — the kernel may Configure a +// running plugin again when its configuration changes — so it replaces +// state wholesale rather than mutating it in place. A real provider +// rebuilds its vendor client here for the same reason. +func (p *echoProvider) Configure(_ context.Context, cfg *structpb.Struct) error { + greeting := "hello" + if v, ok := cfg.GetFields()[attrGreeting]; ok && v.GetStringValue() != "" { + greeting = v.GetStringValue() + } + p.greeting = greeting + return nil +} + +// StreamCompletion echoes the last user message back. +// +// Note what a Provider is responsible for even with no vendor involved: +// exactly one terminal event (Sink enforces this), a usage event so the +// kernel has something to account, and treating cancellation as normal +// control flow rather than an error. +func (p *echoProvider) StreamCompletion(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if req.GetModelId() != modelID { + return &model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, + Message: fmt.Sprintf("example: unknown model %q", req.GetModelId()), + Retryable: false, + } + } + + // provider_options is pass-through: the kernel never reads a key, so a + // vendor knob lives here rather than needing a protocol change. This + // example uses it to let an operator override the greeting per request. + greeting := p.greeting + if v, ok := model.ProviderOptions(req).LookupString(attrGreeting); ok { + greeting = v + } + + reply := greeting + ", " + lastUserText(req.GetMessages()) + + // Cancellation is normal control flow: return ctx.Err() unwrapped so + // errors.Is(err, context.Canceled) works, and let pkg/model map it to + // a bare codes.Canceled rather than an application error. + if err := ctx.Err(); err != nil { + return err + } + if err := sink.TextDelta(reply); err != nil { + return err + } + if err := sink.Usage(model.Usage{ + InputTokens: int64(len(reply) / 4), + OutputTokens: int64(len(reply) / 4), + }); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") +} + +// CountTokens implements the optional TokenCounter. +// +// It counts a whole request — messages, assembled context, and tool +// declarations — because that is what the RPC measures. A real provider +// calls its vendor's counting endpoint with the same three; the point +// here is that tool schemas are counted at all, since they are frequently +// the largest contributor and the easiest thing to forget. +func (p *echoProvider) CountTokens(_ context.Context, req *modelv1.CountTokensRequest) (int64, error) { + var bytes int + for _, m := range req.GetMessages() { + for _, b := range m.GetContent() { + bytes += len(b.GetText().GetText()) + } + } + for _, s := range req.GetAssembledContext() { + for _, b := range s.GetContent() { + bytes += len(b.GetText().GetText()) + } + } + for _, t := range req.GetTools() { + bytes += len(t.GetName()) + len(t.GetDescription()) + } + return int64((bytes + 3) / 4), nil +} + +// lastUserText returns the text of the most recent user message, or a +// placeholder when there is none. +func lastUserText(messages []*contentv1.Message) string { + for i := len(messages) - 1; i >= 0; i-- { + if messages[i].GetRole() != contentv1.Role_ROLE_USER { + continue + } + var sb strings.Builder + for _, b := range messages[i].GetContent() { + sb.WriteString(b.GetText().GetText()) + } + if sb.Len() > 0 { + return sb.String() + } + } + return "(nothing to echo)" +} + +func main() { + identity := plugin.Identity{ + Name: pluginName, + Version: pluginVersion, + Source: pluginSource, + } + + // Constructed here and handed to both Serve and the service, but never + // dialed from main: pkg/plugin's callback-timing trap means + // Callback.Client may only be called from inside an RPC handler. + callback := plugin.NewCallback() + + plugin.Serve(plugin.Config{ + Identity: identity, + Category: commonv1.Category_CATEGORY_MODEL, + Callback: callback, + Services: []plugin.Service{model.NewService(&echoProvider{greeting: "hello"}, identity, callback)}, + }) +} From 11b0820be654048b2fb8d831e0d0151c304bd88b Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 12:56:05 -0400 Subject: [PATCH 09/16] pkg: add the model-provider conformance suite MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit An out-of-band session building a provider had no way to know it was correct short of re-reading the spec. pkg/model/modeltest turns conformance.md's MUST/SHOULD matrix into assertions, in two drive modes sharing one implementation so they cannot drift: in-process over a real gRPC round trip, and against a built binary through a real handshake. The round trip is deliberate. Most of what this checks lives in the pkg/model service adapter and the wire types — terminal-event bookkeeping, error-to-status mapping, the conversion layer — and a direct method call on a Provider would exercise none of it. RunBinary is the only mode that reaches a plugin's own main() wiring, and the only one that works on a plugin written in another language. Assertions produce Findings rather than driving *testing.T. That is what lets the suite's own tests prove it rejects a bad provider — a conformance suite that cannot be shown to fail is worth very little — and what will let a non-test binary reuse it. Skips are reported rather than omitted, so a check the run could not reach never reads as a pass. Writing the tests found four real defects, three in the suite itself: - WithExpectedIdentity could never fail in-process, because Check was serving the very identity it then compared against. It now serves a fixed identity and reports the expectation as unverifiable in that mode, since a check that cannot fail is worse than no check. - checkCancellation had no bound, so a provider ignoring its context would hang the suite forever. Worse, the assertion it claimed to make is impossible from a black-box client: gRPC returns codes.Canceled to the canceling client whatever the server does, so a provider that keeps generating is invisible from this side. The check now asserts only what it can and documents the limit rather than implying coverage it does not have. - A BudgetControl whose range admits only zero was accepted. - The example provider silently accepted image and document blocks on a model declaring neither, which is exactly the violation the suite exists to catch. Fixed, and it now exercises the capability gates the Anthropic roster skips — between the two, every gate is covered. Both real providers pass: internal/anthropic against a canned in-process vendor, and examples/provider from its own module, which also proves modeltest is reachable by a third party. --- .github/workflows/ci.yml | 4 + examples/provider/conformance_test.go | 23 + examples/provider/main.go | 17 + internal/anthropic/conformance_test.go | 78 +++ pkg/model/modeltest/checks_test.go | 446 ++++++++++++++++++ pkg/model/modeltest/doc.go | 48 ++ pkg/model/modeltest/finding.go | 135 ++++++ pkg/model/modeltest/launch.go | 51 ++ pkg/model/modeltest/launch_test.go | 59 +++ pkg/model/modeltest/modeltest_test.go | 239 ++++++++++ pkg/model/modeltest/options.go | 134 ++++++ pkg/model/modeltest/options_test.go | 109 +++++ pkg/model/modeltest/report_test.go | 86 ++++ pkg/model/modeltest/run.go | 199 ++++++++ .../modeltest/runbinary_integration_test.go | 67 +++ pkg/model/modeltest/stream.go | 386 +++++++++++++++ pkg/model/modeltest/suite.go | 320 +++++++++++++ 17 files changed, 2401 insertions(+) create mode 100644 examples/provider/conformance_test.go create mode 100644 internal/anthropic/conformance_test.go create mode 100644 pkg/model/modeltest/checks_test.go create mode 100644 pkg/model/modeltest/doc.go create mode 100644 pkg/model/modeltest/finding.go create mode 100644 pkg/model/modeltest/launch.go create mode 100644 pkg/model/modeltest/launch_test.go create mode 100644 pkg/model/modeltest/modeltest_test.go create mode 100644 pkg/model/modeltest/options.go create mode 100644 pkg/model/modeltest/options_test.go create mode 100644 pkg/model/modeltest/report_test.go create mode 100644 pkg/model/modeltest/run.go create mode 100644 pkg/model/modeltest/runbinary_integration_test.go create mode 100644 pkg/model/modeltest/stream.go create mode 100644 pkg/model/modeltest/suite.go diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 04a5503..3adc5f3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -109,6 +109,10 @@ jobs: git diff --exit-code -- go.mod go.sum go build ./... go vet ./... + # Also runs the example's own conformance test, which proves + # pkg/model/modeltest is reachable and usable by a third party — + # the premise of shipping a conformance suite in pkg/ at all. + go test ./... # --------------------------------------------------------------------------- # test — race-enabled tests on every platform we release for. diff --git a/examples/provider/conformance_test.go b/examples/provider/conformance_test.go new file mode 100644 index 0000000..5a445d7 --- /dev/null +++ b/examples/provider/conformance_test.go @@ -0,0 +1,23 @@ +package main + +import ( + "testing" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +// TestConformance runs the shared conformance suite against this example. +// +// It is the check that makes the example trustworthy as a starting point: +// a reference an author copies from must itself satisfy the requirements +// it is meant to demonstrate. Running from a separate module also proves +// modeltest is reachable and usable by a third party, which is the whole +// premise of shipping it in pkg/. +func TestConformance(t *testing.T) { + t.Parallel() + + // No WithExpectedIdentity: in-process the identity is modeltest's own, + // so the expectation is unverifiable there. RunBinary is where a + // plugin's own identity stamping gets checked. + modeltest.Run(t, &echoProvider{greeting: "hello"}) +} diff --git a/examples/provider/main.go b/examples/provider/main.go index 21cf405..6397b21 100644 --- a/examples/provider/main.go +++ b/examples/provider/main.go @@ -143,6 +143,23 @@ func (p *echoProvider) StreamCompletion(ctx context.Context, req *modelv1.Stream } } + // Content this model does not declare support for MUST be rejected, + // never silently dropped: a dropped image means the model answers a + // question about a picture it was never shown, and nothing upstream + // can tell that happened. + for _, m := range req.GetMessages() { + for _, b := range m.GetContent() { + switch b.GetBlock().(type) { + case *contentv1.ContentBlock_Image, *contentv1.ContentBlock_Document: + return &model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, + Message: "example: this model accepts text only", + Retryable: false, + } + } + } + } + // provider_options is pass-through: the kernel never reads a key, so a // vendor knob lives here rather than needing a protocol change. This // example uses it to let an operator override the greeting per request. diff --git a/internal/anthropic/conformance_test.go b/internal/anthropic/conformance_test.go new file mode 100644 index 0000000..b259b24 --- /dev/null +++ b/internal/anthropic/conformance_test.go @@ -0,0 +1,78 @@ +package anthropic_test + +import ( + "io" + "net/http" + "strings" + "testing" + + "google.golang.org/protobuf/types/known/structpb" + + "github.com/pluggableharness/agent/internal/anthropic" + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +// TestConformance runs the shared conformance suite against the real +// Anthropic provider, pointed at a canned in-process vendor rather than +// the network. +// +// This is what keeps the suite honest in both directions: a protocol +// change that the suite does not understand fails here, and a suite +// assertion that no real provider could satisfy fails here too. The +// declarative half — every capability and pricing invariant across the +// whole roster — is exercised regardless of what the fake vendor returns. +func TestConformance(t *testing.T) { + t.Parallel() + + p := anthropic.New(anthropic.WithTransport(cannedVendor{})) + + cfg, err := structpb.NewStruct(map[string]any{ + "api_key": "sk-ant-conformance-fixture", + // Loopback http is permitted precisely so a test can point the + // provider at a fake vendor; see internal/anthropic/CLAUDE.md. + "base_url": "http://127.0.0.1:1", + }) + if err != nil { + t.Fatalf("structpb.NewStruct: %v", err) + } + + modeltest.Run(t, p, modeltest.WithConfig(cfg)) +} + +// cannedVendor answers every request with a minimal, well-formed +// Anthropic SSE stream, so the behavioral checks exercise the real +// translation path with no network. +type cannedVendor struct{} + +func (cannedVendor) RoundTrip(req *http.Request) (*http.Response, error) { + const stream = `event: message_start +data: {"type":"message_start","message":{"usage":{"input_tokens":8,"output_tokens":0}}} + +event: content_block_start +data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}} + +event: content_block_delta +data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"pong"}} + +event: content_block_stop +data: {"type":"content_block_stop","index":0} + +event: message_delta +data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":1}} + +event: message_stop +data: {"type":"message_stop"} + +` + return &http.Response{ + StatusCode: http.StatusOK, + Header: http.Header{ + "Content-Type": []string{"text/event-stream"}, + // Exercises the stream_start path: the adapter reads this from + // headers before any content arrives. + "Request-Id": []string{"req_conformance_fixture"}, + }, + Body: io.NopCloser(strings.NewReader(stream)), + Request: req, + }, nil +} diff --git a/pkg/model/modeltest/checks_test.go b/pkg/model/modeltest/checks_test.go new file mode 100644 index 0000000..df62f25 --- /dev/null +++ b/pkg/model/modeltest/checks_test.go @@ -0,0 +1,446 @@ +package modeltest_test + +import ( + "context" + "strings" + "testing" + "time" + + "google.golang.org/protobuf/types/known/structpb" + + configv1 "github.com/pluggableharness/agent/pkg/config/proto/v1" + "github.com/pluggableharness/agent/pkg/model" + "github.com/pluggableharness/agent/pkg/model/modeltest" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// rawProvider returns a hand-built Capabilities, bypassing +// model.NewCapabilities' own validation. +// +// That bypass is the point: NewCapabilities rejects most malformed +// advertisements before they ever reach the wire, so without it the +// suite's own declarative checks are unreachable from a Go provider. A +// provider written in another language has no such guard, and RunBinary +// is exactly how it would be checked — so these checks have to work, and +// have to be tested. +type rawProvider struct { + caps *model.Capabilities + stream func(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error +} + +func (p *rawProvider) Capabilities(context.Context) (*model.Capabilities, error) { + return p.caps, nil +} + +func (p *rawProvider) Configure(context.Context, *structpb.Struct) error { return nil } + +func (p *rawProvider) StreamCompletion(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if p.stream != nil { + return p.stream(ctx, req, sink) + } + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") +} + +var _ model.Provider = (*rawProvider)(nil) + +// rawCaps wraps one Spec into a Capabilities with a schema present, so a +// case fails on the property it names rather than on a missing schema. +func rawCaps(spec model.Spec) *model.Capabilities { + return &model.Capabilities{Models: []model.Spec{spec}, ConfigSchema: &configv1.ConfigSchema{}} +} + +// baseSpec is a valid Spec each case bends in exactly one way. +func baseSpec() model.Spec { + return model.Spec{ + ID: "raw-1", + ContextWindow: 1000, + MaxOutputTokens: 100, + Pricing: model.Pricing{Currency: "USD", Free: true}, + } +} + +func TestCheck_declarativeViolations(t *testing.T) { + t.Parallel() + + tokens := func(v int64) *int64 { return &v } + + tests := map[string]struct { + caps *model.Capabilities + wantMsg string + }{ + "no config schema": { + caps: &model.Capabilities{Models: []model.Spec{baseSpec()}}, + wantMsg: "no ConfigSchema is advertised", + }, + "no models": { + caps: &model.Capabilities{ConfigSchema: &configv1.ConfigSchema{}}, + wantMsg: "unroutable", + }, + "empty model id": { + caps: func() *model.Capabilities { + s := baseSpec() + s.ID = "" + return rawCaps(s) + }(), + wantMsg: "empty id", + }, + "duplicate model id": { + caps: &model.Capabilities{ + Models: []model.Spec{baseSpec(), baseSpec()}, + ConfigSchema: &configv1.ConfigSchema{}, + }, + wantMsg: "more than once", + }, + "max output not positive": { + caps: func() *model.Capabilities { + s := baseSpec() + s.MaxOutputTokens = 0 + return rawCaps(s) + }(), + wantMsg: "max_output_tokens", + }, + "thinking unsupported but a control is declared": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{Effort: &model.EffortControl{Levels: []string{"low"}, Default: "low"}} + return rawCaps(s) + }(), + wantMsg: "thinking is unsupported but a reasoning control is declared", + }, + "thinking unsupported but adaptive": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{AdaptiveByDefault: true} + return rawCaps(s) + }(), + wantMsg: "adaptive_by_default is set", + }, + "thinking unsupported but disable claims otherwise": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS} + return rawCaps(s) + }(), + wantMsg: "disable claims reasoning can be turned off", + }, + "effort control with no levels": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{ + Supported: true, + Effort: &model.EffortControl{Default: "low"}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + return rawCaps(s) + }(), + wantMsg: "no levels", + }, + "effort control with no default": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{ + Supported: true, + Effort: &model.EffortControl{Levels: []string{"low"}}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + return rawCaps(s) + }(), + wantMsg: "no default level", + }, + "budget control with no range": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{ + Supported: true, + Budget: &model.BudgetControl{}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + return rawCaps(s) + }(), + wantMsg: "admits no usable budget", + }, + "budget range inverted": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{ + Supported: true, + Budget: &model.BudgetControl{Range: model.ThinkingBudgetRange{Min: 90, Max: 10}}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + return rawCaps(s) + }(), + wantMsg: "inverted", + }, + "budget default outside the range": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Thinking = model.ThinkingSpec{ + Supported: true, + Budget: &model.BudgetControl{ + Range: model.ThinkingBudgetRange{Min: 10, Max: 90}, + Default: tokens(500), + }, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + return rawCaps(s) + }(), + wantMsg: "outside the declared range", + }, + "caching unsupported but a mechanism is declared": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Caching = model.CachingSpec{ExplicitMarkers: true} + return rawCaps(s) + }(), + wantMsg: "caching is unsupported but a caching mechanism is declared", + }, + "no pricing currency": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Pricing = model.Pricing{Free: true} + return rawCaps(s) + }(), + wantMsg: "no pricing currency", + }, + "no tiers and not free": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Pricing = model.Pricing{Currency: "USD"} + return rawCaps(s) + }(), + wantMsg: "no pricing tiers", + }, + "negative rate": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Pricing = model.Pricing{ + Currency: "USD", + Tiers: []model.PricingTier{{InputPerMtok: -1, OutputPerMtok: 5}}, + } + return rawCaps(s) + }(), + wantMsg: "negative rate", + }, + "caching supported but tier omits cache rates": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Caching = model.CachingSpec{Supported: true, ExplicitMarkers: true} + s.Pricing = model.Pricing{ + Currency: "USD", + Tiers: []model.PricingTier{{InputPerMtok: 1, OutputPerMtok: 5}}, + } + return rawCaps(s) + }(), + wantMsg: "omits a cache rate", + }, + // Disjoint on time but overlapping on input size is still an + // overlap only if both dimensions overlap — these do. + "tiers overlapping on both dimensions": { + caps: func() *model.Capabilities { + s := baseSpec() + s.Pricing = model.Pricing{ + Currency: "USD", + Tiers: []model.PricingTier{ + {InputPerMtok: 1, OutputPerMtok: 5, InputTokensFrom: tokens(0), InputTokensUntil: tokens(100)}, + {InputPerMtok: 2, OutputPerMtok: 6, InputTokensFrom: tokens(50), InputTokensUntil: tokens(200)}, + }, + } + return rawCaps(s) + }(), + wantMsg: "overlap", + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &rawProvider{caps: tt.caps}, + modeltest.WithCallTimeout(2*time.Second)) + if rep.OK() { + t.Fatalf("the suite passed a violating advertisement; report:\n%s", rep) + } + if !strings.Contains(rep.String(), tt.wantMsg) { + t.Errorf("no finding mentioned %q; the suite reported:\n%s", tt.wantMsg, rep) + } + }) + } +} + +// TestCheck_disjointTiersAreNotAnOverlap is the negative control for the +// tier-overlap rule: adjacent half-open ranges are exactly what a correct +// provider declares, and flagging them would make the check unusable. +func TestCheck_disjointTiersAreNotAnOverlap(t *testing.T) { + t.Parallel() + + from := int64(0) + mid := int64(100) + + s := baseSpec() + s.Pricing = model.Pricing{ + Currency: "USD", + Tiers: []model.PricingTier{ + {InputPerMtok: 1, OutputPerMtok: 5, InputTokensFrom: &from, InputTokensUntil: &mid}, + {InputPerMtok: 2, OutputPerMtok: 6, InputTokensFrom: &mid}, + }, + } + + rep := modeltest.Check(t.Context(), &rawProvider{caps: rawCaps(s)}, modeltest.WithCallTimeout(2*time.Second)) + if strings.Contains(rep.String(), "overlap") { + t.Errorf("adjacent half-open tiers were reported as overlapping:\n%s", rep) + } +} + +// TestCheck_streamViolations covers the stream checks a conforming +// provider never reaches. +func TestCheck_streamViolations(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + stream func(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error + wantMsg string + }{ + "error event with an unspecified category": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + return sink.Error(&model.Error{Message: "something went wrong"}) + }, + wantMsg: "unspecified category", + }, + "unknown error without raw detail": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + return sink.Error(&model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNKNOWN, + Message: "unclassifiable", + }) + }, + wantMsg: "omits raw_detail", + }, + "error event with no message": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + return sink.Error(&model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_OVERLOADED, + }) + }, + wantMsg: "no message", + }, + "tool call started but never closed": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.ToolCallStart("call-1", "read"); err != nil { + return err + } + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }, + wantMsg: "every started call must be closed", + }, + "tool call delta with no start": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.ToolCallDelta("orphan", `{"a":1}`); err != nil { + return err + } + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }, + wantMsg: "no preceding tool_call_start", + }, + "rate-limit snapshot with no kind": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.Usage(model.Usage{ + InputTokens: 1, + OutputTokens: 1, + RateLimits: []model.RateLimitSnapshot{{}}, + }); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }, + wantMsg: "names no budget kind", + }, + "matched_stop_sequence set for the wrong reason": { + stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + // Sink only forwards the sequence for STOP_SEQUENCE, so the + // violation is produced by claiming STOP_SEQUENCE with an + // empty sequence instead. + return sink.Stop(modelv1.StopReason_STOP_REASON_STOP_SEQUENCE, "") + }, + wantMsg: "matched_stop_sequence is empty", + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), + &rawProvider{caps: rawCaps(baseSpec()), stream: tt.stream}, + modeltest.WithCallTimeout(2*time.Second)) + if !strings.Contains(rep.String(), tt.wantMsg) { + t.Errorf("no finding mentioned %q; the suite reported:\n%s", tt.wantMsg, rep) + } + }) + } +} + +// TestCheck_identityExpectationsAreSkippedInProcess asserts the suite +// says so rather than passing a check it cannot make. +// +// In-process, modeltest supplies the identity the service reports, so +// comparing it against the caller's expectation would only ever compare +// modeltest against itself — a check that can never fail is worse than +// no check, because it reads as coverage. +func TestCheck_identityExpectationsAreSkippedInProcess(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &rawProvider{caps: rawCaps(baseSpec())}, + modeltest.WithCallTimeout(2*time.Second), + modeltest.WithExpectedIdentity("not-the-name", "9.9.9", "not-the-source")) + + var found bool + for _, f := range rep.Findings { + if strings.Contains(f.Check, "expected-identity") { + found = true + if f.Severity != modeltest.SeveritySkip { + t.Errorf("the identity expectation was reported as %v, want a skip", f.Severity) + } + } + } + if !found { + t.Errorf("the unverifiable identity expectation was not reported at all:\n%s", rep) + } +} + +// TestCheck_withoutStreamCompletionReportsTheGap asserts that opting out +// of the behavioral checks is visible rather than quiet — it is a real +// reduction in coverage, not a pass. +func TestCheck_withoutStreamCompletionReportsTheGap(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &conformingProvider{}, modeltest.WithoutStreamCompletion()) + if !rep.OK() { + t.Fatalf("skipping behavioral checks produced failures:\n%s", rep) + } + if !strings.Contains(rep.String(), "WithoutStreamCompletion") { + t.Errorf("the skipped behavioral checks were not reported:\n%s", rep) + } +} + +// TestCheck_unknownModelIDIsReported covers WithModelID naming a model +// the provider does not advertise. +func TestCheck_unknownModelIDIsReported(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &conformingProvider{}, modeltest.WithModelID("not-advertised")) + if !strings.Contains(rep.String(), "not advertised") { + t.Errorf("selecting an unadvertised model was not reported:\n%s", rep) + } +} diff --git a/pkg/model/modeltest/doc.go b/pkg/model/modeltest/doc.go new file mode 100644 index 0000000..f85ba8e --- /dev/null +++ b/pkg/model/modeltest/doc.go @@ -0,0 +1,48 @@ +// Package modeltest is a conformance suite for model-provider plugins. +// +// A provider author calls Run from their own test to check their plugin +// against the MUST/SHOULD matrix in +// docs/specifications/model/conformance.md, mechanically, rather than by +// re-reading the spec: +// +// func TestConformance(t *testing.T) { +// modeltest.Run(t, myprovider.New(), modeltest.WithConfig(cfg)) +// } +// +// # Two drive modes +// +// Run exercises a model.Provider in-process, over a real gRPC round trip +// on an in-memory listener. That is deliberately not a direct method call: +// most of what this suite checks lives in the pkg/model service adapter +// and the wire types — terminal-event bookkeeping, error-to-status +// mapping, the conversion layer — none of which a direct call would +// touch. +// +// RunBinary exercises an already-built plugin binary as the kernel does, +// through a real handshake and subprocess. It belongs in the integration +// tier (.claude/rules/go-testing.md), and it is the only mode that proves +// the plugin's own main() wiring is correct. It also works on a plugin +// written in any language, since it speaks only the wire protocol. +// +// # What it can and cannot check +// +// The declarative checks — capability invariants, pricing tier coverage, +// identity — are complete: they read what the provider advertises and +// need no vendor behind it. +// +// The behavioral checks can only assert what they can drive. A suite +// cannot make an arbitrary vendor emit an encrypted-reasoning block or a +// rate-limit header, so those are checked opportunistically: if the +// provider produces one during the run, its handling is asserted. A +// provider whose vendor emits such content SHOULD supply a request that +// triggers it via WithStreamRequest, so the check has something to bite +// on. What is never done is passing a check by not exercising it — a +// skipped check is reported as skipped. +// +// # Hermetic by construction +// +// This suite makes no network calls of its own. A provider under test +// that reaches a real vendor makes the run non-hermetic and billed; point +// it at a recorded transcript or a local test server instead, which is +// what WithConfig is for. +package modeltest diff --git a/pkg/model/modeltest/finding.go b/pkg/model/modeltest/finding.go new file mode 100644 index 0000000..7cafc6c --- /dev/null +++ b/pkg/model/modeltest/finding.go @@ -0,0 +1,135 @@ +package modeltest + +import ( + "fmt" + "sort" + "strings" +) + +// Severity classifies one check's outcome. +type Severity int + +const ( + // SeverityFail is a violated requirement. + SeverityFail Severity = iota + // SeveritySkip is a check the run could not reach — most often + // because the provider never produced the content it inspects. A skip + // is reported rather than silently omitted, so an unexercised check + // never reads as a pass. + SeveritySkip +) + +// String renders a Severity for a report line. +func (s Severity) String() string { + if s == SeveritySkip { + return "SKIP" + } + return "FAIL" +} + +// Finding is one check's outcome. +type Finding struct { + // Check names the requirement, e.g. "StreamCompletion/terminal-event". + Check string + // Severity is whether this is a violation or an unreached check. + Severity Severity + // Message states what was observed and why it matters. + Message string +} + +// String renders a Finding as one report line. +func (f Finding) String() string { + return fmt.Sprintf("%s %s: %s", f.Severity, f.Check, f.Message) +} + +// Report is the full outcome of one conformance run. +type Report struct { + Findings []Finding +} + +// Failures returns only the violated requirements. +func (r Report) Failures() []Finding { + var out []Finding + for _, f := range r.Findings { + if f.Severity == SeverityFail { + out = append(out, f) + } + } + return out +} + +// Skips returns only the checks the run could not reach. +func (r Report) Skips() []Finding { + var out []Finding + for _, f := range r.Findings { + if f.Severity == SeveritySkip { + out = append(out, f) + } + } + return out +} + +// OK reports whether the run found no violations. Skips do not fail a run +// — they are reported so a reader can see what was not covered. +func (r Report) OK() bool { + return len(r.Failures()) == 0 +} + +// String renders the whole report, failures first, each group sorted by +// check name so two runs over the same provider produce identical output. +func (r Report) String() string { + var sb strings.Builder + write := func(group []Finding) { + sort.Slice(group, func(i, j int) bool { return group[i].Check < group[j].Check }) + for _, f := range group { + sb.WriteString(f.String()) + sb.WriteString("\n") + } + } + write(r.Failures()) + write(r.Skips()) + return sb.String() +} + +// recorder accumulates findings under a check-name prefix. +// +// The suite's checks talk to this rather than to *testing.T, which is +// what makes the suite testable — a conformance suite that cannot itself +// be shown to reject a bad provider is worth very little — and what lets +// a non-test binary reuse the identical assertions. +type recorder struct { + prefix string + findings *[]Finding +} + +// newRecorder returns a recorder writing into findings. +func newRecorder(findings *[]Finding) *recorder { + return &recorder{findings: findings} +} + +// sub returns a recorder whose findings are named under name. +func (r *recorder) sub(name string) *recorder { + prefix := name + if r.prefix != "" { + prefix = r.prefix + "/" + name + } + return &recorder{prefix: prefix, findings: r.findings} +} + +// failf records a violated requirement. +func (r *recorder) failf(check, format string, args ...any) { + r.record(check, SeverityFail, fmt.Sprintf(format, args...)) +} + +// skipf records a check the run could not reach. +func (r *recorder) skipf(check, format string, args ...any) { + r.record(check, SeveritySkip, fmt.Sprintf(format, args...)) +} + +func (r *recorder) record(check string, sev Severity, msg string) { + name := check + if r.prefix != "" { + name = r.prefix + "/" + check + } + *r.findings = append(*r.findings, Finding{Check: name, Severity: sev, Message: msg}) +} diff --git a/pkg/model/modeltest/launch.go b/pkg/model/modeltest/launch.go new file mode 100644 index 0000000..3b3c939 --- /dev/null +++ b/pkg/model/modeltest/launch.go @@ -0,0 +1,51 @@ +package modeltest + +import ( + "context" + "errors" + "io" + "os" + "os/exec" + + "github.com/hashicorp/go-hclog" +) + +// errServeUnsupported is returned by the kernel-side plugin adapter's +// GRPCServer, which is never reached: this package only ever runs on the +// launching side of the connection. +var errServeUnsupported = errors.New("modeltest: this adapter only runs kernel-side and never serves") + +// commandContext builds the subprocess command for a plugin binary. +// +// The environment is a deliberate allowlist rather than os.Environ(), +// mirroring what the real launcher does: a plugin that only works because +// it inherited a credential from the test runner's environment would pass +// here and fail under the kernel, which is the opposite of what a +// conformance run is for. A provider needing configuration receives it +// through Configure, via WithConfig. +func commandContext(ctx context.Context, binaryPath string) *exec.Cmd { + cmd := exec.CommandContext(ctx, binaryPath) // #nosec G204 -- the caller names the binary under test; that is this function's entire purpose + cmd.Env = allowedEnv() + return cmd +} + +// allowedEnv returns the minimal environment a launched plugin gets. +func allowedEnv() []string { + env := make([]string, 0, 3) + for _, key := range []string{"PATH", "HOME", "TMPDIR"} { + if v, ok := os.LookupEnv(key); ok { + env = append(env, key+"="+v) + } + } + return env +} + +// discardLogger silences go-plugin's own subprocess-management chatter. +// +// It is handshake and process bookkeeping, not the plugin's application +// output, and surfacing it would bury a conformance failure in noise. A +// plugin's own logs cross the kernel-callback channel, which a +// conformance run does not serve. +func discardLogger() hclog.Logger { + return hclog.New(&hclog.LoggerOptions{Output: io.Discard, Level: hclog.Off}) +} diff --git a/pkg/model/modeltest/launch_test.go b/pkg/model/modeltest/launch_test.go new file mode 100644 index 0000000..b7a1386 --- /dev/null +++ b/pkg/model/modeltest/launch_test.go @@ -0,0 +1,59 @@ +package modeltest + +import ( + "context" + "slices" + "strings" + "testing" +) + +func TestCommandContext_usesAnEnvAllowlistNotTheAmbientEnvironment(t *testing.T) { + // Not parallel: t.Setenv forbids it. + t.Setenv("PATH", "/usr/bin") + t.Setenv("ANTHROPIC_API_KEY", "leaked-credential") + + cmd := commandContext(context.Background(), "/nonexistent/plugin") + + // The allowlist is the point. A plugin that only works because it + // inherited a credential from the test runner's environment would pass + // a conformance run and fail under the real kernel, which launches + // with the same narrow allowlist — the opposite of what conformance is + // for. Configuration reaches a provider through Configure. + for _, entry := range cmd.Env { + if strings.HasPrefix(entry, "ANTHROPIC_API_KEY=") { + t.Errorf("the ambient environment leaked into the subprocess: %q", entry) + } + } + if !slices.Contains(cmd.Env, "PATH=/usr/bin") { + t.Errorf("PATH was not passed through; got %v", cmd.Env) + } + if cmd.Path != "/nonexistent/plugin" && !strings.HasSuffix(cmd.Path, "plugin") { + t.Errorf("cmd.Path = %q, want the named binary", cmd.Path) + } +} + +func TestAllowedEnv_omitsUnsetKeys(t *testing.T) { + // Not parallel: t.Setenv forbids it. + t.Setenv("HOME", "/home/tester") + + for _, entry := range allowedEnv() { + if strings.HasPrefix(entry, "=") { + t.Errorf("an unset key produced a malformed entry: %q", entry) + } + } + if !slices.Contains(allowedEnv(), "HOME=/home/tester") { + t.Error("HOME was not passed through") + } +} + +func TestDiscardLogger_isUsableAndSilent(t *testing.T) { + t.Parallel() + + // go-plugin requires a non-nil logger; this one exists so its + // subprocess bookkeeping does not bury a conformance failure in noise. + logger := discardLogger() + if logger == nil { + t.Fatal("discardLogger() = nil, want a usable logger") + } + logger.Error("this must not reach the test output") +} diff --git a/pkg/model/modeltest/modeltest_test.go b/pkg/model/modeltest/modeltest_test.go new file mode 100644 index 0000000..4349213 --- /dev/null +++ b/pkg/model/modeltest/modeltest_test.go @@ -0,0 +1,239 @@ +package modeltest_test + +import ( + "context" + "strings" + "testing" + + "google.golang.org/protobuf/types/known/structpb" + + configv1 "github.com/pluggableharness/agent/pkg/config/proto/v1" + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" + "github.com/pluggableharness/agent/pkg/model" + "github.com/pluggableharness/agent/pkg/model/modeltest" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +const conformingID = "conforming-1" + +// conformingProvider satisfies every MUST the suite checks. It is the +// positive control: if the suite fails against this, the suite is wrong. +// +// mutate bends one declaration per case, so a negative control differs +// from the positive one in exactly the property under test. +type conformingProvider struct { + mutate func(*model.Spec) + stream func(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error +} + +func (p *conformingProvider) Capabilities(context.Context) (*model.Capabilities, error) { + spec := model.Spec{ + ID: conformingID, + ContextWindow: 200_000, + MaxOutputTokens: 4096, + Thinking: model.ThinkingSpec{}, + Caching: model.CachingSpec{}, + Pricing: model.Pricing{Currency: "USD", Free: true}, + } + if p.mutate != nil { + p.mutate(&spec) + } + return model.NewCapabilities([]model.Spec{spec}, &configv1.ConfigSchema{}) +} + +func (p *conformingProvider) Configure(context.Context, *structpb.Struct) error { return nil } + +func (p *conformingProvider) StreamCompletion(ctx context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if p.stream != nil { + return p.stream(ctx, req, sink) + } + if req.GetModelId() != conformingID { + return &model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, + Message: "unknown model", + } + } + // Reject content this model declares no support for, rather than + // silently dropping it. + for _, m := range req.GetMessages() { + for _, b := range m.GetContent() { + switch b.GetBlock().(type) { + case *contentv1.ContentBlock_Image, *contentv1.ContentBlock_Document: + return &model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, + Message: "unsupported content block", + } + } + } + } + if err := ctx.Err(); err != nil { + return err + } + if err := sink.TextDelta("pong"); err != nil { + return err + } + if err := sink.Usage(model.Usage{InputTokens: 3, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") +} + +var _ model.Provider = (*conformingProvider)(nil) + +func TestCheck_conformingProviderHasNoFailures(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &conformingProvider{}) + if !rep.OK() { + t.Fatalf("a conforming provider produced failures:\n%s", rep) + } + // Skips are expected — the suite cannot force this provider to emit a + // tool call or an encrypted reasoning block — but they must be + // reported rather than silently omitted, so an unexercised check never + // reads as a pass. + if len(rep.Skips()) == 0 { + t.Error("no skips were reported; unreachable checks must be visible, not omitted") + } +} + +func TestRun_conformingProviderPasses(t *testing.T) { + t.Parallel() + modeltest.Run(t, &conformingProvider{}) +} + +// TestCheck_catchesRealViolations is the suite's own regression guard. A +// conformance suite that cannot be shown to fail is worth very little, so +// each case is a provider with one genuine defect and the suite must name +// it. +func TestCheck_catchesRealViolations(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + provider model.Provider + wantMsg string + }{ + // The worst failure mode: to the kernel this looks like a clean + // turn that produced nothing, which is indistinguishable from + // success. + "no terminal event": { + provider: &conformingProvider{stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + return sink.TextDelta("no stop follows this") + }}, + wantMsg: "no terminal event", + }, + "no usage event": { + provider: &conformingProvider{stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.TextDelta("pong"); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }}, + wantMsg: "no usage event", + }, + "unspecified stop reason": { + provider: &conformingProvider{stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_UNSPECIFIED, "") + }}, + wantMsg: "STOP_REASON_UNSPECIFIED", + }, + // Accepts anything, including an image on a model declaring no + // vision support. + "unsupported content accepted": { + provider: &conformingProvider{stream: func(_ context.Context, _ *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }}, + wantMsg: "MUST be rejected", + }, + // Caught by model.NewCapabilities before the suite ever sees the + // advertisement, so the suite reports the failed RPC. That is the + // right outcome — a provider cannot ship this — and the assertion + // tracks the message that is actually produced. + "thinking supported without a disable value": { + provider: &conformingProvider{mutate: func(s *model.Spec) { + s.Thinking = model.ThinkingSpec{Supported: true} + }}, + wantMsg: "disable required when thinking is supported", + }, + "effort default is not a declared level": { + provider: &conformingProvider{mutate: func(s *model.Spec) { + s.Thinking = model.ThinkingSpec{ + Supported: true, + Effort: &model.EffortControl{Levels: []string{"low", "high"}, Default: "medium"}, + Disable: modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + } + }}, + wantMsg: "not one of the declared levels", + }, + "caching supported with no mechanism": { + provider: &conformingProvider{mutate: func(s *model.Spec) { + s.Caching = model.CachingSpec{Supported: true} + }}, + wantMsg: "neither explicit_markers nor implicit_automatic declared", + }, + "context window is not positive": { + provider: &conformingProvider{mutate: func(s *model.Spec) { s.ContextWindow = 0 }}, + wantMsg: "context_window", + }, + "overlapping pricing tiers": { + provider: &conformingProvider{mutate: func(s *model.Spec) { + s.Pricing = model.Pricing{ + Currency: "USD", + Tiers: []model.PricingTier{ + {InputPerMtok: 3, OutputPerMtok: 15}, + {InputPerMtok: 4, OutputPerMtok: 20}, + }, + } + }}, + wantMsg: "overlap", + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), tt.provider) + if rep.OK() { + t.Fatalf("the suite passed a provider that violates the spec; report:\n%s", rep) + } + if !strings.Contains(rep.String(), tt.wantMsg) { + t.Errorf("no finding mentioned %q; the suite reported:\n%s", tt.wantMsg, rep) + } + }) + } +} + +// TestCheck_cancellationIsNotReportedAsAFailure asserts the positive +// case, which is what this check can actually establish from a black-box +// client. Detecting a provider that IGNORES cancellation is not possible +// from this side — gRPC returns to the canceling client immediately, +// whatever the server does — and checkCancellation's own comment records +// that limit rather than implying coverage it does not have. +func TestCheck_cancellationIsNotReportedAsAFailure(t *testing.T) { + t.Parallel() + + rep := modeltest.Check(t.Context(), &conformingProvider{}) + for _, f := range rep.Findings { + if strings.HasPrefix(f.Check, "Cancellation/") && f.Severity == modeltest.SeverityFail { + t.Errorf("a well-behaved provider produced a cancellation failure: %s", f) + } + } +} + +func TestReport_stringIsDeterministic(t *testing.T) { + t.Parallel() + + // Two runs over the same provider must produce byte-identical output, + // or a report diff is unreadable noise. + first := modeltest.Check(t.Context(), &conformingProvider{}).String() + second := modeltest.Check(t.Context(), &conformingProvider{}).String() + if first != second { + t.Errorf("two runs produced different reports:\n--- first ---\n%s\n--- second ---\n%s", first, second) + } +} diff --git a/pkg/model/modeltest/options.go b/pkg/model/modeltest/options.go new file mode 100644 index 0000000..50b3139 --- /dev/null +++ b/pkg/model/modeltest/options.go @@ -0,0 +1,134 @@ +package modeltest + +import ( + "time" + + "google.golang.org/protobuf/proto" + "google.golang.org/protobuf/types/known/structpb" + + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// Option configures a conformance run. +type Option func(*config) + +// config is the resolved option set for one run. +type config struct { + configure *structpb.Struct + modelID string + streamRequest *modelv1.StreamCompletionRequest + skipStream bool + identity identityExpectation + callTimeout time.Duration + // inProcess records which drive mode resolved this config, so a check + // that only means something against a built binary can say so. + inProcess bool +} + +// identityExpectation is what Describe must report, when the caller cares. +// Empty fields are not checked, since a plugin stamping its version with +// -ldflags legitimately reports "0.0.0" in a test build. +type identityExpectation struct { + name string + version string + source string +} + +// WithConfig supplies the config object the run passes to Configure. +// +// This is also how a provider is pointed at a recorded transcript or a +// local test server rather than a real vendor: a conformance run must not +// make billed network calls, and the provider's own base-URL attribute is +// the supported way to redirect it. +func WithConfig(cfg *structpb.Struct) Option { + return func(c *config) { c.configure = cfg } +} + +// WithCallTimeout bounds each RPC the suite issues. Defaults to +// DefaultCallTimeout. +// +// Lower it against a fake or a recorded transcript, where any real delay +// means the provider is wedged and waiting the full default only makes +// the failure slower to see. Raise it for a provider that legitimately +// talks to a slow vendor. +func WithCallTimeout(d time.Duration) Option { + return func(c *config) { + if d > 0 { + c.callTimeout = d + } + } +} + +// WithModelID selects which advertised model the behavioral checks +// exercise. Defaults to the first model in GetCapabilities' response. +func WithModelID(id string) Option { + return func(c *config) { c.modelID = id } +} + +// WithStreamRequest replaces the request the behavioral checks send. +// +// Use it to drive content the default request cannot reach — a tool call, +// an image, or the prompt that makes a vendor emit encrypted reasoning — +// so the opportunistic checks have something to bite on. The request's +// model_id is overwritten with the resolved model, so a caller does not +// have to keep the two in sync. +func WithStreamRequest(req *modelv1.StreamCompletionRequest) Option { + return func(c *config) { c.streamRequest = req } +} + +// WithoutStreamCompletion skips every behavioral check, leaving only the +// declarative ones. +// +// Intended for a provider that genuinely cannot complete a request in a +// hermetic environment. It is a real reduction in coverage and the run +// reports it as skipped rather than passing quietly. +func WithoutStreamCompletion() Option { + return func(c *config) { c.skipStream = true } +} + +// WithExpectedIdentity asserts what Describe reports. Empty arguments are +// not checked — a test build legitimately carries an unstamped version. +func WithExpectedIdentity(name, version, source string) Option { + return func(c *config) { + c.identity = identityExpectation{name: name, version: version, source: source} + } +} + +// resolve applies opts over the defaults. +func resolve(opts []Option) *config { + c := &config{configure: &structpb.Struct{}, callTimeout: DefaultCallTimeout} + for _, opt := range opts { + opt(c) + } + return c +} + +// streamRequestFor returns the request the behavioral checks should send +// for modelID, either the caller's or a minimal default. +func (c *config) streamRequestFor(modelID string) *modelv1.StreamCompletionRequest { + req := c.streamRequest + if req == nil { + req = &modelv1.StreamCompletionRequest{ + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{{ + Block: &contentv1.ContentBlock_Text{ + Text: &contentv1.TextBlock{Text: "Reply with the single word: pong"}, + }, + }}, + }}, + } + } + // Cloned so a caller reusing one request across runs, or across a + // Run/RunBinary pair, never has it mutated underneath them. + out, ok := proto.Clone(req).(*modelv1.StreamCompletionRequest) + if !ok { + // Unreachable for a well-formed request: proto.Clone returns the + // same concrete type it was given. Checked rather than asserted + // per go-style.md's comma-ok rule. + return req + } + out.ModelId = modelID + return out +} diff --git a/pkg/model/modeltest/options_test.go b/pkg/model/modeltest/options_test.go new file mode 100644 index 0000000..b65d4bf --- /dev/null +++ b/pkg/model/modeltest/options_test.go @@ -0,0 +1,109 @@ +package modeltest_test + +import ( + "context" + "strings" + "testing" + "time" + + "google.golang.org/protobuf/types/known/structpb" + + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" + "github.com/pluggableharness/agent/pkg/model" + "github.com/pluggableharness/agent/pkg/model/modeltest" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +func TestWithConfig_reachesConfigure(t *testing.T) { + t.Parallel() + + cfg, err := structpb.NewStruct(map[string]any{"api_key": "supplied-by-the-caller"}) + if err != nil { + t.Fatalf("structpb.NewStruct: %v", err) + } + + var got *structpb.Struct + p := &configRecordingProvider{onConfigure: func(c *structpb.Struct) { got = c }} + + rep := modeltest.Check(t.Context(), p, modeltest.WithConfig(cfg), modeltest.WithCallTimeout(2*time.Second)) + if !rep.OK() { + t.Fatalf("unexpected failures:\n%s", rep) + } + if got.GetFields()["api_key"].GetStringValue() != "supplied-by-the-caller" { + t.Errorf("Configure received %v, want the caller's config", got) + } +} + +func TestWithStreamRequest_isUsedAndItsModelIDOverwritten(t *testing.T) { + t.Parallel() + + var seen *modelv1.StreamCompletionRequest + p := &conformingProvider{stream: func(_ context.Context, req *modelv1.StreamCompletionRequest, sink *model.Sink) error { + if seen == nil { + seen = req + } + if err := sink.Usage(model.Usage{InputTokens: 1, OutputTokens: 1}); err != nil { + return err + } + return sink.Stop(modelv1.StopReason_STOP_REASON_END_TURN, "") + }} + + custom := &modelv1.StreamCompletionRequest{ + // A deliberately wrong model id: the suite overwrites it with the + // resolved model, so a caller does not have to keep the two in + // sync. + ModelId: "stale-id-the-caller-forgot", + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{{ + Block: &contentv1.ContentBlock_Text{Text: &contentv1.TextBlock{Text: "custom prompt"}}, + }}, + }}, + } + + modeltest.Check(t.Context(), p, modeltest.WithStreamRequest(custom), modeltest.WithCallTimeout(2*time.Second)) + + if seen == nil { + t.Fatal("the provider was never called") + } + if seen.GetModelId() != conformingID { + t.Errorf("model_id = %q, want the resolved %q", seen.GetModelId(), conformingID) + } + if got := seen.GetMessages()[0].GetContent()[0].GetText().GetText(); got != "custom prompt" { + t.Errorf("prompt = %q, want the caller's", got) + } + // The caller's request must not be mutated: reusing one across runs is + // the obvious thing to do, and a rewritten model_id would silently + // change the second run. + if custom.GetModelId() != "stale-id-the-caller-forgot" { + t.Errorf("the caller's request was mutated: model_id is now %q", custom.GetModelId()) + } +} + +func TestWithCallTimeout_ignoresNonPositive(t *testing.T) { + t.Parallel() + + // A caller passing an unset config field gets the default rather than + // a suite whose every RPC times out instantly. + rep := modeltest.Check(t.Context(), &conformingProvider{}, modeltest.WithCallTimeout(0)) + if !rep.OK() { + t.Errorf("a zero timeout was not ignored:\n%s", rep) + } + if strings.Contains(rep.String(), "DeadlineExceeded") { + t.Errorf("a zero timeout took effect:\n%s", rep) + } +} + +// configRecordingProvider is a conforming provider that reports what +// Configure received. +type configRecordingProvider struct { + conformingProvider + onConfigure func(*structpb.Struct) +} + +func (p *configRecordingProvider) Configure(_ context.Context, cfg *structpb.Struct) error { + if p.onConfigure != nil { + p.onConfigure(cfg) + } + return nil +} diff --git a/pkg/model/modeltest/report_test.go b/pkg/model/modeltest/report_test.go new file mode 100644 index 0000000..cd8854a --- /dev/null +++ b/pkg/model/modeltest/report_test.go @@ -0,0 +1,86 @@ +package modeltest_test + +import ( + "strings" + "testing" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +func TestReport_okIgnoresSkips(t *testing.T) { + t.Parallel() + + // A skip means a requirement was not reached, which is worth seeing + // but is not a violation — a run that skipped everything and violated + // nothing has still not failed. + rep := modeltest.Report{Findings: []modeltest.Finding{ + {Check: "a", Severity: modeltest.SeveritySkip, Message: "not reached"}, + }} + if !rep.OK() { + t.Error("OK() = false for a skip-only report, want true") + } + if len(rep.Skips()) != 1 || len(rep.Failures()) != 0 { + t.Errorf("Skips=%d Failures=%d, want 1 and 0", len(rep.Skips()), len(rep.Failures())) + } +} + +func TestReport_okIsFalseWithAnyFailure(t *testing.T) { + t.Parallel() + + rep := modeltest.Report{Findings: []modeltest.Finding{ + {Check: "a", Severity: modeltest.SeveritySkip, Message: "not reached"}, + {Check: "b", Severity: modeltest.SeverityFail, Message: "violated"}, + }} + if rep.OK() { + t.Error("OK() = true with a failure present, want false") + } +} + +func TestReport_stringOrdersFailuresFirstAndSortsEachGroup(t *testing.T) { + t.Parallel() + + // Failures first because they are what a reader is looking for, and + // each group sorted so two runs over the same provider produce + // byte-identical output — an unsorted report makes a diff unreadable. + rep := modeltest.Report{Findings: []modeltest.Finding{ + {Check: "zeta", Severity: modeltest.SeveritySkip, Message: "s1"}, + {Check: "beta", Severity: modeltest.SeverityFail, Message: "f1"}, + {Check: "alpha", Severity: modeltest.SeveritySkip, Message: "s2"}, + {Check: "alpha", Severity: modeltest.SeverityFail, Message: "f2"}, + }} + + got := rep.String() + want := strings.Join([]string{ + "FAIL alpha: f2", + "FAIL beta: f1", + "SKIP alpha: s2", + "SKIP zeta: s1", + "", + }, "\n") + if got != want { + t.Errorf("String() =\n%q\nwant\n%q", got, want) + } +} + +func TestSeverity_string(t *testing.T) { + t.Parallel() + + if got := modeltest.SeverityFail.String(); got != "FAIL" { + t.Errorf("SeverityFail = %q, want FAIL", got) + } + if got := modeltest.SeveritySkip.String(); got != "SKIP" { + t.Errorf("SeveritySkip = %q, want SKIP", got) + } +} + +func TestReport_emptyIsOK(t *testing.T) { + t.Parallel() + + var rep modeltest.Report + if !rep.OK() { + t.Error("OK() = false for an empty report, want true") + } + if rep.String() != "" { + t.Errorf("String() = %q, want empty", rep.String()) + } +} diff --git a/pkg/model/modeltest/run.go b/pkg/model/modeltest/run.go new file mode 100644 index 0000000..c9cd4ff --- /dev/null +++ b/pkg/model/modeltest/run.go @@ -0,0 +1,199 @@ +package modeltest + +import ( + "context" + "fmt" + "net" + "testing" + + goplugin "github.com/hashicorp/go-plugin" + "google.golang.org/grpc" + "google.golang.org/grpc/credentials/insecure" + "google.golang.org/grpc/test/bufconn" + + "github.com/pluggableharness/agent/pkg/common" + commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" + "github.com/pluggableharness/agent/pkg/model" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" + "github.com/pluggableharness/agent/pkg/plugin" +) + +// bufSize is the in-memory listener's buffer. Generous enough that a +// large frame never blocks the writer during a test. +const bufSize = 1 << 20 + +// Run checks p against the conformance suite and fails t for every +// violation. This is the entry point a provider author calls from their +// own test. +// +// Skipped checks are logged rather than failed: a skip means the run +// could not reach that requirement, which is worth seeing but is not a +// violation. +func Run(t *testing.T, p model.Provider, opts ...Option) { + t.Helper() + report(t, Check(t.Context(), p, opts...)) +} + +// RunBinary checks an already-built plugin binary and fails t for every +// violation. +// +// It spawns a subprocess, so it belongs in the integration tier +// (.claude/rules/go-testing.md), never the unit tier. +func RunBinary(t *testing.T, binaryPath string, opts ...Option) { + t.Helper() + + rep, err := CheckBinary(t.Context(), binaryPath, opts...) + if err != nil { + t.Fatalf("modeltest: %v", err) + } + report(t, rep) +} + +// report fails t for each violation and logs each skip. +func report(t *testing.T, rep Report) { + t.Helper() + + for _, f := range rep.Skips() { + t.Logf("SKIP %s: %s", f.Check, f.Message) + } + for _, f := range rep.Failures() { + t.Errorf("%s: %s", f.Check, f.Message) + } +} + +// Check runs the conformance suite against p in-process, over a real gRPC +// round trip on an in-memory listener, and returns what it found. +// +// The round trip is the point: most of what this suite checks lives in +// the pkg/model service adapter and the generated wire types — terminal +// event bookkeeping, error-to-status mapping, the conversion layer — and +// a direct method call on p would exercise none of it. +// +// Returning a Report rather than driving *testing.T is what lets a +// non-test binary reuse these assertions, and what lets the suite's own +// tests prove it rejects a bad provider. A conformance suite that cannot +// be shown to fail is worth very little. +func Check(ctx context.Context, p model.Provider, opts ...Option) Report { + cfg := resolve(opts) + cfg.inProcess = true + + // A fixed identity, deliberately NOT the caller's expectation: in + // this mode modeltest supplies the identity itself, so serving the + // expected one would make WithExpectedIdentity a check that compares + // a value against itself and can never fail. + identity := plugin.Identity{ + Name: "modeltest", + Version: "0.0.0", + Source: "github.com/pluggableharness/agent/pkg/model/modeltest", + } + + lis := bufconn.Listen(bufSize) + svc := model.NewService(p, identity, plugin.NewCallback()) + + gs := grpc.NewServer() + svc.Register(gs) + go func() { _ = gs.Serve(lis) }() + defer gs.Stop() + + dialer := func(context.Context, string) (net.Conn, error) { return lis.Dial() } + conn, err := grpc.NewClient("passthrough:///modeltest", + grpc.WithContextDialer(dialer), + grpc.WithTransportCredentials(insecure.NewCredentials())) + if err != nil { + return Report{Findings: []Finding{{ + Check: "dial", + Severity: SeverityFail, + Message: fmt.Sprintf("could not dial the in-process listener: %v", err), + }}} + } + defer func() { _ = conn.Close() }() + + return runSuite(ctx, modelv1.NewModelServiceClient(conn), cfg) +} + +// CheckBinary runs the conformance suite against an already-built plugin +// binary, launching it the way the kernel does: a real handshake, a real +// subprocess, a real dispense. +// +// This is the only mode that exercises the plugin's own main() wiring — +// its handshake config, its Serve call, its identity stamping — and the +// only one that works on a plugin written in a language other than Go, +// since it speaks nothing but the wire protocol. +// +// The returned error is for a failure to launch at all, which is distinct +// from a conformance violation: a binary that will not start has not +// failed the suite, it has failed to be tested. +func CheckBinary(ctx context.Context, binaryPath string, opts ...Option) (Report, error) { + cfg := resolve(opts) + client, cleanup, err := launch(ctx, binaryPath) + if err != nil { + return Report{}, err + } + defer cleanup() + return runSuite(ctx, client, cfg), nil +} + +// launch starts binaryPath as a go-plugin subprocess and returns a client +// for its ModelService plus a teardown func. +func launch(ctx context.Context, binaryPath string) (modelv1.ModelServiceClient, func(), error) { + // The subprocess is bound to a context derived from the caller's, so + // an abandoned run tears the process down rather than leaking it. + ctx, cancel := context.WithCancel(ctx) + + categoryKey := common.PluginKey(commonv1.Category_CATEGORY_MODEL) + client := goplugin.NewClient(&goplugin.ClientConfig{ + HandshakeConfig: common.Handshake, + Plugins: goplugin.PluginSet{categoryKey: &clientPlugin{}}, + Cmd: commandContext(ctx, binaryPath), + AllowedProtocols: []goplugin.Protocol{goplugin.ProtocolGRPC}, + Logger: discardLogger(), + }) + cleanup := func() { + client.Kill() + cancel() + } + + rpc, err := client.Client() + if err != nil { + cleanup() + return nil, nil, fmt.Errorf("modeltest: handshake with %s: %w", binaryPath, err) + } + + // The negotiated version gate is checked here for the same reason the + // kernel checks it before the first category RPC: a version mismatch + // is a startup error, not a mystery failure on some later call. + if got := client.NegotiatedVersion(); got != int(common.ProtocolVersion) { + cleanup() + return nil, nil, fmt.Errorf("modeltest: protocol version mismatch: plugin=%d, this SDK=%d", got, common.ProtocolVersion) + } + + raw, err := rpc.Dispense(categoryKey) + if err != nil { + cleanup() + return nil, nil, fmt.Errorf("modeltest: dispense %q: %w", categoryKey, err) + } + got, ok := raw.(modelv1.ModelServiceClient) + if !ok { + cleanup() + return nil, nil, fmt.Errorf("modeltest: dispensed %T, want a ModelServiceClient — is this binary a model provider?", raw) + } + return got, cleanup, nil +} + +// clientPlugin adapts the dispensed connection into a ModelServiceClient. +// It never serves, so GRPCServer is not reachable in this direction. +type clientPlugin struct { + goplugin.Plugin +} + +var _ goplugin.GRPCPlugin = (*clientPlugin)(nil) + +// GRPCServer is never called: this adapter only ever runs kernel-side. +func (*clientPlugin) GRPCServer(*goplugin.GRPCBroker, *grpc.Server) error { + return errServeUnsupported +} + +// GRPCClient returns the generated client over the dispensed connection. +func (*clientPlugin) GRPCClient(_ context.Context, _ *goplugin.GRPCBroker, conn *grpc.ClientConn) (any, error) { + return modelv1.NewModelServiceClient(conn), nil +} diff --git a/pkg/model/modeltest/runbinary_integration_test.go b/pkg/model/modeltest/runbinary_integration_test.go new file mode 100644 index 0000000..4a9d65b --- /dev/null +++ b/pkg/model/modeltest/runbinary_integration_test.go @@ -0,0 +1,67 @@ +//go:build integration + +package modeltest_test + +import ( + "os" + "os/exec" + "path/filepath" + "testing" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +// TestRunBinary_againstTheExampleProvider drives the conformance suite +// through a real handshake and subprocess, against a binary built from +// examples/provider. +// +// This is the only path that exercises a plugin's own main() wiring — its +// handshake config, its Serve call, its identity stamping — none of which +// the in-process mode touches. It is also the mode a plugin written in +// another language would be checked by, since it speaks nothing but the +// wire protocol. +// +// The example is used as the subject because it is already this +// repository's proof that pkg/ works from outside the main module; +// running the suite against it closes the loop. +func TestRunBinary_againstTheExampleProvider(t *testing.T) { + binary := buildExampleProvider(t) + modeltest.RunBinary(t, binary) +} + +// TestRunBinary_reportsALaunchFailureDistinctly asserts that a binary +// which cannot start is reported as a launch error rather than as a +// conformance violation. The two are genuinely different: a binary that +// will not run has not failed the suite, it has failed to be tested. +func TestRunBinary_reportsALaunchFailureDistinctly(t *testing.T) { + t.Parallel() + + _, err := modeltest.CheckBinary(t.Context(), filepath.Join(t.TempDir(), "does-not-exist")) + if err == nil { + t.Fatal("CheckBinary() = nil error for a missing binary, want a launch failure") + } +} + +// buildExampleProvider compiles examples/provider into the repo's bin/ +// and returns the path. +// +// bin/, not t.TempDir(): the project CLAUDE.md's "bin/ only, no +// exceptions" rule covers test fixtures too, even where a temp dir would +// be the obvious choice. +func buildExampleProvider(t *testing.T) string { + t.Helper() + + root, err := filepath.Abs(filepath.Join("..", "..", "..")) + if err != nil { + t.Fatalf("resolve repo root: %v", err) + } + out := filepath.Join(root, "bin", "modeltest-example-provider") + + cmd := exec.CommandContext(t.Context(), "go", "build", "-o", out, ".") + cmd.Dir = filepath.Join(root, "examples", "provider") + if combined, err := cmd.CombinedOutput(); err != nil { + t.Fatalf("building the example provider: %v\n%s", err, combined) + } + t.Cleanup(func() { _ = os.Remove(out) }) + return out +} diff --git a/pkg/model/modeltest/stream.go b/pkg/model/modeltest/stream.go new file mode 100644 index 0000000..8866477 --- /dev/null +++ b/pkg/model/modeltest/stream.go @@ -0,0 +1,386 @@ +package modeltest + +import ( + "context" + "errors" + "io" + + "google.golang.org/grpc/codes" + + contentv1 "github.com/pluggableharness/agent/pkg/content/proto/v1" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// collected is what one drained StreamCompletion produced. +type collected struct { + events []*modelv1.StreamEvent + stops int + errs int + terminals int + // lastIsTerminal reports whether the final event was the terminal one, + // which is how "exactly one terminal event" is checked as a position + // rather than only a count. + lastIsTerminal bool + usage *modelv1.Usage + err error +} + +// drain reads a stream to completion, recording what it saw. +func drain(stream modelv1.ModelService_StreamCompletionClient) collected { + var got collected + for { + ev, err := stream.Recv() + if errors.Is(err, io.EOF) { + return got + } + if err != nil { + got.err = err + return got + } + got.events = append(got.events, ev) + + terminal := false + switch e := ev.GetEvent().(type) { + case *modelv1.StreamEvent_Stop_: + got.stops++ + terminal = true + case *modelv1.StreamEvent_Error_: + got.errs++ + terminal = true + case *modelv1.StreamEvent_Usage: + got.usage = e.Usage + } + if terminal { + got.terminals++ + } + got.lastIsTerminal = terminal + } +} + +// checkStream drives one completion and asserts the stream's shape. +func checkStream(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config, spec *modelv1.ModelSpec) { + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + stream, err := client.StreamCompletion(ctx, cfg.streamRequestFor(spec.GetId())) + if err != nil { + rec.failf("implemented", "StreamCompletion is a MUST and returned %v", err) + return + } + got := drain(stream) + + if got.err != nil && statusCode(got.err) != codes.OK { + rec.failf("transport", "the stream ended with a transport error: %v", got.err) + return + } + + // A stream that ends without a terminal event looks to the kernel like + // a clean turn that produced nothing, which is indistinguishable from + // success and therefore worse than a failure. + if got.terminals == 0 { + rec.failf("terminal-event", "the stream produced no terminal event; exactly one stop or error is required") + return + } + if got.terminals > 1 { + rec.failf("terminal-event", "the stream produced %d terminal events, want exactly 1", got.terminals) + } + if !got.lastIsTerminal { + rec.failf("terminal-event", "the terminal event was not the last event on the stream") + } + + if got.errs > 0 { + checkErrorEvents(rec, got.events) + // An error terminal is a legitimate outcome — the provider may be + // pointed at a fixture that fails — so the remaining assertions, + // which describe a successful completion, do not apply. + return + } + + if got.usage == nil { + rec.failf("usage", "the stream produced no usage event, so the kernel has no token counts to compute cost from") + } + checkStopReason(rec, got.events) + checkOpportunistic(rec, got.events, spec) +} + +// checkStopReason asserts the terminal stop names a real reason. +func checkStopReason(rec *recorder, events []*modelv1.StreamEvent) { + for _, ev := range events { + stop, ok := ev.GetEvent().(*modelv1.StreamEvent_Stop_) + if !ok { + continue + } + reason := stop.Stop.GetReason() + if reason == modelv1.StopReason_STOP_REASON_UNSPECIFIED { + rec.failf("stop-reason", "the stop event carries STOP_REASON_UNSPECIFIED, which tells the kernel nothing about why the turn ended") + } + // matched_stop_sequence is set iff the reason is STOP_SEQUENCE. + if reason == modelv1.StopReason_STOP_REASON_STOP_SEQUENCE && stop.Stop.GetMatchedStopSequence() == "" { + rec.failf("stop-reason", "the stop reason is STOP_SEQUENCE but matched_stop_sequence is empty") + } + if reason != modelv1.StopReason_STOP_REASON_STOP_SEQUENCE && stop.Stop.MatchedStopSequence != nil { + rec.failf("stop-reason", "matched_stop_sequence is set alongside reason %v, where it is meaningless", reason) + } + } +} + +// checkErrorEvents asserts every in-band error is classified. +func checkErrorEvents(rec *recorder, events []*modelv1.StreamEvent) { + for _, ev := range events { + e, ok := ev.GetEvent().(*modelv1.StreamEvent_Error_) + if !ok { + continue + } + modelErr := e.Error.GetError() + if modelErr.GetCategory() == modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNSPECIFIED { + rec.failf("error-category", "an error event carries an unspecified category; the kernel's retry and fallback behavior depends on telling categories apart") + } + if modelErr.GetMessage() == "" { + rec.failf("error-message", "an error event carries no message") + } + if modelErr.GetCategory() == modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_UNKNOWN && modelErr.GetRawDetail() == "" { + rec.failf("error-raw-detail", "an UNKNOWN error omits raw_detail, leaving nothing to debug from") + } + } +} + +// checkOpportunistic asserts the handling of content the suite cannot +// force a vendor to produce. Each check reports itself as skipped when the +// run did not reach it, so an unexercised check never reads as a pass. +func checkOpportunistic(rec *recorder, events []*modelv1.StreamEvent, spec *modelv1.ModelSpec) { + func() { + rec := rec.sub("RedactedThinking") + var seen int + for _, ev := range events { + r, ok := ev.GetEvent().(*modelv1.StreamEvent_RedactedThinking_) + if !ok { + continue + } + seen++ + // The block is opaque and carries a vendor integrity check. An + // empty payload means the adapter dropped or mangled it, which + // typically makes the vendor reject the WHOLE conversation on + // the next turn — a delayed, silent failure. + if len(r.RedactedThinking.GetData()) == 0 { + rec.failf("verbatim", "a redacted_thinking event carries no data; the block must be forwarded verbatim") + } + } + if seen == 0 { + rec.skipf("verbatim", "no redacted_thinking block was produced; supply a request that triggers one via WithStreamRequest to exercise this") + return + } + if !spec.GetThinking().GetSupported() { + rec.failf("capability", "a redacted_thinking block was produced by a model declaring no thinking capability") + } + }() + + func() { + rec := rec.sub("ToolCallPairing") + starts := map[string]string{} + var dones int + for _, ev := range events { + switch e := ev.GetEvent().(type) { + case *modelv1.StreamEvent_ToolCallStart_: + if e.ToolCallStart.GetId() == "" { + rec.failf("ids", "a tool_call_start carries no id, so its deltas cannot be correlated") + } + if e.ToolCallStart.GetName() == "" { + rec.failf("ids", "a tool_call_start carries no tool name") + } + starts[e.ToolCallStart.GetId()] = e.ToolCallStart.GetName() + case *modelv1.StreamEvent_ToolCallDelta_: + if _, ok := starts[e.ToolCallDelta.GetId()]; !ok { + rec.failf("pairing", "tool_call_delta for id %q has no preceding tool_call_start", e.ToolCallDelta.GetId()) + } + case *modelv1.StreamEvent_ToolCallDone_: + dones++ + if _, ok := starts[e.ToolCallDone.GetId()]; !ok { + rec.failf("pairing", "tool_call_done for id %q has no preceding tool_call_start", e.ToolCallDone.GetId()) + } + } + } + if len(starts) == 0 { + rec.skipf("pairing", "no tool call was produced; declare a tool via WithStreamRequest to exercise this") + return + } + if dones != len(starts) { + rec.failf("pairing", "%d tool calls started but %d completed; every started call must be closed", len(starts), dones) + } + }() + + func() { + rec := rec.sub("RateLimits") + var usage *modelv1.Usage + for _, ev := range events { + if u, ok := ev.GetEvent().(*modelv1.StreamEvent_Usage); ok { + usage = u.Usage + } + } + if usage == nil || len(usage.GetRateLimits()) == 0 { + rec.skipf("kind", "the vendor published no rate-limit state") + return + } + for _, rl := range usage.GetRateLimits() { + if rl.GetKind() == modelv1.RateLimitKind_RATE_LIMIT_KIND_UNSPECIFIED { + rec.failf("kind", "a rate-limit snapshot names no budget kind; \"you have 2%% left\" is unactionable without saying 2%% of what") + } + } + }() +} + +// checkCancellation asserts a canceled stream is not reported as a +// failure. +// +// This check is weaker than it appears, and saying so is more useful than +// implying otherwise. Two things limit it, both properties of gRPC rather +// than of any provider: +// +// - Once a client cancels, its own Recv returns immediately with +// codes.Canceled. It does not wait for the server, so a provider that +// ignores its context and keeps generating — still billing the +// operator for a turn the kernel has abandoned — is invisible from +// this side. Detecting that needs the server's own view, which a +// black-box conformance client does not have; it stays a code-review +// item, and RunBinary cannot check it at all. +// - For the same reason, an application error the provider returns +// *after* cancellation is masked by codes.Canceled before it reaches +// here. +// +// What remains is still worth asserting: the stream must start, and +// whatever terminal code does surface must be a cancellation rather than +// a failure. A provider that mishandles cancellation upstream of its own +// return — classifying it as an error before the stream unwinds — is +// caught here, and that is the common shape of the mistake. +func checkCancellation(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config, modelID string) { + streamCtx, cancelStream := context.WithCancel(ctx) + defer cancelStream() + + stream, err := client.StreamCompletion(streamCtx, cfg.streamRequestFor(modelID)) + if err != nil { + rec.failf("stream", "StreamCompletion returned %v", err) + return + } + cancelStream() + + for { + _, recvErr := stream.Recv() + if recvErr == nil { + continue + } + if errors.Is(recvErr, io.EOF) { + return + } + switch statusCode(recvErr) { + case codes.Canceled, codes.DeadlineExceeded: + return + default: + rec.failf("not-an-error", + "a canceled stream reported %v (%v); cancellation is normal control flow, never an application error", + statusCode(recvErr), recvErr) + return + } + } +} + +// checkUnknownModel asserts an unroutable model id is rejected as a +// malformed request rather than, say, silently served by a default. +func checkUnknownModel(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config) { + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + const bogus = "modeltest-no-such-model" + stream, err := client.StreamCompletion(ctx, &modelv1.StreamCompletionRequest{ModelId: bogus}) + if err != nil { + assertInvalidArgument(rec, err, "an unknown model id") + return + } + got := drain(stream) + if got.err != nil { + assertInvalidArgument(rec, got.err, "an unknown model id") + return + } + for _, ev := range got.events { + if e, ok := ev.GetEvent().(*modelv1.StreamEvent_Error_); ok { + if e.Error.GetError().GetCategory() != modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST { + rec.failf("rejected", "an unknown model id produced category %v, want INVALID_REQUEST", e.Error.GetError().GetCategory()) + } + return + } + } + rec.failf("rejected", "an unknown model id was served without an error; the kernel would attribute the result to a model that does not exist") +} + +// checkCapabilityGates asserts content a model declares it cannot accept +// is rejected rather than silently dropped. +func checkCapabilityGates(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config, spec *modelv1.ModelSpec) { + gates := []struct { + name string + supported bool + block *contentv1.ContentBlock + }{ + { + name: "image", + supported: spec.GetSupportsVision(), + block: &contentv1.ContentBlock{Block: &contentv1.ContentBlock_Image{ + Image: &contentv1.ImageBlock{MediaType: "image/png", Data: []byte{0x89, 'P', 'N', 'G'}}, + }}, + }, + { + name: "document", + supported: spec.GetSupportsDocuments(), + block: &contentv1.ContentBlock{Block: &contentv1.ContentBlock_Document{ + Document: &contentv1.DocumentBlock{MediaType: "application/pdf", Data: []byte("%PDF-")}, + }}, + }, + } + + for _, g := range gates { + func() { + rec := rec.sub(g.name) + if g.supported { + rec.skipf("rejected", "this model declares %s support, so there is no rejection to check", g.name) + return + } + + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + req := &modelv1.StreamCompletionRequest{ + ModelId: spec.GetId(), + Messages: []*contentv1.Message{{ + Role: contentv1.Role_ROLE_USER, + Content: []*contentv1.ContentBlock{g.block}, + }}, + } + stream, err := client.StreamCompletion(ctx, req) + if err != nil { + assertInvalidArgument(rec, err, g.name+" sent to a model that does not support it") + return + } + got := drain(stream) + if got.err != nil { + assertInvalidArgument(rec, got.err, g.name+" sent to a model that does not support it") + return + } + for _, ev := range got.events { + if e, ok := ev.GetEvent().(*modelv1.StreamEvent_Error_); ok { + if e.Error.GetError().GetCategory() != modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST { + rec.failf("rejected", "the %s rejection used category %v, want INVALID_REQUEST", g.name, e.Error.GetError().GetCategory()) + } + return + } + } + rec.failf("rejected", + "a %s block was accepted by a model declaring no %s support; it MUST be rejected, not silently dropped", + g.name, g.name) + }() + } +} + +// assertInvalidArgument checks err carries the code the taxonomy maps +// invalid_request to. +func assertInvalidArgument(rec *recorder, err error, what string) { + if got := statusCode(err); got != codes.InvalidArgument { + rec.failf("rejected", "%s produced %v (%v), want codes.InvalidArgument", what, got, err) + } +} diff --git a/pkg/model/modeltest/suite.go b/pkg/model/modeltest/suite.go new file mode 100644 index 0000000..a11de84 --- /dev/null +++ b/pkg/model/modeltest/suite.go @@ -0,0 +1,320 @@ +package modeltest + +import ( + "context" + "slices" + "time" + + "google.golang.org/grpc/codes" + grpcstatus "google.golang.org/grpc/status" + + commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" + modelv1 "github.com/pluggableharness/agent/pkg/model/proto/v1" +) + +// DefaultCallTimeout bounds every RPC the suite issues, so a provider +// that hangs produces a clear finding rather than stalling until the +// caller's own deadline. Override it with WithCallTimeout. +const DefaultCallTimeout = 30 * time.Second + +// runSuite drives every check against client and returns what it found. +// Both the in-process and subprocess entry points funnel here, so the two +// modes can never drift apart in what they assert. +func runSuite(ctx context.Context, client modelv1.ModelServiceClient, cfg *config) Report { + var findings []Finding + rec := newRecorder(&findings) + + checkDescribe(ctx, rec.sub("Describe"), client, cfg) + + caps := checkCapabilities(ctx, rec.sub("Capabilities"), client, cfg) + if caps == nil { + // Every remaining check needs a model to address. Continuing would + // report a cascade of findings that all have one cause. + return Report{Findings: findings} + } + + if !checkConfigure(ctx, rec.sub("Configure"), client, cfg) { + return Report{Findings: findings} + } + + if cfg.skipStream { + rec.sub("StreamCompletion").skipf("behavioral", + "disabled by WithoutStreamCompletion; every behavioral requirement is unchecked") + return Report{Findings: findings} + } + + modelID := cfg.modelID + if modelID == "" { + modelID = caps.GetModels()[0].GetId() + } + spec := specByID(caps.GetModels(), modelID) + if spec == nil { + rec.failf("model-selection", "the selected model %q is not advertised by this provider", modelID) + return Report{Findings: findings} + } + + checkStream(ctx, rec.sub("StreamCompletion"), client, cfg, spec) + checkCancellation(ctx, rec.sub("Cancellation"), client, cfg, modelID) + checkUnknownModel(ctx, rec.sub("UnknownModel"), client, cfg) + checkCapabilityGates(ctx, rec.sub("CapabilityGates"), client, cfg, spec) + + return Report{Findings: findings} +} + +// checkDescribe asserts the plugin reports a well-formed identity. +func checkDescribe(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config) { + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + resp, err := client.Describe(ctx, &modelv1.DescribeRequest{}) + if err != nil { + rec.failf("implemented", "Describe is a MUST and returned %v", err) + return + } + p := resp.GetProducer() + if p.GetName() == "" { + rec.failf("name", "Describe reported an empty name; a dev_overrides binary has no lock-file entry to fall back on") + } + if p.GetCategory() != commonv1.Category_CATEGORY_MODEL { + rec.failf("category", "Describe reported category %v, want CATEGORY_MODEL", p.GetCategory()) + } + if cfg.inProcess && cfg.identity != (identityExpectation{}) { + // The identity a plugin reports is a property of its own binary. + // In-process, modeltest supplies it, so there is nothing of the + // provider's to verify — say so rather than passing a check that + // only ever compares modeltest against itself. + rec.skipf("expected-identity", + "identity expectations apply to a built binary; in-process the identity is modeltest's own. Use RunBinary or CheckBinary") + return + } + if want := cfg.identity.name; want != "" && p.GetName() != want { + rec.failf("expected-name", "Describe reported name %q, want %q", p.GetName(), want) + } + if want := cfg.identity.version; want != "" && p.GetVersion() != want { + rec.failf("expected-version", "Describe reported version %q, want %q", p.GetVersion(), want) + } + if want := cfg.identity.source; want != "" && p.GetSource() != want { + rec.failf("expected-source", "Describe reported source %q, want %q", p.GetSource(), want) + } +} + +// checkCapabilities asserts the advertisement's own invariants and +// returns it, or nil when it is unusable. +func checkCapabilities(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config) *modelv1.Capabilities { + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + resp, err := client.GetCapabilities(ctx, &modelv1.GetCapabilitiesRequest{}) + if err != nil { + rec.failf("implemented", "GetCapabilities is a MUST and returned %v", err) + return nil + } + caps := resp.GetCapabilities() + if len(caps.GetModels()) == 0 { + rec.failf("models", "no models are advertised, which makes this provider unroutable") + return nil + } + if caps.GetConfigSchema() == nil { + rec.failf("config-schema", "no ConfigSchema is advertised; the kernel needs it before it can call Configure") + } + + seen := make(map[string]bool, len(caps.GetModels())) + for i, spec := range caps.GetModels() { + id := spec.GetId() + if id == "" { + rec.failf("model-id", "the model at index %d has an empty id", i) + continue + } + if seen[id] { + rec.failf("model-id", "model id %q is advertised more than once, so routing to it is ambiguous", id) + } + seen[id] = true + checkModelSpec(rec.sub(id), spec) + } + return caps +} + +// checkModelSpec asserts one advertised model's declarative invariants. +func checkModelSpec(rec *recorder, spec *modelv1.ModelSpec) { + if spec.GetContextWindow() <= 0 { + rec.failf("context-window", "context_window is %d, want a positive budget", spec.GetContextWindow()) + } + if spec.GetMaxOutputTokens() <= 0 { + rec.failf("max-output-tokens", "max_output_tokens is %d, want a positive ceiling", spec.GetMaxOutputTokens()) + } + checkThinkingSpec(rec.sub("thinking"), spec.GetThinking()) + checkCachingSpec(rec.sub("caching"), spec.GetCaching()) + checkPricing(rec.sub("pricing"), spec.GetPricing(), spec.GetCaching().GetSupported()) +} + +// checkThinkingSpec asserts ThinkingSpec's per-axis invariants +// (docs/specifications/model/data-types.md#thinkingspec). +func checkThinkingSpec(rec *recorder, ts *modelv1.ThinkingSpec) { + if !ts.GetSupported() { + if ts.GetEffort() != nil || ts.GetBudget() != nil { + rec.failf("unsupported", "thinking is unsupported but a reasoning control is declared") + } + if ts.GetAdaptiveByDefault() { + rec.failf("unsupported", "thinking is unsupported but adaptive_by_default is set") + } + switch ts.GetDisable() { + case modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_ALWAYS, + modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_CONDITIONAL: + rec.failf("unsupported", "thinking is unsupported but disable claims reasoning can be turned off") + default: + } + return + } + + if ts.GetDisable() == modelv1.ThinkingDisableSupport_THINKING_DISABLE_SUPPORT_UNSPECIFIED { + rec.failf("disable", "thinking is supported but disable is unset, so the kernel cannot tell whether it may turn reasoning off") + } + if e := ts.GetEffort(); e != nil { + switch { + case len(e.GetLevels()) == 0: + rec.failf("effort", "an effort control is declared with no levels; a model without one omits the control instead") + case e.GetDefault() == "": + rec.failf("effort", "an effort control is declared with no default level") + case !slices.Contains(e.GetLevels(), e.GetDefault()): + // The default exists so the kernel can send it as an explicit + // override; naming a level the vendor rejects makes that + // override a guaranteed error. + rec.failf("effort", "the effort default %q is not one of the declared levels %v", e.GetDefault(), e.GetLevels()) + } + } + if b := ts.GetBudget(); b != nil { + r := b.GetRange() + switch { + case r == nil: + rec.failf("budget", "a budget control is declared with no range") + case r.GetMin() > r.GetMax(): + rec.failf("budget", "the budget range [%d, %d] is inverted", r.GetMin(), r.GetMax()) + case r.GetMax() <= 0: + // A control admitting only a budget of zero declares a + // capability no caller can use; a model with no budget control + // omits it instead. + rec.failf("budget", "the budget range [%d, %d] admits no usable budget", r.GetMin(), r.GetMax()) + case b.Default != nil && (b.GetDefault() < r.GetMin() || b.GetDefault() > r.GetMax()): + rec.failf("budget", "the budget default %d is outside the declared range [%d, %d]", b.GetDefault(), r.GetMin(), r.GetMax()) + } + } +} + +// checkCachingSpec asserts CachingSpec's per-axis invariants. +func checkCachingSpec(rec *recorder, cs *modelv1.CachingSpec) { + if !cs.GetSupported() { + if cs.GetExplicitMarkers() || cs.GetImplicitAutomatic() { + rec.failf("unsupported", "caching is unsupported but a caching mechanism is declared") + } + return + } + if !cs.GetExplicitMarkers() && !cs.GetImplicitAutomatic() { + rec.failf("mechanism", "caching is supported but neither mechanism is declared, which reads as no caching to every caller") + } +} + +// checkPricing asserts Pricing's coverage invariant: exactly one tier +// matches any (timestamp, input_token_count) pair. +func checkPricing(rec *recorder, p *modelv1.Pricing, cachingSupported bool) { + if p == nil { + rec.failf("present", "no pricing is declared; it is required on every model, including a free one") + return + } + if p.GetCurrency() == "" { + rec.failf("currency", "no pricing currency is declared") + } + if p.GetFree() { + return + } + if len(p.GetTiers()) == 0 { + rec.failf("tiers", "no pricing tiers are declared and the model is not marked free") + return + } + + for i, tier := range p.GetTiers() { + if tier.GetInputPerMtok() < 0 || tier.GetOutputPerMtok() < 0 { + rec.failf("rates", "tier %d declares a negative rate", i) + } + if cachingSupported && (tier.CacheWritePerMtok == nil || tier.CacheReadPerMtok == nil) { + rec.failf("cache-rates", "tier %d omits a cache rate on a model that supports caching, so cached turns cannot be priced", i) + } + } + + // Overlap makes cost non-deterministic, and the kernel persists + // cost_usd at usage-event time — so a wrong pick is wrong in the + // ledger forever, with nothing to notice it by. + tiers := p.GetTiers() + for i := range tiers { + for j := i + 1; j < len(tiers); j++ { + if tiersOverlap(tiers[i], tiers[j]) { + rec.failf("tier-overlap", + "tiers %d and %d overlap; exactly one tier must match any (timestamp, input_token_count) pair", i, j) + } + } + } +} + +// tiersOverlap reports whether a and b both match some (timestamp, +// input_token_count) pair. Both dimensions are half-open, so an overlap +// requires overlapping on both. +func tiersOverlap(a, b *modelv1.PricingTier) bool { + return timeRangesOverlap(a, b) && tokenRangesOverlap(a, b) +} + +func timeRangesOverlap(a, b *modelv1.PricingTier) bool { + // An absent bound is unbounded on that side. + aFrom, aUntil := a.GetEffectiveFrom(), a.GetEffectiveUntil() + bFrom, bUntil := b.GetEffectiveFrom(), b.GetEffectiveUntil() + if aUntil != nil && bFrom != nil && !aUntil.AsTime().After(bFrom.AsTime()) { + return false + } + if bUntil != nil && aFrom != nil && !bUntil.AsTime().After(aFrom.AsTime()) { + return false + } + return true +} + +func tokenRangesOverlap(a, b *modelv1.PricingTier) bool { + aFrom, aUntil := a.InputTokensFrom, a.InputTokensUntil + bFrom, bUntil := b.InputTokensFrom, b.InputTokensUntil + if aUntil != nil && bFrom != nil && *aUntil <= *bFrom { + return false + } + if bUntil != nil && aFrom != nil && *bUntil <= *aFrom { + return false + } + return true +} + +// checkConfigure calls Configure and reports whether it succeeded. A +// failure here stops the run, because every behavioral check afterwards +// would fail for this one reason. +func checkConfigure(ctx context.Context, rec *recorder, client modelv1.ModelServiceClient, cfg *config) bool { + ctx, cancel := context.WithTimeout(ctx, cfg.callTimeout) + defer cancel() + + if _, err := client.Configure(ctx, &modelv1.ConfigureRequest{Config: cfg.configure}); err != nil { + rec.failf("accepts-config", + "Configure returned %v; pass this provider's configuration with modeltest.WithConfig", err) + return false + } + return true +} + +// specByID finds the advertised model with the given id. +func specByID(models []*modelv1.ModelSpec, id string) *modelv1.ModelSpec { + for _, m := range models { + if m.GetId() == id { + return m + } + } + return nil +} + +// statusCode extracts a gRPC code from err, or codes.OK for nil. +func statusCode(err error) codes.Code { + if err == nil { + return codes.OK + } + return grpcstatus.Code(err) +} From 55565b2ca05b2e152f0a1e533819988963e50fcd Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:03:38 -0400 Subject: [PATCH 10/16] config: add a per-provider environment passthrough MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit internal/pluginruntime launches every plugin with PATH/HOME/TMPDIR and nothing else, deliberately: ambient inheritance would leak every variable the kernel holds, including secrets meant for other plugins, into every subprocess. Config.ExtraEnv existed to widen that per launch, but nothing ever populated it and there was no operator surface at all. The consequence was a hard block on real providers. A plugin behind a corporate proxy could not see HTTPS_PROXY, one resolving an ambient cloud credential chain could not see its SDK's variables, and pkg/telemetry's Bootstrap — which reads OTEL exporter config from the process environment — could never be configured, so a plugin could start spans that went nowhere. provider{} gains an environment{} block, declared per provider so one plugin's variables stay invisible to every other. The kernel lifts it out before decoding the rest against the provider's own ConfigSchema, since it is the one name in that body the kernel owns and leaving it in would collide with a provider declaring an attribute of the same name. Entries are emitted sorted: a subprocess environment assembled in Go map order differs run to run, which is what determinism.md exists to prevent. Two things this surfaced that were not obvious up front: HCL's remain body from PartialContent does NOT hide an already-consumed block from JustAttributes. validateSensitiveAttrs used JustAttributes, so the first working version rejected every provider that declared environment{} at all — including, and especially, the ones carrying a credential. It now asks for the specific sensitive attributes by name, and a test pins the secret-plus-environment combination end to end. The "=" and empty-name checks are unreachable from agent.hcl, because HCL's own grammar forbids a quoted argument name. They are kept as defense in depth and tested directly rather than through a fixture, with the reason recorded so a later reader does not mistake them for a reachable path. --- .../configuration/blocks-reference.md | 24 ++ internal/config/bridge.go | 14 +- internal/config/errors.go | 18 ++ internal/config/load.go | 15 +- internal/config/providerenv.go | 120 +++++++++ internal/config/providerenv_test.go | 249 ++++++++++++++++++ internal/config/types.go | 7 + internal/kernel/bringup.go | 1 + internal/pluginhost/supervisor.go | 14 + 9 files changed, 459 insertions(+), 3 deletions(-) create mode 100644 internal/config/providerenv.go create mode 100644 internal/config/providerenv_test.go diff --git a/docs/specifications/configuration/blocks-reference.md b/docs/specifications/configuration/blocks-reference.md index f97653b..1f230dd 100644 --- a/docs/specifications/configuration/blocks-reference.md +++ b/docs/specifications/configuration/blocks-reference.md @@ -53,6 +53,30 @@ provider "claude-md-reader" { - **Reserved convention name:** `token_budget` (integer). A context or memory provider's token cap is not a separate mechanism — it's this ordinary, reserved-by-convention field in the provider's own config, decoded the same way as any other attribute. See [`context/data-types.md#budget-mechanics`](../context/data-types.md#budget-mechanics). A provider outside those categories simply doesn't declare `token_budget` in its schema, and the field is absent for it. - `Configure` MUST reject with a structured error on missing required fields or an unresolvable `env(...)`, rather than deferring failure to first use. +### `environment { ... }` — the one kernel-owned block inside `provider{}` + +```hcl +provider "anthropic" { + api_key = env("ANTHROPIC_API_KEY") + + environment { + HTTPS_PROXY = env("HTTPS_PROXY") + NO_PROXY = "localhost,127.0.0.1" + } +} +``` + +`environment{}` declares environment variables the kernel passes to **that one plugin's** subprocess, on top of the launcher's own minimal allowlist. Every value MUST be a string; `env(...)` is available, since the value an operator wants to forward is usually already in their own environment and naming it beats copying it into a config file. + +This exists because the launcher deliberately never inherits the kernel's environment — ambient inheritance would leak every variable the kernel process holds, including secrets meant for other plugins, into every subprocess. That default is right, and it is also why a plugin behind a corporate proxy, or one resolving an ambient cloud credential chain, has no way to see `HTTPS_PROXY` or its SDK's own variables without a declared passthrough. Declaring it per provider rather than globally keeps one plugin's variables invisible to every other, which is the property the allowlist was protecting in the first place. + +Rules: + +- The kernel MUST lift this block out of the body **before** decoding the rest against the provider's `ConfigSchema`. It is the one name in a `provider{}` body the kernel owns, so leaving it in would collide with any provider that declares an attribute of the same name. +- A name that is not a usable POSIX environment variable — empty, or containing `=` (which would let one entry smuggle in a second) — MUST be rejected at config-load time. +- Entries MUST be assembled in a deterministic order. Go map iteration is randomized, and a subprocess whose environment differs run to run is the kind of nondeterminism [`.claude/rules/determinism.md`](../../../.claude/rules/determinism.md) exists to prevent. +- This is a passthrough, not a secret channel. A credential belongs in the provider's own `sensitive`-marked attribute, which the kernel resolves and delivers through `Configure`; a secret placed here is visible to anything that can read the process table on some platforms, and bypasses the secret-handling rules below. + A `provider{}` block's body is not decoded when `agent.hcl` is first loaded. A `ConfigSchema` only exists once the named plugin's subprocess is running and has answered `GetCapabilities`/`GetSchema`, so there is nothing to decode against at load time — a genuine chicken-and-egg constraint. The body is decoded later, once a schema is available — see the schema-to-`cty` bridge below. ### HCL single-line blocks take only one argument diff --git a/internal/config/bridge.go b/internal/config/bridge.go index 43a2c58..f9f3eaf 100644 --- a/internal/config/bridge.go +++ b/internal/config/bridge.go @@ -126,12 +126,22 @@ func validateSensitiveAttrs(body hcl.Body, sensitiveAttrs map[string]bool) error if len(sensitiveAttrs) == 0 { return nil } - attrs, diags := body.JustAttributes() + // PartialContent naming exactly the sensitive attributes, rather than + // JustAttributes: a provider{} body may legitimately contain the + // kernel-owned environment{} block, and JustAttributes rejects ANY + // block outright — including one already consumed by an earlier + // PartialContent, since HCL's remain body does not hide it. Asking for + // the specific attributes sidesteps that entirely. + schema := &hcl.BodySchema{Attributes: make([]hcl.AttributeSchema, 0, len(sensitiveAttrs))} + for name := range sensitiveAttrs { + schema.Attributes = append(schema.Attributes, hcl.AttributeSchema{Name: name}) + } + content, _, diags := body.PartialContent(schema) if diags.HasErrors() { return fmt.Errorf("config: %w", diags) } for name := range sensitiveAttrs { - attr, ok := attrs[name] + attr, ok := content.Attributes[name] if !ok { continue // absent optional sensitive attribute — nothing to validate } diff --git a/internal/config/errors.go b/internal/config/errors.go index dcfdbb0..0ff8116 100644 --- a/internal/config/errors.go +++ b/internal/config/errors.go @@ -18,3 +18,21 @@ var ErrInvalidValue = errors.New("config: invalid attribute value") // AttrType this package doesn't recognize (configuration.md §4's fixed // 7-value subset). var ErrInvalidAttrType = errors.New("config: invalid ConfigAttribute type") + +// ErrEnvNameEmpty reports an environment{} entry with an empty name, +// which would produce a malformed entry a subprocess silently ignores. +var ErrEnvNameEmpty = errors.New("config: environment variable name is empty") + +// ErrEnvNameInvalid reports an environment{} name that cannot be a POSIX +// environment variable — notably one containing "=", which would let a +// single entry smuggle in a second. +var ErrEnvNameInvalid = errors.New("config: invalid environment variable name") + +// ErrEnvValueNotString reports an environment{} value that is not a +// string. A subprocess environment carries only strings, and silently +// stringifying a number would hide the config error rather than fix it. +var ErrEnvValueNotString = errors.New("config: environment variable value must be a string") + +// ErrEnvValueUnusable reports an environment{} value that is null or not +// knowable at load time. +var ErrEnvValueUnusable = errors.New("config: environment variable value is null or unknown") diff --git a/internal/config/load.go b/internal/config/load.go index dcbc4bb..89b7f29 100644 --- a/internal/config/load.go +++ b/internal/config/load.go @@ -59,6 +59,7 @@ func decode(body hcl.Body) (*Config, error) { cfg := &Config{ RequiredProviders: map[string]RequiredProvider{}, ProviderBodies: map[string]hcl.Body{}, + ProviderEnv: map[string]map[string]string{}, ProviderRanges: map[string]hcl.Range{}, AgentProfiles: map[string]agentprofile.AgentProfile{}, // A config with no settings{} block at all never reaches @@ -87,7 +88,19 @@ func decode(body hcl.Body) (*Config, error) { if _, exists := cfg.ProviderBodies[name]; exists { return nil, fmt.Errorf("config: provider %q: %w", name, ErrDuplicateBlock) } - cfg.ProviderBodies[name] = block.Body + // The environment{} block is kernel-owned, so it is lifted out + // here and the REMAINING body is what later gets decoded against + // the provider's own ConfigSchema. Leaving it in would make it + // collide with any provider that declares an attribute of the + // same name. + env, remain, err := extractProviderEnv(block.Body) + if err != nil { + return nil, fmt.Errorf("config: provider %q: %w", name, err) + } + if len(env) > 0 { + cfg.ProviderEnv[name] = env + } + cfg.ProviderBodies[name] = remain cfg.ProviderRanges[name] = block.DefRange case "policy": diff --git a/internal/config/providerenv.go b/internal/config/providerenv.go new file mode 100644 index 0000000..064bfd8 --- /dev/null +++ b/internal/config/providerenv.go @@ -0,0 +1,120 @@ +package config + +import ( + "fmt" + "sort" + "strings" + + "github.com/hashicorp/hcl/v2" + "github.com/zclconf/go-cty/cty/function" + + "github.com/pluggableharness/agent/internal/hclsecret" +) + +// providerEnvBlockType is the kernel-owned block inside provider{} that +// names environment variables to pass through to that plugin's +// subprocess. +const providerEnvBlockType = "environment" + +// providerEnvSchema matches the environment{} block without consuming +// anything else, so the remaining body is still the provider's own. +var providerEnvSchema = &hcl.BodySchema{ + Blocks: []hcl.BlockHeaderSchema{{Type: providerEnvBlockType}}, +} + +// extractProviderEnv lifts the environment{} block out of a provider{} +// body, returning the resolved variables and the remaining body. +// +// The remaining body is what later gets decoded against the provider's +// own ConfigSchema. Splitting here rather than at decode time is what +// keeps a kernel-owned name from colliding with a provider that happens +// to declare an attribute called "environment". +// +// A body with no environment{} block yields a nil map and the body +// unchanged, which is the ordinary case. +func extractProviderEnv(body hcl.Body) (map[string]string, hcl.Body, error) { + content, remain, diags := body.PartialContent(providerEnvSchema) + if diags.HasErrors() { + return nil, nil, fmt.Errorf("environment: %w", diags) + } + if len(content.Blocks) == 0 { + return nil, body, nil + } + if len(content.Blocks) > 1 { + return nil, nil, fmt.Errorf("environment: %w", ErrDuplicateBlock) + } + + attrs, diags := content.Blocks[0].Body.JustAttributes() + if diags.HasErrors() { + return nil, nil, fmt.Errorf("environment: %w", diags) + } + + // env(...) is available here for the same reason it is in a provider's + // own attributes: the value an operator wants to forward is usually + // already in their own environment, and naming it beats copying it + // into a config file. + evalCtx := &hcl.EvalContext{ + Functions: map[string]function.Function{hclsecret.EnvFunctionName: hclsecret.EnvFunction}, + } + + out := make(map[string]string, len(attrs)) + for name, attr := range attrs { + if err := validateEnvName(name); err != nil { + return nil, nil, fmt.Errorf("environment: %w", err) + } + val, valDiags := attr.Expr.Value(evalCtx) + if valDiags.HasErrors() { + return nil, nil, fmt.Errorf("environment: %s: %w", name, valDiags) + } + if val.IsNull() || !val.IsKnown() { + return nil, nil, fmt.Errorf("environment: %s: %w", name, ErrEnvValueUnusable) + } + if val.Type().FriendlyName() != "string" { + return nil, nil, fmt.Errorf("environment: %s: %w", name, ErrEnvValueNotString) + } + out[name] = val.AsString() + } + return out, remain, nil +} + +// validateEnvName rejects a name that cannot be a POSIX environment +// variable. An "=" would let one entry smuggle in a second, and an empty +// name produces a malformed entry the subprocess silently ignores. +// +// Defense in depth rather than a reachable config path: HCL's own grammar +// forbids a quoted argument name, so neither case can be written in +// agent.hcl today. It is kept because the cost is one comparison and the +// failure it guards against is silent. +func validateEnvName(name string) error { + if name == "" { + return ErrEnvNameEmpty + } + if strings.ContainsAny(name, "=\x00") { + return fmt.Errorf("%w: %q", ErrEnvNameInvalid, name) + } + return nil +} + +// EnvEntries renders env as sorted "KEY=VALUE" entries, the shape +// exec.Cmd expects. +// +// Sorted because a subprocess's environment is otherwise assembled in Go +// map order, which is randomized — and a launch that differs run to run +// is exactly the kind of nondeterminism .claude/rules/determinism.md +// exists to prevent. +func EnvEntries(env map[string]string) []string { + if len(env) == 0 { + return nil + } + names := make([]string, 0, len(env)) + for name := range env { + names = append(names, name) + } + sort.Strings(names) + + out := make([]string, 0, len(names)) + for _, name := range names { + out = append(out, name+"="+env[name]) + } + return out +} diff --git a/internal/config/providerenv_test.go b/internal/config/providerenv_test.go new file mode 100644 index 0000000..b3c5442 --- /dev/null +++ b/internal/config/providerenv_test.go @@ -0,0 +1,249 @@ +package config + +import ( + "errors" + "slices" + "testing" + + "github.com/hashicorp/hcl/v2" + "github.com/hashicorp/hcl/v2/hclparse" + + configv1 "github.com/pluggableharness/agent/pkg/config/proto/v1" +) + +// providerBody parses src as agent.hcl and returns the named provider +// block's raw body. +func providerBody(t *testing.T, src string) hcl.Body { + t.Helper() + + f, diags := hclparse.NewParser().ParseHCL([]byte(src), "test.hcl") + if diags.HasErrors() { + t.Fatalf("parse: %v", diags) + } + content, _, diags := f.Body.PartialContent(topLevelSchema) + if diags.HasErrors() { + t.Fatalf("root content: %v", diags) + } + for _, block := range content.Blocks { + if block.Type == "provider" { + return block.Body + } + } + t.Fatal("no provider block in fixture") + return nil +} + +func TestExtractProviderEnv_liftsTheBlockAndLeavesTheRest(t *testing.T) { + t.Parallel() + + body := providerBody(t, ` +provider "anthropic" { + api_key = "sk-test" + + environment { + HTTPS_PROXY = "http://proxy.internal:3128" + NO_PROXY = "localhost" + } +} +`) + + env, remain, err := extractProviderEnv(body) + if err != nil { + t.Fatalf("extractProviderEnv: %v", err) + } + if got, want := env["HTTPS_PROXY"], "http://proxy.internal:3128"; got != want { + t.Errorf("HTTPS_PROXY = %q, want %q", got, want) + } + if got, want := env["NO_PROXY"], "localhost"; got != want { + t.Errorf("NO_PROXY = %q, want %q", got, want) + } + + // The remaining body must still carry the provider's own attributes. + // + // Read via PartialContent, not JustAttributes: HCL's remain body does + // NOT hide an already-consumed block, so JustAttributes still rejects + // the environment{} block outright. That is exactly why + // validateSensitiveAttrs in bridge.go asks for named attributes rather + // than all of them — a provider using environment{} would otherwise + // fail the secret check. + content, _, diags := remain.PartialContent(&hcl.BodySchema{ + Attributes: []hcl.AttributeSchema{{Name: "api_key"}}, + }) + if diags.HasErrors() { + t.Fatalf("remaining attributes: %v", diags) + } + if _, ok := content.Attributes["api_key"]; !ok { + t.Error("the provider's own api_key did not survive the split") + } +} + +func TestExtractProviderEnv_absentBlockLeavesTheBodyUntouched(t *testing.T) { + t.Parallel() + + body := providerBody(t, ` +provider "anthropic" { + api_key = "sk-test" +} +`) + + env, remain, err := extractProviderEnv(body) + if err != nil { + t.Fatalf("extractProviderEnv: %v", err) + } + if env != nil { + t.Errorf("env = %v, want nil for a provider declaring none", env) + } + if remain == nil { + t.Fatal("remain = nil, want the body unchanged") + } +} + +func TestExtractProviderEnv_resolvesEnvIndirection(t *testing.T) { + // Not parallel: t.Setenv forbids it. + t.Setenv("MODELTEST_PROXY", "http://resolved:3128") + + body := providerBody(t, ` +provider "anthropic" { + environment { + HTTPS_PROXY = env("MODELTEST_PROXY") + } +} +`) + + env, _, err := extractProviderEnv(body) + if err != nil { + t.Fatalf("extractProviderEnv: %v", err) + } + if got, want := env["HTTPS_PROXY"], "http://resolved:3128"; got != want { + t.Errorf("HTTPS_PROXY = %q, want %q", got, want) + } +} + +func TestExtractProviderEnv_rejectsUnusableDeclarations(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + src string + wantErr error + }{ + // A subprocess environment carries only strings; stringifying a + // number silently would hide the config error rather than fix it. + "non-string value": { + src: `provider "p" { + environment { + PORT = 8080 + } +}`, + wantErr: ErrEnvValueNotString, + }, + "null value": { + src: `provider "p" { + environment { + A = null + } +}`, + wantErr: ErrEnvValueUnusable, + }, + "two environment blocks": { + src: `provider "p" { + environment { + A = "1" + } + environment { + B = "2" + } +}`, + wantErr: ErrDuplicateBlock, + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + _, _, err := extractProviderEnv(providerBody(t, tt.src)) + if !errors.Is(err, tt.wantErr) { + t.Errorf("err = %v, want wrapping %v", err, tt.wantErr) + } + }) + } +} + +func TestValidateEnvName(t *testing.T) { + t.Parallel() + + // Tested directly rather than through a config fixture because HCL's + // own grammar forbids a quoted argument name, so neither of these is + // reachable from agent.hcl today. The check is defense in depth: an + // "=" would let a single entry smuggle in a second, and an empty name + // produces a malformed entry the subprocess silently ignores. + if err := validateEnvName(""); !errors.Is(err, ErrEnvNameEmpty) { + t.Errorf("validateEnvName(\"\") = %v, want ErrEnvNameEmpty", err) + } + if err := validateEnvName("A=B"); !errors.Is(err, ErrEnvNameInvalid) { + t.Errorf("validateEnvName(\"A=B\") = %v, want ErrEnvNameInvalid", err) + } + if err := validateEnvName("HTTPS_PROXY"); err != nil { + t.Errorf("validateEnvName(\"HTTPS_PROXY\") = %v, want nil", err) + } +} + +func TestEnvEntries_isSortedAndShaped(t *testing.T) { + t.Parallel() + + // Sorted because Go map order is randomized, and a subprocess whose + // environment differs run to run is exactly the nondeterminism + // determinism.md exists to prevent. + got := EnvEntries(map[string]string{"ZED": "3", "ALPHA": "1", "MID": "2"}) + want := []string{"ALPHA=1", "MID=2", "ZED=3"} + if !slices.Equal(got, want) { + t.Errorf("EnvEntries = %v, want %v", got, want) + } + if EnvEntries(nil) != nil { + t.Error("EnvEntries(nil) is non-nil, want nil so an absent block adds nothing") + } +} + +// TestExtractProviderEnv_coexistsWithASensitiveAttribute is the +// regression guard for the interaction that broke first: HCL's remain +// body does not hide an already-consumed block, so validateSensitiveAttrs +// reading the body with JustAttributes rejected every provider that used +// environment{} at all — including, and especially, the ones carrying a +// credential. +func TestExtractProviderEnv_coexistsWithASensitiveAttribute(t *testing.T) { + // Not parallel: t.Setenv forbids it. + t.Setenv("MODELTEST_KEY", "sk-from-the-environment") + + body := providerBody(t, ` +provider "anthropic" { + api_key = env("MODELTEST_KEY") + + environment { + HTTPS_PROXY = "http://proxy.internal:3128" + } +} +`) + + env, remain, err := extractProviderEnv(body) + if err != nil { + t.Fatalf("extractProviderEnv: %v", err) + } + if env["HTTPS_PROXY"] == "" { + t.Fatal("the environment block was not extracted") + } + + schema := &configv1.ConfigSchema{Attributes: []*configv1.ConfigAttribute{{ + Name: "api_key", + Type: configv1.AttrType_ATTR_TYPE_STRING, + Required: true, + Sensitive: true, + }}} + + decoded, err := DecodeProviderConfig(remain, schema) + if err != nil { + t.Fatalf("DecodeProviderConfig alongside an environment block: %v", err) + } + if got := decoded.GetFields()["api_key"].GetStringValue(); got != "sk-from-the-environment" { + t.Errorf("api_key = %q, want the resolved secret", got) + } +} diff --git a/internal/config/types.go b/internal/config/types.go index 74a9dae..ae0c36e 100644 --- a/internal/config/types.go +++ b/internal/config/types.go @@ -241,6 +241,13 @@ type Config struct { // DecodeProviderConfig once that provider's ConfigSchema is known. ProviderBodies map[string]hcl.Body + // ProviderEnv holds each provider{} block's environment{} entries, + // keyed by local name — the environment variables that provider's + // subprocess is launched with, on top of the launcher's own minimal + // allowlist. Absent for a provider that declared none, which is the + // ordinary case. + ProviderEnv map[string]map[string]string + // ProviderRanges holds each provider{} block's source position, // keyed the same way as ProviderBodies — for hook-ordering resolution // (configuration.md §8.6), which this package does not itself perform. diff --git a/internal/kernel/bringup.go b/internal/kernel/bringup.go index e9f670e..ab6e9cd 100644 --- a/internal/kernel/bringup.go +++ b/internal/kernel/bringup.go @@ -251,6 +251,7 @@ func (k *kernel) startPlugins(ctx context.Context) error { Sessions: k.sessions, Tokens: k.tokens, ProviderBodies: k.cfg.ProviderBodies, + ProviderEnv: k.cfg.ProviderEnv, BusSubscribeQueueBound: k.cfg.Settings.EventBus.SubscribeQueueBound, Logger: k.logger, }) diff --git a/internal/pluginhost/supervisor.go b/internal/pluginhost/supervisor.go index 32d59cc..8ddd819 100644 --- a/internal/pluginhost/supervisor.go +++ b/internal/pluginhost/supervisor.go @@ -115,6 +115,19 @@ type Config struct { // ordinary case for a provider that takes no config. ProviderBodies map[string]hcl.Body + // ProviderEnv is config.Config.ProviderEnv — each provider{} block's + // environment{} entries, keyed by local name. These are appended to + // the launcher's own minimal allowlist for that one subprocess. + // + // This is what makes a provider behind a corporate proxy, or one + // resolving an ambient cloud credential chain, configurable at all: + // internal/pluginruntime deliberately never inherits the kernel's + // environment, so without a declared passthrough such a plugin has no + // way to see HTTPS_PROXY or an SDK's own variables. Declared per + // provider rather than globally, so one plugin's variables are not + // visible to every other. + ProviderEnv map[string]map[string]string + // BusSubscribeQueueBound is the per-Subscribe-stream backpressure // bound passed through to every plugin's kernel-callback server // (configuration/blocks-reference.md#event_bus). A value <= 0 leaves @@ -389,6 +402,7 @@ func (s *Supervisor) launchConfig(resolved providerresolve.Resolved, category co Callback: slot, Telemetry: s.cfg.Telemetry, Logger: s.logger, + ExtraEnv: config.EnvEntries(s.cfg.ProviderEnv[resolved.LocalName]), } } From 294c81880bbcf0aec826433646bf1216b481d299 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:08:01 -0400 Subject: [PATCH 11/16] plugin: decouple category protocol versions from the handshake MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit common.v1.ProducerRef.protocol_version was documented as the go-plugin handshake version, with the rule that bumping it "always accompanies a proto package version bump for that category". That coupling is wrong, and the first v* tag would have frozen it. A handshake-version mismatch rejects a plugin before any category RPC is issued. So under the old rule, a breaking change to the model protocol would bump the shared handshake version and thereby reject every tool, context, and memory plugin ever published — none of which changed — until each was rebuilt. Categories already version independently at the proto level, where a v2 lands alongside v1 rather than replacing it, so the handshake was the one place that made them move in lockstep. Separate them. common.ProtocolVersion now versions only the runtime contract it actually describes: the handshake, the fixed callback broker id, and service muxing. Each category SDK carries its own ProtocolVersion constant, and Describe reports that one. Correctness never depended on the field: a category's proto package version is part of its gRPC service name, so a plugin serving pluggableharness.model.v2.ModelService and a kernel dispensing v1 already cannot match. What the field buys is a clear version error at bring-up instead of an opaque "unimplemented service" on the first real call, and a lock file that records it lets preflightVersionCheck — a documented no-op today — reject a plugin before spawning it at all. The identity test previously asserted the field equalled common.ProtocolVersion, which was the coupling expressed as a test. It now covers two categories reporting different protocol versions, which is the property that has to hold. --- .claude/rules/plugin-runtime.md | 30 +++++++++++++++++++--- api/pluggableharness/common/v1/types.proto | 23 ++++++++++++++--- pkg/common/plugin.go | 12 +++++++++ pkg/common/proto/v1/types.pb.go | 23 ++++++++++++++--- pkg/context/doc.go | 8 ++++++ pkg/context/server.go | 2 +- pkg/frontend/doc.go | 8 ++++++ pkg/frontend/server.go | 2 +- pkg/memory/doc.go | 8 ++++++ pkg/memory/server.go | 2 +- pkg/model/doc.go | 8 ++++++ pkg/model/server.go | 2 +- pkg/plugin/identity.go | 19 +++++++++----- pkg/plugin/identity_test.go | 19 +++++++++----- pkg/slashcommand/doc.go | 8 ++++++ pkg/slashcommand/server.go | 2 +- pkg/tool/doc.go | 8 ++++++ pkg/tool/server.go | 2 +- pkg/widget/doc.go | 8 ++++++ pkg/widget/server.go | 2 +- 20 files changed, 163 insertions(+), 33 deletions(-) diff --git a/.claude/rules/plugin-runtime.md b/.claude/rules/plugin-runtime.md index 80c6469..a8b56d6 100644 --- a/.claude/rules/plugin-runtime.md +++ b/.claude/rules/plugin-runtime.md @@ -15,10 +15,32 @@ covers RPC shape and `proto.md` covers wire typing. magic cookie key/value and a `ProtocolVersion` field. Do not give categories different cookies — the uniform handshake is what lets the kernel reject a mismatched-protocol plugin before ever calling into it. -- `ProtocolVersion` is bumped only on a breaking wire change (see `proto.md`'s - `buf breaking` rule) — bumping it and shipping a `v1`→`v2` proto package - bump happen together, never independently. -- The kernel-side plugin client always checks the negotiated protocol +- **Two different versions exist, and they move independently. Do not couple them.** + - `pkg/common.ProtocolVersion` is the go-plugin **handshake** version. It + versions the *runtime* contract only — the handshake, the fixed callback + broker id, and how services are muxed onto one connection. Bump it only + when one of those changes. + - Each category SDK's own `ProtocolVersion` constant (e.g. + `pkg/model.ProtocolVersion`) is that **category's** protocol version — + the `v1` in `pluggableharness.model.v1`. It is bumped alongside a + `v1`→`v2` proto package bump for that category, and for no other reason. + + The separation is load-bearing. A handshake-version mismatch rejects a + plugin *before any category RPC is issued*, so folding category versions + into the handshake would mean a breaking change in one category forced + every plugin of every other category ever published to rebuild in order + to keep working. An earlier revision of `common.v1.ProducerRef`'s comment + said a handshake bump "always accompanies a proto package version bump + for that category"; that was the coupling, and it is no longer the rule. +- **A category's version is already part of its gRPC service name**, so + correctness does not depend on any negotiation field: a plugin serving + `pluggableharness.model.v2.ModelService` and a kernel dispensing `v1` + simply do not match. `ProducerRef.protocol_version` exists so that + mismatch surfaces as a clear version error at bring-up rather than as an + opaque "unimplemented service" on the first real call — and so a lock + file recording it lets `preflightVersionCheck` reject a plugin before + spawning it at all. +- The kernel-side plugin client always checks the negotiated handshake version before issuing the first category RPC; a mismatch is a startup error, not a runtime error discovered on first call. diff --git a/api/pluggableharness/common/v1/types.proto b/api/pluggableharness/common/v1/types.proto index 49624c4..d8a9369 100644 --- a/api/pluggableharness/common/v1/types.proto +++ b/api/pluggableharness/common/v1/types.proto @@ -120,10 +120,25 @@ message ProducerRef { // Which of the seven plugin categories this producer implements. Category category = 4; - // The go-plugin handshake protocol version this producer build was - // compiled against (.claude/rules/plugin-runtime.md). A version bump - // here always accompanies a proto package version bump (v1 -> v2) for - // that category — it never happens independently of one. + // The version of THIS PRODUCER'S OWN CATEGORY protocol it implements — + // the "v1" in pluggableharness..v1, reported per category + // rather than globally. + // + // Deliberately NOT the go-plugin handshake version, which versions the + // runtime contract (handshake, callback broker, service muxing) that + // every category shares. The two move independently: coupling them + // would mean a breaking change in one category forced every plugin of + // every OTHER category to rebuild, because a handshake-version bump + // rejects a plugin before any category RPC is issued. + // + // Correctness does not depend on this field — a category's proto + // package version is part of its gRPC service name, so a plugin serving + // pluggableharness.model.v2.ModelService and a kernel dispensing v1 + // simply do not match. It exists so that mismatch is reported as a + // clear version error at bring-up rather than as an opaque + // "unimplemented service" on the first real call, and so a lock file + // recording it lets the kernel reject a plugin before spawning it at + // all (.claude/rules/plugin-runtime.md). uint32 protocol_version = 5; } diff --git a/pkg/common/plugin.go b/pkg/common/plugin.go index cc146c6..6691b57 100644 --- a/pkg/common/plugin.go +++ b/pkg/common/plugin.go @@ -19,6 +19,18 @@ import ( // is bumped only together with a breaking proto v1->v2 change // (.claude/rules/plugin-runtime.md's "Handshake" section) — never // independently. +// It versions the RUNTIME contract only — the handshake itself, the fixed +// callback broker id, and how services are muxed onto one connection — +// never any category's own protocol. Bump it only when one of those +// changes. +// +// This separation is load-bearing rather than tidy. A handshake-version +// mismatch rejects a plugin before any category RPC is issued, so folding +// category versions into it would mean a breaking change in, say, the +// model protocol forced every tool, context, and memory plugin ever +// published to rebuild in order to keep working. Each category SDK +// carries its own ProtocolVersion constant instead, reported per plugin +// through Describe's ProducerRef. const ProtocolVersion uint = 1 const ( diff --git a/pkg/common/proto/v1/types.pb.go b/pkg/common/proto/v1/types.pb.go index 47a998f..7ce2941 100644 --- a/pkg/common/proto/v1/types.pb.go +++ b/pkg/common/proto/v1/types.pb.go @@ -226,10 +226,25 @@ type ProducerRef struct { Source string `protobuf:"bytes,3,opt,name=source,proto3" json:"source,omitempty"` // Which of the seven plugin categories this producer implements. Category Category `protobuf:"varint,4,opt,name=category,proto3,enum=pluggableharness.common.v1.Category" json:"category,omitempty"` - // The go-plugin handshake protocol version this producer build was - // compiled against (.claude/rules/plugin-runtime.md). A version bump - // here always accompanies a proto package version bump (v1 -> v2) for - // that category — it never happens independently of one. + // The version of THIS PRODUCER'S OWN CATEGORY protocol it implements — + // the "v1" in pluggableharness..v1, reported per category + // rather than globally. + // + // Deliberately NOT the go-plugin handshake version, which versions the + // runtime contract (handshake, callback broker, service muxing) that + // every category shares. The two move independently: coupling them + // would mean a breaking change in one category forced every plugin of + // every OTHER category to rebuild, because a handshake-version bump + // rejects a plugin before any category RPC is issued. + // + // Correctness does not depend on this field — a category's proto + // package version is part of its gRPC service name, so a plugin serving + // pluggableharness.model.v2.ModelService and a kernel dispensing v1 + // simply do not match. It exists so that mismatch is reported as a + // clear version error at bring-up rather than as an opaque + // "unimplemented service" on the first real call, and so a lock file + // recording it lets the kernel reject a plugin before spawning it at + // all (.claude/rules/plugin-runtime.md). ProtocolVersion uint32 `protobuf:"varint,5,opt,name=protocol_version,json=protocolVersion,proto3" json:"protocol_version,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache diff --git a/pkg/context/doc.go b/pkg/context/doc.go index 57b330f..e3d538f 100644 --- a/pkg/context/doc.go +++ b/pkg/context/doc.go @@ -60,3 +60,11 @@ // which matters more here than avoiding "context.Request" reading // redundant under an aliased import. package context + +// ProtocolVersion is the version of the context category's own protocol this +// SDK implements — the "v1" in pluggableharness.context.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/context/server.go b/pkg/context/server.go index ed729e1..1b891ee 100644 --- a/pkg/context/server.go +++ b/pkg/context/server.go @@ -157,5 +157,5 @@ func (s *Service) Render(ctx context.Context, req *contextv1.RenderRequest) (*co // s.identity, per configuration/lock-file.md's dev_overrides identity // mechanism. func (s *Service) Describe(context.Context, *contextv1.DescribeRequest) (*contextv1.DescribeResponse, error) { - return &contextv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_CONTEXT)}, nil + return &contextv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_CONTEXT, ProtocolVersion)}, nil } diff --git a/pkg/frontend/doc.go b/pkg/frontend/doc.go index b2b585a..bb4527b 100644 --- a/pkg/frontend/doc.go +++ b/pkg/frontend/doc.go @@ -121,3 +121,11 @@ // unchanged as an [ActionTrigger] on activation, never rewritten // (render-tree.md#interactive-content-the-action-node). package frontend + +// ProtocolVersion is the version of the frontend category's own protocol this +// SDK implements — the "v1" in pluggableharness.frontend.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/frontend/server.go b/pkg/frontend/server.go index 568d171..7291366 100644 --- a/pkg/frontend/server.go +++ b/pkg/frontend/server.go @@ -70,6 +70,6 @@ func (svc *Service) Configure(ctx context.Context, req *frontendv1.ConfigureRequ // from svc.identity rather than a lock-file row — see NewService. func (svc *Service) Describe(context.Context, *frontendv1.DescribeRequest) (*frontendv1.DescribeResponse, error) { return &frontendv1.DescribeResponse{ - Producer: svc.identity.ProducerRef(commonv1.Category_CATEGORY_FRONTEND), + Producer: svc.identity.ProducerRef(commonv1.Category_CATEGORY_FRONTEND, ProtocolVersion), }, nil } diff --git a/pkg/memory/doc.go b/pkg/memory/doc.go index fcb90b1..d93dbd6 100644 --- a/pkg/memory/doc.go +++ b/pkg/memory/doc.go @@ -56,3 +56,11 @@ // provider-local heuristic. CountTokens in this package is the obvious, // hard-to-avoid call for that. package memory + +// ProtocolVersion is the version of the memory category's own protocol this +// SDK implements — the "v1" in pluggableharness.memory.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/memory/server.go b/pkg/memory/server.go index d7974e4..2d3625f 100644 --- a/pkg/memory/server.go +++ b/pkg/memory/server.go @@ -252,7 +252,7 @@ func (s *Service) GetRecord(ctx context.Context, req *memoryv1.GetRecordRequest) // s.provider (docs/specifications/memory/protocol.md#describe). func (s *Service) Describe(context.Context, *memoryv1.DescribeRequest) (*memoryv1.DescribeResponse, error) { return &memoryv1.DescribeResponse{ - Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_MEMORY), + Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_MEMORY, ProtocolVersion), }, nil } diff --git a/pkg/model/doc.go b/pkg/model/doc.go index 95305c5..8e1de9b 100644 --- a/pkg/model/doc.go +++ b/pkg/model/doc.go @@ -59,3 +59,11 @@ // RPC-boundary error in this package's server.go goes through // pkg/plugin.StatusError, never a bare gRPC status. package model + +// ProtocolVersion is the version of the model category's own protocol this +// SDK implements — the "v1" in pluggableharness.model.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/model/server.go b/pkg/model/server.go index b58d12c..69a2431 100644 --- a/pkg/model/server.go +++ b/pkg/model/server.go @@ -61,7 +61,7 @@ func (svc *Service) Register(s *grpc.Server) { // from svc.identity — no Provider method is involved. func (svc *Service) Describe(_ context.Context, _ *modelv1.DescribeRequest) (*modelv1.DescribeResponse, error) { return &modelv1.DescribeResponse{ - Producer: svc.identity.ProducerRef(commonv1.Category_CATEGORY_MODEL), + Producer: svc.identity.ProducerRef(commonv1.Category_CATEGORY_MODEL, ProtocolVersion), }, nil } diff --git a/pkg/plugin/identity.go b/pkg/plugin/identity.go index e9fa0c4..b4df3ad 100644 --- a/pkg/plugin/identity.go +++ b/pkg/plugin/identity.go @@ -1,7 +1,6 @@ package plugin import ( - "github.com/pluggableharness/agent/pkg/common" commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" ) @@ -28,18 +27,24 @@ type Identity struct { } // ProducerRef builds the common.v1.ProducerRef every category's Describe -// RPC returns, from this identity plus category (the plugin category this -// build is serving) and the protocol version this build was compiled -// against (pkg/common.ProtocolVersion). Category-specific SDKs (pkg/tool, -// pkg/model, ...) call this from their own Describe implementation — this +// RPC returns, from this identity plus the category this build serves and +// the version of THAT CATEGORY's protocol it implements. Category-specific +// SDKs (pkg/tool, pkg/model, ...) call this from their own Describe +// implementation, passing their own ProtocolVersion constant — this // package does not implement Describe itself, since DescribeResponse is a // distinct generated type per category. -func (id Identity) ProducerRef(category commonv1.Category) *commonv1.ProducerRef { +// +// protocolVersion is the category's protocol version, NOT +// pkg/common.ProtocolVersion. The two version different things and move +// independently: see that constant's own documentation for why coupling +// them would force every plugin of every category to rebuild whenever any +// one category made a breaking change. +func (id Identity) ProducerRef(category commonv1.Category, protocolVersion uint32) *commonv1.ProducerRef { return &commonv1.ProducerRef{ Name: id.Name, Version: id.Version, Source: id.Source, Category: category, - ProtocolVersion: uint32(common.ProtocolVersion), + ProtocolVersion: protocolVersion, } } diff --git a/pkg/plugin/identity_test.go b/pkg/plugin/identity_test.go index 3235b7b..32fb38d 100644 --- a/pkg/plugin/identity_test.go +++ b/pkg/plugin/identity_test.go @@ -3,7 +3,6 @@ package plugin_test import ( "testing" - "github.com/pluggableharness/agent/pkg/common" commonv1 "github.com/pluggableharness/agent/pkg/common/proto/v1" "github.com/pluggableharness/agent/pkg/plugin" ) @@ -12,9 +11,10 @@ func TestIdentity_ProducerRef(t *testing.T) { t.Parallel() tests := []struct { - name string - id plugin.Identity - category commonv1.Category + name string + id plugin.Identity + category commonv1.Category + protocolVersion uint32 }{ { name: "tool category", @@ -23,7 +23,8 @@ func TestIdentity_ProducerRef(t *testing.T) { Version: "1.2.3", Source: "github.com/agentco/filesystem-provider", }, - category: commonv1.Category_CATEGORY_TOOL, + category: commonv1.Category_CATEGORY_TOOL, + protocolVersion: 1, }, { name: "model category, empty source", @@ -32,6 +33,10 @@ func TestIdentity_ProducerRef(t *testing.T) { Version: "0.1.0", }, category: commonv1.Category_CATEGORY_MODEL, + // A second category on a different protocol version: the field + // reports THIS category's protocol, not one global number, so a + // model v2 must not imply anything about a tool plugin. + protocolVersion: 2, }, } @@ -39,7 +44,7 @@ func TestIdentity_ProducerRef(t *testing.T) { t.Run(tt.name, func(t *testing.T) { t.Parallel() - ref := tt.id.ProducerRef(tt.category) + ref := tt.id.ProducerRef(tt.category, tt.protocolVersion) if got, want := ref.GetName(), tt.id.Name; got != want { t.Errorf("GetName() = %q, want %q", got, want) @@ -53,7 +58,7 @@ func TestIdentity_ProducerRef(t *testing.T) { if got, want := ref.GetCategory(), tt.category; got != want { t.Errorf("GetCategory() = %v, want %v", got, want) } - if got, want := ref.GetProtocolVersion(), uint32(common.ProtocolVersion); got != want { + if got, want := ref.GetProtocolVersion(), tt.protocolVersion; got != want { t.Errorf("GetProtocolVersion() = %d, want %d", got, want) } }) diff --git a/pkg/slashcommand/doc.go b/pkg/slashcommand/doc.go index d0ca566..74f1b03 100644 --- a/pkg/slashcommand/doc.go +++ b/pkg/slashcommand/doc.go @@ -45,3 +45,11 @@ // needs to change shape, and update every consumer (including this // package) in lockstep. package slashcommand + +// ProtocolVersion is the version of the slashcommand category's own protocol this +// SDK implements — the "v1" in pluggableharness.slashcommand.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/slashcommand/server.go b/pkg/slashcommand/server.go index 742b78e..0a7ab7d 100644 --- a/pkg/slashcommand/server.go +++ b/pkg/slashcommand/server.go @@ -162,5 +162,5 @@ func (s *Service) Preview(ctx context.Context, req *slashcommandv1.PreviewReques // from s.identity, per // docs/specifications/slashcommand/protocol.md#describe. func (s *Service) Describe(context.Context, *slashcommandv1.DescribeRequest) (*slashcommandv1.DescribeResponse, error) { - return &slashcommandv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_SLASHCOMMAND)}, nil + return &slashcommandv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_SLASHCOMMAND, ProtocolVersion)}, nil } diff --git a/pkg/tool/doc.go b/pkg/tool/doc.go index dd99a79..eef14ea 100644 --- a/pkg/tool/doc.go +++ b/pkg/tool/doc.go @@ -30,3 +30,11 @@ // other package — extend this package instead if their shape ever needs // to change, and update every consumer in lockstep. package tool + +// ProtocolVersion is the version of the tool category's own protocol this +// SDK implements — the "v1" in pluggableharness.tool.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/tool/server.go b/pkg/tool/server.go index ba63f97..7e8d689 100644 --- a/pkg/tool/server.go +++ b/pkg/tool/server.go @@ -156,5 +156,5 @@ func (s *Service) Preview(ctx context.Context, req *toolv1.PreviewRequest) (*too // Describe implements toolv1.ToolServiceServer directly from s.identity, // per docs/specifications/tool/protocol.md#describe. func (s *Service) Describe(context.Context, *toolv1.DescribeRequest) (*toolv1.DescribeResponse, error) { - return &toolv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_TOOL)}, nil + return &toolv1.DescribeResponse{Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_TOOL, ProtocolVersion)}, nil } diff --git a/pkg/widget/doc.go b/pkg/widget/doc.go index ed4b45c..f0d3224 100644 --- a/pkg/widget/doc.go +++ b/pkg/widget/doc.go @@ -77,3 +77,11 @@ // docs/specifications/frontend/widget-protocol.md#error-taxonomy and // docs/specifications/frontend/conformance.md#error-taxonomy. package widget + +// ProtocolVersion is the version of the widget category's own protocol this +// SDK implements — the "v1" in pluggableharness.widget.v1. +// +// Deliberately NOT pkg/common.ProtocolVersion, which versions the +// go-plugin runtime contract shared by every category. The two move +// independently: see that constant's documentation for why. +const ProtocolVersion uint32 = 1 diff --git a/pkg/widget/server.go b/pkg/widget/server.go index 4dc6fac..d687659 100644 --- a/pkg/widget/server.go +++ b/pkg/widget/server.go @@ -57,7 +57,7 @@ func (s *Service) Register(gs *grpc.Server) { // (docs/specifications/frontend/widget-protocol.md#transport). func (s *Service) Describe(context.Context, *widgetv1.DescribeRequest) (*widgetv1.DescribeResponse, error) { return &widgetv1.DescribeResponse{ - Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_WIDGET), + Producer: s.identity.ProducerRef(commonv1.Category_CATEGORY_WIDGET, ProtocolVersion), }, nil } From af3413860d3d3eee4168754d33f1806da9371e94 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:09:44 -0400 Subject: [PATCH 12/16] model: require Configure to be safely re-callable MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Configure was specified as the place a bad configuration must fail, but said nothing about being called more than once. The kernel calls it exactly once at bring-up today, so a provider that only works the first time looks perfectly healthy in production — right up until a credential rotation or endpoint change needs a second call, which is when a failure is most expensive and least expected. State the requirement now rather than when that path is built. A plugin written against the weaker "called exactly once" reading has to be reworked, not merely re-invoked, so the cost of leaving it unstated grows with every provider written in the meantime. The rule is replace-wholesale, not merge: a second call carries the operator's complete intent, so a field absent from it is absent rather than inherited from the first. A provider holding a vendor client rebuilds it here for the same reason — a client built from one configuration alongside a setting from another is silently inconsistent, and surfaces as vendor errors that look like anything but a config problem. The conformance suite now calls Configure twice, which is the entire check. internal/anthropic and examples/provider both already satisfied it; a deliberately single-use provider is the regression guard proving the check bites. This is the contract half of the re-Configure work. The kernel-side trigger — noticing a changed agent.hcl and re-invoking — is a separate change and is called out as not yet existing rather than implied. --- docs/specifications/model/conformance.md | 1 + docs/specifications/model/protocol.md | 3 +++ pkg/model/modeltest/checks_test.go | 32 ++++++++++++++++++++++++ pkg/model/modeltest/suite.go | 13 ++++++++++ 4 files changed, 49 insertions(+) diff --git a/docs/specifications/model/conformance.md b/docs/specifications/model/conformance.md index f952cee..a70905d 100644 --- a/docs/specifications/model/conformance.md +++ b/docs/specifications/model/conformance.md @@ -29,6 +29,7 @@ On the wire, each category maps to a `grpc/codes.Code`: `context_length_exceeded | Credential attribute declared `required` only when every supported deployment needs one | MUST | [`protocol.md#gateway-and-locally-served-providers`](protocol.md#gateway-and-locally-served-providers) — a loopback-served runtime typically has no auth; validate the combination in `Configure` instead | | `Describe` RPC | MUST | [`protocol.md#describe`](protocol.md#describe) — identity for `dev_overrides` binaries with no lock-file entry | | Structured error taxonomy (above) | MUST | | +| `Configure` is safely re-callable | MUST | [`protocol.md#configure`](protocol.md#configure) — replaces configured state wholesale; a second call carries the operator's complete intent, so a field absent from it is absent, not inherited | | `tool_use` / `tool_result` | MUST, if any served model has `supports_tool_use = true` | | | `image` (vision) | MUST support where `supports_vision = true`; MUST reject cleanly where `false` | | | `document` | MUST support where `supports_documents = true`; MUST reject cleanly where `false` | [`data-types.md#canonical-message--content-block-schema`](data-types.md#canonical-message--content-block-schema) — mirrors `image`/`supports_vision`'s rule | diff --git a/docs/specifications/model/protocol.md b/docs/specifications/model/protocol.md index bdd7b7d..a41031c 100644 --- a/docs/specifications/model/protocol.md +++ b/docs/specifications/model/protocol.md @@ -45,6 +45,9 @@ SHOULD be implemented per model, using that vendor's real tokenizer: rather than Accepts a config object decoded from the provider's `agent.hcl` block via the schema-to-cty bridge (see [`configuration/blocks-reference.md`](../configuration/blocks-reference.md)). Field contents are provider-specific (API key, base URL override, org/project IDs, etc.) — this protocol doesn't mandate a shape beyond: - `Configure` MUST reject with a clear, structured error on missing required fields (e.g. no API key) rather than deferring the failure to the first `StreamCompletion` call. +- **`Configure` MUST be safely re-callable.** A plugin MUST accept it more than once over its lifetime and MUST replace its configured state wholesale rather than merging into it: a second call carries the operator's complete intent, so a field absent from it is absent, not inherited from the first. A provider holding a vendor client rebuilds that client here for the same reason — a client built from one configuration and a setting from another is a silently inconsistent provider, and the inconsistency surfaces as vendor errors that look like anything but a config problem. + + The kernel does not re-invoke `Configure` on a running plugin today; it is called once at bring-up. The requirement is stated now because it is the difference between a credential rotation or endpoint change costing a process restart and costing nothing, and because a plugin written against the weaker "called exactly once" reading would have to be reworked rather than merely re-invoked once that path exists. A provider is conformant only if a second `Configure` leaves it working. - A plugin MUST NOT echo any received secret value into an `Emit`'d event, a `Render` output, a log line, or an error message. Secrets flow into the process once, at `Configure` time, and stay there. - Resolving `env(...)`-style indirection in `agent.hcl` is the kernel's job (part of the HCL/`cty` bridge), not the plugin's — by the time `Configure` is called, the plugin receives resolved literal values regardless of how the operator wrote them in HCL. The `env(name)` argument MUST be a literal string, syntax-validated before evaluation (whether the named variable is actually set is a separate, evaluation-time check). diff --git a/pkg/model/modeltest/checks_test.go b/pkg/model/modeltest/checks_test.go index df62f25..bfe76e9 100644 --- a/pkg/model/modeltest/checks_test.go +++ b/pkg/model/modeltest/checks_test.go @@ -444,3 +444,35 @@ func TestCheck_unknownModelIDIsReported(t *testing.T) { t.Errorf("selecting an unadvertised model was not reported:\n%s", rep) } } + +// TestCheck_configureThatOnlyWorksOnceIsCaught guards the requirement +// whose failure is most expensive to discover late: the kernel calls +// Configure once at bring-up, so a provider that cannot take a second +// call looks perfectly healthy until a credential rotation needs one. +func TestCheck_configureThatOnlyWorksOnceIsCaught(t *testing.T) { + t.Parallel() + + p := &singleUseConfigureProvider{} + rep := modeltest.Check(t.Context(), p, modeltest.WithCallTimeout(2*time.Second)) + + if !strings.Contains(rep.String(), "re-callable") { + t.Errorf("the suite did not catch a Configure that only works once:\n%s", rep) + } +} + +// singleUseConfigureProvider accepts Configure exactly once. +type singleUseConfigureProvider struct { + conformingProvider + configured bool +} + +func (p *singleUseConfigureProvider) Configure(context.Context, *structpb.Struct) error { + if p.configured { + return &model.Error{ + Category: modelv1.ModelErrorCategory_MODEL_ERROR_CATEGORY_INVALID_REQUEST, + Message: "already configured", + } + } + p.configured = true + return nil +} diff --git a/pkg/model/modeltest/suite.go b/pkg/model/modeltest/suite.go index a11de84..4010586 100644 --- a/pkg/model/modeltest/suite.go +++ b/pkg/model/modeltest/suite.go @@ -298,6 +298,19 @@ func checkConfigure(ctx context.Context, rec *recorder, client modelv1.ModelServ "Configure returned %v; pass this provider's configuration with modeltest.WithConfig", err) return false } + + // Configure MUST be safely re-callable + // (docs/specifications/model/protocol.md#configure). The kernel calls + // it once today, so a provider that only works the first time looks + // fine in production right up until a credential rotation needs it — + // which is exactly when a failure is most expensive. Calling it twice + // here is the whole check: a provider that merges rather than replaces, + // or that panics on a rebuilt client, fails now instead of later. + if _, err := client.Configure(ctx, &modelv1.ConfigureRequest{Config: cfg.configure}); err != nil { + rec.failf("re-callable", + "a second Configure returned %v; it MUST be safely re-callable, replacing configured state wholesale", err) + return false + } return true } From ec7e5aba3c2bb538eb434fd11ee1d963a94d69d5 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:13:53 -0400 Subject: [PATCH 13/16] cmd: add providerconform, the conformance CLI MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A thin wrapper over pkg/model/modeltest.CheckBinary, for the two cases a Go author's own `go test` cannot reach: a plugin not written in Go, and an operator checking a binary they did not build. It launches the plugin the way the kernel does and speaks nothing but the wire protocol, so the implementation language never comes up. The assertions are modeltest's, unchanged. That is deliberate — a second copy of the rules would eventually disagree with the first, and the whole point of the suite is that there is one answer to "is this conformant". Exit 1 and exit 2 are kept distinct: a binary that will not start has not failed the suite, it has failed to be tested, and a CI job conflating the two reports a conformance regression when the real problem is a bad path or a missing execute bit. Three lint findings on the first version were worth fixing rather than suppressing: main deferred a signal-context release directly above an os.Exit that would skip it; the report was printed piecemeal with six unchecked writes; and a flag-parse failure returned a nil error, which would have made a usage mistake exit silently. The report is now built as one string and written once, so there is a single write whose error is actually checked, and a parse failure returns a sentinel main recognizes and does not double-print. --- cmd/providerconform/main.go | 174 +++++++++++++++++++++++++++++++ cmd/providerconform/main_test.go | 146 ++++++++++++++++++++++++++ 2 files changed, 320 insertions(+) create mode 100644 cmd/providerconform/main.go create mode 100644 cmd/providerconform/main_test.go diff --git a/cmd/providerconform/main.go b/cmd/providerconform/main.go new file mode 100644 index 0000000..1b8bd5e --- /dev/null +++ b/cmd/providerconform/main.go @@ -0,0 +1,174 @@ +// Command providerconform checks a built model-provider plugin against +// the conformance suite and reports what it found. +// +// providerconform [flags] +// +// It launches the binary the way the kernel does — a real handshake, a +// real subprocess, a real dispense — so it exercises the plugin's own +// main() wiring, and so it works on a plugin written in any language: +// it speaks nothing but the wire protocol. +// +// The assertions are pkg/model/modeltest's, identical to the ones a Go +// author gets from modeltest.Run in their own test. This binary exists +// for the two cases that cannot reach those: a plugin not written in Go, +// and an operator who wants to check a binary they did not build. +// +// Exit codes are meant to be scripted against: +// +// 0 no violations (skips may still be reported) +// 1 at least one violation +// 2 the binary could not be launched or checked at all +// +// 1 and 2 are deliberately distinct. A binary that will not start has not +// failed the suite, it has failed to be tested, and a CI job that +// conflates the two reports a conformance regression when the real +// problem is a bad path or a missing execute bit. +// +// Everything here is flag parsing and wiring, per +// .claude/rules/go-layout.md; the checking itself lives in +// pkg/model/modeltest so a Go author and this binary can never drift +// apart in what they assert. +package main + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "os/signal" + "strings" + "syscall" + + "google.golang.org/protobuf/types/known/structpb" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +// Exit codes, named so the doc comment above and the code cannot drift. +const ( + exitOK = 0 + exitViolated = 1 + exitUnusable = 2 +) + +// errUsage reports a command line the tool cannot act on. +var errUsage = errors.New("usage: providerconform [flags] ") + +// errFlagsReported marks a flag-parsing failure the flag package has +// already written to stderr. main returns it without printing, so the +// operator sees one explanation rather than two. +var errFlagsReported = errors.New("flag parsing failed") + +func main() { + // signal.NotifyContext so an interrupted run tears the plugin + // subprocess down rather than orphaning it. Released explicitly rather + // than deferred, because os.Exit below would skip a defer. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + + code, err := run(ctx, os.Args[1:], os.Stdout) + if err != nil && !errors.Is(err, errFlagsReported) { + fmt.Fprintln(os.Stderr, "providerconform:", err) + } + + stop() + os.Exit(code) +} + +// run parses args, runs the suite, and writes the report to out. +// +// Split from main so every path returns rather than calling os.Exit, +// which would skip deferred cleanup (go-style.md). +func run(ctx context.Context, args []string, out io.Writer) (int, error) { + fs := flag.NewFlagSet("providerconform", flag.ContinueOnError) + fs.SetOutput(os.Stderr) + + configPath := fs.String("config", "", + "path to a JSON file passed to the plugin's Configure RPC. Point the provider at a recorded transcript or a local test server here — a conformance run must not make billed vendor calls.") + modelID := fs.String("model", "", + "which advertised model to exercise. Defaults to the first the plugin advertises.") + timeout := fs.Duration("timeout", modeltest.DefaultCallTimeout, + "per-RPC timeout. Lower it against a fake, where any real delay means the plugin is wedged.") + + if err := fs.Parse(args); err != nil { + return exitUnusable, errFlagsReported + } + if fs.NArg() != 1 { + fs.Usage() + return exitUnusable, errUsage + } + binary := fs.Arg(0) + + opts := []modeltest.Option{modeltest.WithCallTimeout(*timeout)} + if *modelID != "" { + opts = append(opts, modeltest.WithModelID(*modelID)) + } + if *configPath != "" { + cfg, err := loadConfig(*configPath) + if err != nil { + return exitUnusable, err + } + opts = append(opts, modeltest.WithConfig(cfg)) + } + + report, err := modeltest.CheckBinary(ctx, binary, opts...) + if err != nil { + return exitUnusable, err + } + + if _, err := io.WriteString(out, formatReport(binary, report)); err != nil { + return exitUnusable, fmt.Errorf("writing the report: %w", err) + } + if !report.OK() { + return exitViolated, nil + } + return exitOK, nil +} + +// loadConfig reads a JSON object into the Struct Configure expects. +func loadConfig(path string) (*structpb.Struct, error) { + data, err := os.ReadFile(path) // #nosec G304 -- the operator names this file; reading it is the flag's purpose + if err != nil { + return nil, fmt.Errorf("reading -config: %w", err) + } + var raw map[string]any + if err := json.Unmarshal(data, &raw); err != nil { + return nil, fmt.Errorf("parsing -config as a JSON object: %w", err) + } + cfg, err := structpb.NewStruct(raw) + if err != nil { + return nil, fmt.Errorf("converting -config: %w", err) + } + return cfg, nil +} + +// formatReport renders the findings and a one-line summary. +// +// Built as a string and written once rather than printed piecemeal, so +// the single write is the only place that can fail and its error is +// actually checked. +// +// Skips appear as prominently as failures on purpose: a skip means a +// requirement was not reached, and the whole point of reporting them is +// that an unexercised check must never read as a pass. +func formatReport(binary string, report modeltest.Report) string { + var sb strings.Builder + fmt.Fprintf(&sb, "conformance: %s\n\n", binary) + if body := report.String(); body != "" { + sb.WriteString(body) + sb.WriteString("\n") + } + + failures, skips := len(report.Failures()), len(report.Skips()) + switch { + case failures == 0 && skips == 0: + sb.WriteString("PASS — every check satisfied\n") + case failures == 0: + fmt.Fprintf(&sb, "PASS — no violations, %d check(s) not reached\n", skips) + default: + fmt.Fprintf(&sb, "FAIL — %d violation(s), %d check(s) not reached\n", failures, skips) + } + return sb.String() +} diff --git a/cmd/providerconform/main_test.go b/cmd/providerconform/main_test.go new file mode 100644 index 0000000..1be4b84 --- /dev/null +++ b/cmd/providerconform/main_test.go @@ -0,0 +1,146 @@ +package main + +import ( + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +func TestRun_exitCodesDistinguishUnusableFromViolated(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + args []string + wantCode int + }{ + // A binary that will not start has not failed the suite, it has + // failed to be tested. A CI job conflating the two reports a + // conformance regression when the real problem is a bad path. + "missing binary": { + args: []string{filepath.Join(t.TempDir(), "does-not-exist")}, + wantCode: exitUnusable, + }, + "no binary named": { + args: nil, + wantCode: exitUnusable, + }, + "too many arguments": { + args: []string{"a", "b"}, + wantCode: exitUnusable, + }, + "unreadable config": { + args: []string{"-config", filepath.Join(t.TempDir(), "absent.json"), "some-binary"}, + wantCode: exitUnusable, + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + code, _ := run(context.Background(), tt.args, io.Discard) + if code != tt.wantCode { + t.Errorf("run(%v) = %d, want %d", tt.args, code, tt.wantCode) + } + }) + } +} + +func TestRun_missingBinaryNamesTheBinary(t *testing.T) { + t.Parallel() + + missing := filepath.Join(t.TempDir(), "not-a-plugin") + _, err := run(context.Background(), []string{missing}, io.Discard) + if err == nil { + t.Fatal("run() = nil error for a missing binary") + } + // The path has to appear, or an operator with several plugins cannot + // tell which one failed to launch. + if !strings.Contains(err.Error(), missing) { + t.Errorf("error %q does not name the binary %q", err, missing) + } +} + +func TestRun_usageErrorIsDistinguishable(t *testing.T) { + t.Parallel() + + _, err := run(context.Background(), nil, io.Discard) + if !errors.Is(err, errUsage) { + t.Errorf("err = %v, want errUsage", err) + } +} + +func TestLoadConfig(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + + valid := filepath.Join(dir, "valid.json") + if err := os.WriteFile(valid, []byte(`{"api_key":"sk-test","port":8080}`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + cfg, err := loadConfig(valid) + if err != nil { + t.Fatalf("loadConfig: %v", err) + } + if got := cfg.GetFields()["api_key"].GetStringValue(); got != "sk-test" { + t.Errorf("api_key = %q, want sk-test", got) + } + + malformed := filepath.Join(dir, "bad.json") + if err := os.WriteFile(malformed, []byte(`not json`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := loadConfig(malformed); err == nil { + t.Error("loadConfig accepted malformed JSON") + } + + // A JSON array is valid JSON but not a config object; rejecting it + // here beats a confusing failure inside Configure. + array := filepath.Join(dir, "array.json") + if err := os.WriteFile(array, []byte(`[1,2,3]`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := loadConfig(array); err == nil { + t.Error("loadConfig accepted a JSON array as a config object") + } +} + +func TestFormatReport_summarizesEachOutcome(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + findings []modeltest.Finding + want string + }{ + "clean": {want: "PASS — every check satisfied"}, + "skips only": { + findings: []modeltest.Finding{{Check: "a", Severity: modeltest.SeveritySkip, Message: "not reached"}}, + want: "PASS — no violations, 1 check(s) not reached", + }, + "violations": { + findings: []modeltest.Finding{{Check: "a", Severity: modeltest.SeverityFail, Message: "violated"}}, + want: "FAIL — 1 violation(s), 0 check(s) not reached", + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + got := formatReport("some-binary", modeltest.Report{Findings: tt.findings}) + if !strings.Contains(got, tt.want) { + t.Errorf("formatReport =\n%s\nwant a line containing %q", got, tt.want) + } + if !strings.Contains(got, "some-binary") { + t.Errorf("formatReport does not name the binary:\n%s", got) + } + }) + } +} From dadff9678441351cb9a103eb9fd35d0e6c586e9f Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:16:24 -0400 Subject: [PATCH 14/16] docs: update the first-party catalog for the new capability shapes MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The provider catalog still described the old single-mode ThinkingSpec and CachingSpec, and in three places instructed adapter authors to declare something the protocol can now express truthfully: - anthropic.md told authors to "pick one canonical mode" for Sonnet 4.6, whose second mechanism it called "not directly representable in a single ThinkingSpec value". Both controls are now declarable at once, with budget.deprecated marking the transitional one. - google.md carried an open escalation — that CachingSpec "has no way to declare 'this model supports two caching modes concurrently'" and that this was "worth flagging back to the protocol's designers". That gap is closed, so the page now shows the declaration rather than the workaround, and records why it mattered: breakpoints were gated on the mode, so a model forced to declare implicit caching had to discard breakpoints it could have honored. - xai.md mandated thinking.can_disable/mode/default fields that no longer exist. google.md keeps one open item rather than dropping it: PricingTier still carries a single cache-rate pair, so a model billing implicit and explicit hits at different rates can declare both mechanisms but cannot price them separately. That is now cross-referenced to conformance.md's open questions instead of being restated as a protocol-shape complaint. Also fixes three links into .claude/rules/ that I had added in earlier commits. Those files are outside the published docs tree, so they broke `mkdocs build --strict` — caught by running it, not by CI, since the Docs workflow is path-filtered and deliberately not a required check. They are now plain references rather than links. --- docs/first-party/providers/anthropic.md | 10 ++++++++-- docs/first-party/providers/google.md | 8 ++++++-- docs/first-party/providers/openai.md | 4 ++-- docs/first-party/providers/xai.md | 4 ++-- docs/specifications/configuration/blocks-reference.md | 2 +- docs/specifications/model/conformance.md | 2 +- docs/specifications/model/data-types.md | 2 +- 7 files changed, 21 insertions(+), 11 deletions(-) diff --git a/docs/first-party/providers/anthropic.md b/docs/first-party/providers/anthropic.md index 2b6b844..5cd40e9 100644 --- a/docs/first-party/providers/anthropic.md +++ b/docs/first-party/providers/anthropic.md @@ -59,9 +59,15 @@ Unlike the other three first-party vendors, there is no mini/flash/codex-style t ## 6. Implications for PluggableHarness Agent -**`ThinkingSpec`** ([`data-types.md#thinkingspec`](../../specifications/model/data-types.md#thinkingspec)): Anthropic is the vendor that most directly motivated this type's sum-type shape rather than a boolean flag. A conformant adapter cannot declare one `ThinkingSpec` for the whole plugin — it must vary per `ModelSpec`: `mode: always_on_adaptive` for Opus 4.8 and Sonnet 5 (with `can_disable` reflecting that `budget_tokens` now 400s rather than silently degrading), `mode: continuous_budget` for Haiku 4.5 (which never gained adaptive support), and — for Sonnet 4.6, which genuinely supports both mechanisms simultaneously during its transitional window — the adapter author will need to pick one canonical `mode` to declare (adaptive is recommended by Anthropic) while being aware `budget_tokens` still functions underneath as an escape hatch not directly representable in a single `ThinkingSpec` value. The `default` field matters concretely here: because Sonnet 5 runs adaptive thinking even when a request omits `thinking` entirely, a kernel wanting deterministic, budget-bounded behavior must know to always send an explicit override rather than relying on omission meaning "no thinking." +**`ThinkingSpec`** ([`data-types.md#thinkingspec`](../../specifications/model/data-types.md#thinkingspec)): Anthropic is the vendor that most directly motivated this type's per-axis shape, and the one whose lineup an earlier single-mode enum could not describe. A conformant adapter declares a `ThinkingSpec` per `ModelSpec`, not one per plugin, and each model declares every control it actually accepts: -**`CachingSpec`** ([`data-types.md#cachingspec`](../../specifications/model/data-types.md#cachingspec)): every current Anthropic model sets `mode: explicit_markers` — the adapter is responsible for placing `cache_control` breakpoints on content blocks per `data-types.md`'s explicit-markers semantics, not for detecting automatic caching. Given the 5-minute standard TTL, an adapter maintaining a long-running agentic session across tool-execution gaps is a strong candidate for implementing the optional `keepalive_supported` behavior described in `data-types.md`'s cache-keepalive note, since Anthropic's own API gives the adapter no server-side keepalive to rely on. +- Opus 4.8 and Sonnet 5 set `adaptive_by_default: true` **and** an `effort` control. Both are true at once — omitting thinking config still reasons, and `output_config.effort` selects a level on top of that — which is why the adapter sends `thinking: {type: "adaptive"}` alongside the effort level rather than instead of it. +- Haiku 4.5 declares a `budget` control and no `effort` control, with `adaptive_by_default: false`: omitting the thinking parameter there means no reasoning at all, not adaptive reasoning. +- Sonnet 4.6's transitional window — where `budget_tokens` still functions underneath the effort ladder — is declarable directly: both controls present, with `budget.deprecated: true`. No canonical mode has to be picked, and nothing is lost. + +`effort.default` matters concretely: because these models reason even when a request omits thinking entirely, a kernel wanting deterministic behavior reads the declared default and sends an explicit override rather than relying on omission meaning "no thinking." Opus 5's disable is `conditional` — Anthropic accepts an explicit disable at effort `high` or below and rejects it above — which tells the kernel a failed disable is vendor policy rather than an adapter bug. + +**`CachingSpec`** ([`data-types.md#cachingspec`](../../specifications/model/data-types.md#cachingspec)): every current Anthropic model sets `explicit_markers: true` and `implicit_automatic: false` — the adapter is responsible for placing `cache_control` breakpoints on content blocks per `data-types.md`'s explicit-markers semantics, not for detecting automatic caching. Given the 5-minute standard TTL, an adapter maintaining a long-running agentic session across tool-execution gaps is a strong candidate for implementing the optional `keepalive_supported` behavior described in `data-types.md`'s cache-keepalive note, since Anthropic's own API gives the adapter no server-side keepalive to rely on. **Tool schema and `ToolCall`/`ToolResult`** ([`data-types.md#tool-schema`](../../specifications/model/data-types.md#tool-schema)): Anthropic is one of the vendors (with Google and Ollama) whose tool-call arguments already arrive as a parsed object, so the Anthropic adapter's translation at the string/object boundary is the simpler direction — it serializes the kernel's parsed-JSON internal representation directly into `input` with no encode/decode step, unlike an OpenAI-shaped adapter. The adapter does need to handle the parallel-tool-call batching rule from §4 above (all `tool_result` blocks for one turn must land in a single `user` message) since this is stricter than what the generic protocol assumes about one-result-per-message. diff --git a/docs/first-party/providers/google.md b/docs/first-party/providers/google.md index 79723e3..d9da041 100644 --- a/docs/first-party/providers/google.md +++ b/docs/first-party/providers/google.md @@ -65,9 +65,13 @@ As a general pattern: confidence tracks how flagship a model is. `gemini-3-pro` ## 6. Implications for PluggableHarness Agent -**Per-model `ThinkingSpec.mode`, not a per-vendor constant.** Google's own lineup spans three different values of the [`ThinkingSpec`](../../specifications/model/data-types.md#thinkingspec) `mode` enum on its own: `discrete_effort` for the 3.x line (with `effort_levels` populated from the LOW/MEDIUM/HIGH or MINIMAL/LOW/MEDIUM/HIGH sets and `default` set to that model's actual default, e.g. `"HIGH"` for `gemini-3-pro`), `continuous_budget` for the 2.5 line (`budget_range` populated from `thinkingBudget`'s token-count bounds), and effectively `none` for `gemini-1.5-pro`. This is exactly the scenario `data-types.md` cites as the reason `ThinkingSpec` lives on each `ModelSpec` rather than being a single vendor-level flag — a Google adapter must build a distinct `ThinkingSpec` per model, and must not assume the `thinking_level` vs `thinkingBudget` parameter name generalizes across the whole roster. +**Per-model `ThinkingSpec`, not a per-vendor constant.** Google's own lineup occupies three different positions on [`ThinkingSpec`](../../specifications/model/data-types.md#thinkingspec)'s axes: the 3.x line declares an `effort` control (levels from the LOW/MEDIUM/HIGH or MINIMAL/LOW/MEDIUM/HIGH sets, with `effort.default` set to that model's actual default, e.g. `"HIGH"` for `gemini-3-pro`), the 2.5 line declares a `budget` control (range from `thinkingBudget`'s token-count bounds), and `gemini-1.5-pro` declares neither and leaves `supported` false. This is exactly the scenario `data-types.md` cites as the reason `ThinkingSpec` lives on each `ModelSpec` rather than being a single vendor-level flag — a Google adapter must build a distinct `ThinkingSpec` per model, and must not assume the `thinking_level` vs `thinkingBudget` parameter name generalizes across the whole roster. -**`CachingSpec.mode` cannot represent Google's actual behavior as a single value.** [`CachingSpec`](../../specifications/model/data-types.md#cachingspec)'s `mode` field is a single enum per model (`none` / `explicit_markers` / `implicit_automatic`) — a sum in the "one active mode" sense, not a set. But Google's 2.5+ models genuinely run both simultaneously: implicit automatic caching is on by default (75% discount, no caller action) *and* explicit manual declaration is available concurrently for a deeper 90% discount. Neither enum value alone captures this. A plugin author building this adapter has to make a real choice here rather than treating it as a formality: the pragmatic default is `mode: implicit_automatic` (it matches what happens when the caller does nothing, which is the common case), with the explicit/manual pathway and its better discount rate exposed separately — e.g. surfaced only through documentation or a provider-specific `Configure` option rather than through `CachingSpec` itself. This is worth flagging back to the protocol's designers as a real gap: the current `CachingSpec` shape has no way to declare "this model supports two caching modes concurrently, with different discount rates," which is precisely the situation Google's own docs describe. +**Google's two concurrent caching mechanisms are declarable directly.** Its 2.5+ models genuinely run both at once: implicit automatic caching is on by default (75% discount, no caller action) *and* explicit manual declaration is available concurrently for a deeper 90% discount. [`CachingSpec`](../../specifications/model/data-types.md#cachingspec)'s axes are independent, so such a model sets `implicit_automatic: true` **and** `explicit_markers: true`, and no canonical mode has to be chosen. + +This page previously flagged the inability to express that as a gap to raise with the protocol's designers; the gap is closed. It mattered for more than accuracy: `cache_breakpoints` are gated on the explicit-markers axis, so a model forced to declare only implicit caching was thereby required to discard breakpoints it could have honored — losing the deeper discount with no error anywhere. + +One related gap remains open and is recorded in [`conformance.md`](../../specifications/model/conformance.md#open-questions): `PricingTier` carries a single cache-rate pair, so a model billing implicit and explicit hits at different rates cannot price both. Declaring both mechanisms is now correct; pricing them separately still needs a `Pricing` revision. **Tool-call arguments arrive pre-parsed.** Per the [tool schema](../../specifications/model/data-types.md#tool-schema) section's cross-vendor note, Google — like Anthropic and Ollama, unlike OpenAI and Mistral — delivers `functionCall.args` as an already-parsed object rather than a JSON-encoded string. The adapter's `ToolCall` translation should pass this straight through into the kernel's internal parsed-JSON representation without a parse step, and the reverse `function_response` submission needs no re-encoding to a string either. diff --git a/docs/first-party/providers/openai.md b/docs/first-party/providers/openai.md index 5d00ceb..428debc 100644 --- a/docs/first-party/providers/openai.md +++ b/docs/first-party/providers/openai.md @@ -57,9 +57,9 @@ Authentication is an HTTP Bearer token: `Authorization: Bearer Date: Mon, 27 Jul 2026 13:21:16 -0400 Subject: [PATCH 15/16] kernel: give plugin authors a metrics path MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit pkg/telemetry documented the hole in its own source: Instruments() and Config() are excluded because both return internal/telemetry types an out-of-tree plugin cannot name, so "a plugin author has no path to either through this package today". Spans already had a route — NewSpanExporter relays them through the kernel — and metrics had none. pkg/kernel.Client.Metrics closes it, as the metrics counterpart to that exporter: observations travel to the kernel via RecordMetrics, which records them against an instrument named from the calling plugin's server-derived identity. Count and Histogram cover the common shapes; RecordBatch exists because each call is a round trip and a plugin reporting per-request metrics should not pay one per metric. Relaying rather than re-exporting internal/telemetry's instrument API is the point, not a limitation. The kernel owns the instruments and bounds their attribute cardinality — properties of the whole system's metric store, which no single plugin can decide for itself — and observability.md's relay model exists precisely so a plugin never exports off-process directly. A pkg/ wrapper over the internal instrument API would have been a second path to the same place, and the wrong one. An observation missing a name or a kind is rejected locally rather than sent for the kernel to reject, so the caller learns which observation was malformed instead of getting a batch-level failure. An empty batch is a no-op: the wire requires a non-empty one, so forwarding it would be a guaranteed rejection for a caller whose loop simply found nothing to report. pkg/telemetry's note is updated from "a known, tracked gap" to what is now true. --- pkg/kernel/doc.go | 6 ++ pkg/kernel/helpers_test.go | 26 +++++ pkg/kernel/metrics.go | 201 +++++++++++++++++++++++++++++++++++++ pkg/kernel/metrics_test.go | 186 ++++++++++++++++++++++++++++++++++ pkg/telemetry/telemetry.go | 16 ++- 5 files changed, 431 insertions(+), 4 deletions(-) create mode 100644 pkg/kernel/metrics.go create mode 100644 pkg/kernel/metrics_test.go diff --git a/pkg/kernel/doc.go b/pkg/kernel/doc.go index 715d2fa..115eadf 100644 --- a/pkg/kernel/doc.go +++ b/pkg/kernel/doc.go @@ -17,6 +17,12 @@ // that relays completed spans via ExportSpans — a plugin author wires // an ordinary OTel SDK TracerProvider and writes normal // tracer.Start(...) code; the relay transport is invisible. +// - Client.Metrics builds a recorder that relays observations via +// RecordMetrics — the metrics counterpart to NewSpanExporter, and the +// only route a plugin has to a metric at all. The kernel owns the +// instruments and bounds their attribute cardinality, neither of +// which a plugin can do for itself; that is why this relays rather +// than exporting off-process. // - LoadTelemetryConfig/TracingEnabled/MetricsEnabled/LogsEnabled/LogLevel/ // SamplingRatio cache GetTelemetryConfig's result once at startup // (specifications/observability.md#gettelemetryconfig-caching) — diff --git a/pkg/kernel/helpers_test.go b/pkg/kernel/helpers_test.go index 4e168b7..707262a 100644 --- a/pkg/kernel/helpers_test.go +++ b/pkg/kernel/helpers_test.go @@ -3,6 +3,7 @@ package kernel_test import ( "context" "net" + "sync" "testing" "google.golang.org/grpc" @@ -56,6 +57,31 @@ type fakeServer struct { emitFunc func(*kernelv1.EmitRequest) (*kernelv1.EmitResult, error) getSessionFunc func(*kernelv1.GetSessionRequest) (*kernelv1.GetSessionResult, error) readEventsFunc func(*kernelv1.ReadEventsRequest, kernelv1.KernelCallbackService_ReadEventsServer) error + recordMetricsFunc func(*kernelv1.RecordMetricsRequest) (*kernelv1.RecordMetricsResult, error) + + // mu guards recordMetricsSeen, which RecordMetrics appends to from + // whichever goroutine gRPC serves the call on. + mu sync.Mutex + recordMetricsSeen []*kernelv1.RecordMetricsRequest +} + +func (f *fakeServer) RecordMetrics(_ context.Context, req *kernelv1.RecordMetricsRequest) (*kernelv1.RecordMetricsResult, error) { + f.mu.Lock() + f.recordMetricsSeen = append(f.recordMetricsSeen, req) + f.mu.Unlock() + + if f.recordMetricsFunc != nil { + return f.recordMetricsFunc(req) + } + return &kernelv1.RecordMetricsResult{}, nil +} + +// recordMetricsRequests returns every RecordMetrics call this server saw, +// in order. +func (f *fakeServer) recordMetricsRequests() []*kernelv1.RecordMetricsRequest { + f.mu.Lock() + defer f.mu.Unlock() + return append([]*kernelv1.RecordMetricsRequest(nil), f.recordMetricsSeen...) } func (f *fakeServer) Log(ctx context.Context, req *kernelv1.LogRequest) (*kernelv1.LogResult, error) { diff --git a/pkg/kernel/metrics.go b/pkg/kernel/metrics.go new file mode 100644 index 0000000..fbd979b --- /dev/null +++ b/pkg/kernel/metrics.go @@ -0,0 +1,201 @@ +package kernel + +import ( + "context" + "errors" + "fmt" + "time" + + "google.golang.org/protobuf/types/known/timestamppb" + + kernelv1 "github.com/pluggableharness/agent/pkg/kernel/proto/v1" + metricv1 "github.com/pluggableharness/agent/pkg/metric/proto/v1" +) + +// Errors a malformed Observation is rejected with locally, rather than +// being sent for the kernel to reject — so the caller sees which +// observation was wrong instead of a batch-level failure. +var ( + // ErrMetricNameEmpty reports an observation with no metric name. + ErrMetricNameEmpty = errors.New("kernel: metric name is empty") + // ErrMetricKindUnset reports an observation that names no instrument + // shape. The kernel cannot create an instrument without one. + ErrMetricKindUnset = errors.New("kernel: metric kind is unset") +) + +// Metrics records metric observations through the kernel's RecordMetrics +// relay (kernel-callbacks.md#recordmetrics). +// +// It is the metrics counterpart to SpanExporter. A plugin exports no +// telemetry off-process itself: observations travel to the kernel, which +// records them against an instrument named +// "plugin.{category}.{name}.{metric name}" using the calling plugin's +// server-derived identity — a plugin cannot claim to be another producer. +// +// Construct one with Client.Metrics. The zero value is not usable. +// +// # Why the kernel bounds attributes and this type does not +// +// A metric's attribute key set is bounded per instrument by the kernel +// (observability.md's tracing/metrics asymmetry), and a key beyond that +// bound is dropped with a throttled warning rather than rejected. That is +// the kernel's job precisely because cardinality is a property of the +// whole system's metric store, not of any one plugin — so this type +// forwards what it is given rather than second-guessing it. Keep +// attribute values low-cardinality anyway: an unbounded value (a session +// id, a request id) belongs on a span, never on a metric. +type Metrics struct { + client *Client + sessionID string +} + +// Metrics returns a recorder that relays observations through c. +func (c *Client) Metrics(opts ...MetricsOption) *Metrics { + m := &Metrics{client: c} + for _, opt := range opts { + opt(m) + } + return m +} + +// MetricsOption configures a Metrics recorder. +type MetricsOption func(*Metrics) + +// WithMetricsSessionID attaches a session id to every batch this recorder +// sends, so observations are attributable to the session that caused +// them. Optional, matching RecordMetricsRequest's own rule. +func WithMetricsSessionID(sessionID string) MetricsOption { + return func(m *Metrics) { m.sessionID = sessionID } +} + +// Observation is one metric measurement. +// +// Exactly one of Int or Float carries the value; use the Count/Record +// helpers below rather than building this by hand unless you need an +// attribute set or a non-default unit. +type Observation struct { + // Name is the metric's name, without the plugin prefix the kernel + // adds. MUST be non-empty. + Name string + // Description is what this metric measures. MAY be empty. + Description string + // Unit is UCUM-style ("ms", "By", "1"). MAY be empty. + Unit string + // Kind is which instrument shape this belongs to. MUST be set. + // + // The kernel rejects an observation whose kind disagrees with a + // previously-created instrument of the same name, so a metric's kind + // is effectively fixed by its first use — pick it deliberately rather + // than letting two call sites disagree. + Kind metricv1.MetricKind + // Int is an integer observation. Ignored when Float is set. + Int int64 + // Float is a floating-point observation. Takes precedence over Int. + Float *float64 + // Attributes are open-ended labels for this observation. Keep them + // low-cardinality: the kernel bounds the key set per instrument and + // drops what exceeds it. + Attributes map[string]string + // Time is when the observation occurred. Zero means now. + Time time.Time +} + +// Count records a monotonically increasing sum, e.g. a request count. +func (m *Metrics) Count(ctx context.Context, name string, value int64, attributes map[string]string) error { + return m.Record(ctx, Observation{ + Name: name, + Kind: metricv1.MetricKind_METRIC_KIND_COUNTER, + Int: value, + Attributes: attributes, + }) +} + +// Histogram records one observation to be aggregated into a distribution, +// e.g. a call duration. +// +// One observation, never a pre-aggregated bucket set: the kernel's own +// histogram instrument performs the aggregation, the same way an OTel +// Histogram's Record does on the reporting side. +func (m *Metrics) Histogram(ctx context.Context, name string, value float64, unit string, attributes map[string]string) error { + return m.Record(ctx, Observation{ + Name: name, + Kind: metricv1.MetricKind_METRIC_KIND_HISTOGRAM, + Unit: unit, + Float: &value, + Attributes: attributes, + }) +} + +// Record relays one observation. +// +// Prefer Count or Histogram for the common shapes; reach for this when an +// observation needs a description, an up-down counter, or an explicit +// timestamp. +func (m *Metrics) Record(ctx context.Context, obs Observation) error { + return m.RecordBatch(ctx, obs) +} + +// RecordBatch relays several observations in one call. +// +// Batching is the reason this exists alongside Record: each call is a +// round trip to the kernel, and a plugin recording per-request metrics +// should not pay one per metric. An empty batch is a no-op rather than an +// error — RecordMetricsRequest requires a non-empty batch, so sending one +// would be a guaranteed rejection for a caller whose loop simply found +// nothing to report. +func (m *Metrics) RecordBatch(ctx context.Context, observations ...Observation) error { + if len(observations) == 0 { + return nil + } + + records := make([]*metricv1.MetricRecord, 0, len(observations)) + for i, obs := range observations { + record, err := obs.toProto() + if err != nil { + return fmt.Errorf("kernel: record metrics: observation %d: %w", i, err) + } + records = append(records, record) + } + + req := &kernelv1.RecordMetricsRequest{Metrics: records} + if m.sessionID != "" { + id := m.sessionID + req.SessionId = &id + } + if _, err := m.client.raw.RecordMetrics(ctx, req); err != nil { + return fmt.Errorf("kernel: record metrics: %w", err) + } + return nil +} + +// toProto converts one observation into its wire form, rejecting a +// declaration the kernel would reject anyway — locally, where the caller +// can see which observation was wrong. +func (o Observation) toProto() (*metricv1.MetricRecord, error) { + if o.Name == "" { + return nil, ErrMetricNameEmpty + } + if o.Kind == metricv1.MetricKind_METRIC_KIND_UNSPECIFIED { + return nil, fmt.Errorf("%w: %q", ErrMetricKindUnset, o.Name) + } + + at := o.Time + if at.IsZero() { + at = time.Now() + } + + record := &metricv1.MetricRecord{ + Name: o.Name, + Description: o.Description, + Unit: o.Unit, + Kind: o.Kind, + Attributes: o.Attributes, + Time: timestamppb.New(at), + } + if o.Float != nil { + record.Value = &metricv1.MetricRecord_DoubleValue{DoubleValue: *o.Float} + } else { + record.Value = &metricv1.MetricRecord_IntValue{IntValue: o.Int} + } + return record, nil +} diff --git a/pkg/kernel/metrics_test.go b/pkg/kernel/metrics_test.go new file mode 100644 index 0000000..c8ba2bc --- /dev/null +++ b/pkg/kernel/metrics_test.go @@ -0,0 +1,186 @@ +package kernel_test + +import ( + "errors" + "testing" + "time" + + metricv1 "github.com/pluggableharness/agent/pkg/metric/proto/v1" + + "github.com/pluggableharness/agent/pkg/kernel" +) + +func TestMetrics_countAndHistogram(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + if err := client.Metrics().Count(t.Context(), "requests", 3, map[string]string{"status": "ok"}); err != nil { + t.Fatalf("Count: %v", err) + } + if err := client.Metrics().Histogram(t.Context(), "latency", 12.5, "ms", nil); err != nil { + t.Fatalf("Histogram: %v", err) + } + + got := srv.recordMetricsRequests() + if len(got) != 2 { + t.Fatalf("got %d RecordMetrics calls, want 2", len(got)) + } + + counter := got[0].GetMetrics()[0] + if counter.GetName() != "requests" { + t.Errorf("name = %q, want requests", counter.GetName()) + } + if counter.GetKind() != metricv1.MetricKind_METRIC_KIND_COUNTER { + t.Errorf("kind = %v, want COUNTER", counter.GetKind()) + } + if counter.GetIntValue() != 3 { + t.Errorf("int value = %d, want 3", counter.GetIntValue()) + } + if counter.GetAttributes()["status"] != "ok" { + t.Errorf("attributes = %v, want status=ok", counter.GetAttributes()) + } + if counter.GetTime() == nil { + t.Error("time is unset; it is a MUST on the wire") + } + + hist := got[1].GetMetrics()[0] + if hist.GetKind() != metricv1.MetricKind_METRIC_KIND_HISTOGRAM { + t.Errorf("kind = %v, want HISTOGRAM", hist.GetKind()) + } + if hist.GetDoubleValue() != 12.5 { + t.Errorf("double value = %v, want 12.5", hist.GetDoubleValue()) + } + if hist.GetUnit() != "ms" { + t.Errorf("unit = %q, want ms", hist.GetUnit()) + } +} + +func TestMetrics_recordBatchSendsOneCall(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + // The reason RecordBatch exists: each call is a round trip, and a + // plugin reporting per-request metrics should not pay one per metric. + err := client.Metrics().RecordBatch(t.Context(), + kernel.Observation{Name: "a", Kind: metricv1.MetricKind_METRIC_KIND_COUNTER, Int: 1}, + kernel.Observation{Name: "b", Kind: metricv1.MetricKind_METRIC_KIND_COUNTER, Int: 2}, + ) + if err != nil { + t.Fatalf("RecordBatch: %v", err) + } + + got := srv.recordMetricsRequests() + if len(got) != 1 { + t.Fatalf("got %d calls, want 1 — the batch was not sent together", len(got)) + } + if n := len(got[0].GetMetrics()); n != 2 { + t.Errorf("got %d records in the batch, want 2", n) + } +} + +func TestMetrics_emptyBatchIsANoOp(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + // RecordMetricsRequest requires a non-empty batch, so sending one + // would be a guaranteed rejection for a caller whose loop simply found + // nothing to report. + if err := client.Metrics().RecordBatch(t.Context()); err != nil { + t.Fatalf("an empty batch returned %v, want nil", err) + } + if n := len(srv.recordMetricsRequests()); n != 0 { + t.Errorf("an empty batch produced %d calls, want 0", n) + } +} + +func TestMetrics_rejectsMalformedObservationsLocally(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + tests := map[string]struct { + obs kernel.Observation + wantErr error + }{ + "no name": { + obs: kernel.Observation{Kind: metricv1.MetricKind_METRIC_KIND_COUNTER, Int: 1}, + wantErr: kernel.ErrMetricNameEmpty, + }, + "no kind": { + obs: kernel.Observation{Name: "a", Int: 1}, + wantErr: kernel.ErrMetricKindUnset, + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + err := client.Metrics().Record(t.Context(), tt.obs) + if !errors.Is(err, tt.wantErr) { + t.Errorf("err = %v, want wrapping %v", err, tt.wantErr) + } + }) + } + + // Rejected locally, so nothing reached the kernel: the caller learns + // which observation was wrong instead of getting a batch-level + // failure back. + if n := len(srv.recordMetricsRequests()); n != 0 { + t.Errorf("a malformed observation produced %d calls, want 0", n) + } +} + +func TestMetrics_sessionIDIsAttachedWhenSet(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + m := client.Metrics(kernel.WithMetricsSessionID("session-01ARZ3")) + if err := m.Count(t.Context(), "requests", 1, nil); err != nil { + t.Fatalf("Count: %v", err) + } + got := srv.recordMetricsRequests() + if len(got) != 1 || got[0].GetSessionId() != "session-01ARZ3" { + t.Errorf("session_id = %q, want session-01ARZ3", got[0].GetSessionId()) + } + + // Absent by default: it is optional on the wire, and a recorder with + // no session has none to claim. + if err := client.Metrics().Count(t.Context(), "requests", 1, nil); err != nil { + t.Fatalf("Count: %v", err) + } + got = srv.recordMetricsRequests() + if got[1].SessionId != nil { + t.Errorf("session_id = %q, want absent", got[1].GetSessionId()) + } +} + +func TestMetrics_explicitTimeIsPreserved(t *testing.T) { + t.Parallel() + + srv := &fakeServer{} + client := newTestClient(t, srv) + + at := time.Date(2026, 7, 27, 12, 0, 0, 0, time.UTC) + err := client.Metrics().Record(t.Context(), kernel.Observation{ + Name: "a", + Kind: metricv1.MetricKind_METRIC_KIND_UP_DOWN_COUNTER, + Int: 1, + Time: at, + }) + if err != nil { + t.Fatalf("Record: %v", err) + } + + got := srv.recordMetricsRequests()[0].GetMetrics()[0] + if !got.GetTime().AsTime().Equal(at) { + t.Errorf("time = %v, want %v", got.GetTime().AsTime(), at) + } +} diff --git a/pkg/telemetry/telemetry.go b/pkg/telemetry/telemetry.go index 4631f59..e57791d 100644 --- a/pkg/telemetry/telemetry.go +++ b/pkg/telemetry/telemetry.go @@ -33,10 +33,18 @@ const otlpHTTPProtocol = "http/protobuf" // internal/telemetry was needed for Bootstrap's new return type to compile. // // Instruments() and Config() are deliberately not part of this interface: -// both return internal/telemetry-only types (*Instruments, Config), and -// re-exporting either is a metrics-API redesign larger than this fix's -// scope — a plugin author has no path to either through this package -// today. That's a known, tracked gap, not an oversight. +// both return internal/telemetry-only types (*Instruments, Config), which +// an out-of-tree plugin cannot name at all. +// +// That is no longer a gap, because it is no longer the only route. +// pkg/kernel.Client.Metrics records observations through the kernel's +// RecordMetrics relay, which is how a plugin's metrics were always meant +// to reach an exporter (observability.md's relay model) — the kernel +// bounds their attribute cardinality and owns the instruments, neither of +// which a plugin can do for itself. Re-exporting internal/telemetry's own +// instrument API here would be a second, competing path to the same +// place, and the wrong one: it would export off-process directly, which +// the relay model exists to prevent. type Provider interface { // Shutdown flushes and closes the underlying tracer/meter/logger // providers. Idempotent — safe to call more than once, returning the From 926725e2ba931fcada7c889e3de07dd73811a7c4 Mon Sep 17 00:00:00 2001 From: Steven Crothers Date: Mon, 27 Jul 2026 13:23:14 -0400 Subject: [PATCH 16/16] gitignore: catch bare go build output in the working directory MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A `go build ./...` names its output after the package and writes it to the working directory. The existing patterns cover *.exe, *.so, and *.test, none of which match a bare ELF binary on Linux or macOS — so two of them (a 22 MB providerconform at the repo root, and the example provider inside its own module) were committed before being caught. Listed by name rather than by a broad pattern, because anything wide enough to catch an extensionless binary would also ignore real source files. --- .gitignore | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.gitignore b/.gitignore index 90c193a..ee2197a 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,16 @@ *.test __debug_bin* +# A bare `go build ./...` names its output after the package and drops it in +# the working directory, which the extension patterns above do not catch on +# Linux or macOS. These are the ones this repo can actually produce, listed +# by name because a broad pattern here would ignore real source files. +/agent +/anthropic +/tui +/providerconform +examples/*/agent-example-provider + # --- Go: test, coverage & profiling artifacts -------------------------------- *.out coverage.txt