diff --git a/.claude/rules/go-layout.md b/.claude/rules/go-layout.md index c8a208d..b887bd0 100644 --- a/.claude/rules/go-layout.md +++ b/.claude/rules/go-layout.md @@ -39,6 +39,12 @@ pkg/ first-class, third-party-consumable Go integration — the tool/model JSON-Schema subset, ContentBlock) that every category SDK composes rather than reimplementing per category. + sse/ shared vendor-neutral plumbing: SSE frame + decoding, which every model provider needs and + none of them should rewrite. Not a builder and + not tied to one category — the test for belonging + here is that a third-party plugin author would + otherwise copy it out of another provider. docs/specifications/ protocol contracts (already exists, authoritative) ``` diff --git a/.claude/rules/plugin-runtime.md b/.claude/rules/plugin-runtime.md index 80c6469..a8b56d6 100644 --- a/.claude/rules/plugin-runtime.md +++ b/.claude/rules/plugin-runtime.md @@ -15,10 +15,32 @@ covers RPC shape and `proto.md` covers wire typing. magic cookie key/value and a `ProtocolVersion` field. Do not give categories different cookies — the uniform handshake is what lets the kernel reject a mismatched-protocol plugin before ever calling into it. -- `ProtocolVersion` is bumped only on a breaking wire change (see `proto.md`'s - `buf breaking` rule) — bumping it and shipping a `v1`→`v2` proto package - bump happen together, never independently. -- The kernel-side plugin client always checks the negotiated protocol +- **Two different versions exist, and they move independently. Do not couple them.** + - `pkg/common.ProtocolVersion` is the go-plugin **handshake** version. It + versions the *runtime* contract only — the handshake, the fixed callback + broker id, and how services are muxed onto one connection. Bump it only + when one of those changes. + - Each category SDK's own `ProtocolVersion` constant (e.g. + `pkg/model.ProtocolVersion`) is that **category's** protocol version — + the `v1` in `pluggableharness.model.v1`. It is bumped alongside a + `v1`→`v2` proto package bump for that category, and for no other reason. + + The separation is load-bearing. A handshake-version mismatch rejects a + plugin *before any category RPC is issued*, so folding category versions + into the handshake would mean a breaking change in one category forced + every plugin of every other category ever published to rebuild in order + to keep working. An earlier revision of `common.v1.ProducerRef`'s comment + said a handshake bump "always accompanies a proto package version bump + for that category"; that was the coupling, and it is no longer the rule. +- **A category's version is already part of its gRPC service name**, so + correctness does not depend on any negotiation field: a plugin serving + `pluggableharness.model.v2.ModelService` and a kernel dispensing `v1` + simply do not match. `ProducerRef.protocol_version` exists so that + mismatch surfaces as a clear version error at bring-up rather than as an + opaque "unimplemented service" on the first real call — and so a lock + file recording it lets `preflightVersionCheck` reject a plugin before + spawning it at all. +- The kernel-side plugin client always checks the negotiated handshake version before issuing the first category RPC; a mismatch is a startup error, not a runtime error discovered on first call. diff --git a/.claude/rules/proto.md b/.claude/rules/proto.md index 1279f24..dd60e77 100644 --- a/.claude/rules/proto.md +++ b/.claude/rules/proto.md @@ -53,10 +53,29 @@ strongly typed as the Go code that implements it. enumerate: `log.v1.LogEntry.fields` (mirrors `slog.Attr`'s open key/value model), `config.v1`'s `ConfigureRequest.config` and `kernel.v1.GetConfigResult.config` (already-decoded `agent.hcl` values, whose shape is the *provider's* schema, - not the kernel's to name), and `trace.v1.Span`/`SpanEvent`'s `attributes` + not the kernel's to name), `trace.v1.Span`/`SpanEvent`'s `attributes` (an OTel span's attribute set, open-ended per call site by the same - reasoning as `log.v1.LogEntry.fields`) are the precedents. A field whose - keys are actually fixed and enumerable belongs in a real message instead. + reasoning as `log.v1.LogEntry.fields`), and + `model.v1.StreamCompletionRequest.provider_options` (vendor-specific + request knobs the kernel has no semantics for — the same "shape is the + provider's, not the kernel's to name" reasoning as `ConfigureRequest.config`, + applied per-request rather than once at configure time) are the + precedents. A field whose keys are actually fixed and enumerable belongs + in a real message instead. +- **A `Struct` field is pass-through only: if the kernel reads it, it must + be a typed field instead.** This is what keeps the precedent list above + from becoming a general-purpose escape from the strong-typing rule. Every + entry on it is data the kernel *carries* — logs it, relays it, hands it + to the plugin that owns it — never data the kernel *branches on*. The + moment a value affects kernel behavior (routing, capability validation, + cost computation, replay), a `Struct` is the wrong home for it, because + nothing about a `Struct` key is discoverable, validatable, or versionable + at the wire contract level. `model.v1`'s prompt-cache TTL is the worked + example: it looks like an ordinary vendor knob, but the kernel computes + `cost_usd` and the TTL changes the rate, so it cannot live in + `provider_options` and must be a typed field or not exist + (`docs/specifications/model/conformance.md`'s open questions records why + it is currently the latter). - Every field that has a natural bounded domain (status, kind, risk class, error category) is an `enum`, not a `string`. `docs/specifications/tool/conformance.md`'s `ToolErrorCategory`, and `docs/specifications/tool/data-types.md`'s `RiskClass` diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index f28edd8..3adc5f3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -80,6 +80,40 @@ jobs: - name: Vet run: go vet ./... + # pkg/ is the third-party plugin-author surface, consumed from + # outside this module where internal/ is unreachable by definition. + # An internal/ import there compiles fine here and fails for every + # downstream author, so it has to be caught on this side. + # + # pkg/telemetry is the one sanctioned exception, documented in its + # own source: it wraps internal/telemetry but keeps its exported + # surface expressible outside this module. + - name: Verify pkg/ does not import internal/ + run: | + if grep -rn --include='*.go' 'pluggableharness/agent/internal' pkg/ \ + | grep -v '^pkg/telemetry/'; then + echo "::error::pkg/ must not import internal/ — see the exception note in pkg/telemetry" + exit 1 + fi + + # Builds the example provider from its own module, against this + # commit via a replace directive. This is the only check that proves + # pkg/ is genuinely usable from outside the main module: the + # depguard rule on internal/anthropic only simulates that isolation, + # and a simulation cannot catch an unexported type leaking through + # an exported signature. + - name: Build the standalone example provider + working-directory: examples/provider + run: | + go mod tidy + git diff --exit-code -- go.mod go.sum + go build ./... + go vet ./... + # Also runs the example's own conformance test, which proves + # pkg/model/modeltest is reachable and usable by a third party — + # the premise of shipping a conformance suite in pkg/ at all. + go test ./... + # --------------------------------------------------------------------------- # test — race-enabled tests on every platform we release for. # diff --git a/.gitignore b/.gitignore index 90c193a..ee2197a 100644 --- a/.gitignore +++ b/.gitignore @@ -27,6 +27,16 @@ *.test __debug_bin* +# A bare `go build ./...` names its output after the package and drops it in +# the working directory, which the extension patterns above do not catch on +# Linux or macOS. These are the ones this repo can actually produce, listed +# by name because a broad pattern here would ignore real source files. +/agent +/anthropic +/tui +/providerconform +examples/*/agent-example-provider + # --- Go: test, coverage & profiling artifacts -------------------------------- *.out coverage.txt diff --git a/api/pluggableharness/common/v1/types.proto b/api/pluggableharness/common/v1/types.proto index 49624c4..d8a9369 100644 --- a/api/pluggableharness/common/v1/types.proto +++ b/api/pluggableharness/common/v1/types.proto @@ -120,10 +120,25 @@ message ProducerRef { // Which of the seven plugin categories this producer implements. Category category = 4; - // The go-plugin handshake protocol version this producer build was - // compiled against (.claude/rules/plugin-runtime.md). A version bump - // here always accompanies a proto package version bump (v1 -> v2) for - // that category — it never happens independently of one. + // The version of THIS PRODUCER'S OWN CATEGORY protocol it implements — + // the "v1" in pluggableharness..v1, reported per category + // rather than globally. + // + // Deliberately NOT the go-plugin handshake version, which versions the + // runtime contract (handshake, callback broker, service muxing) that + // every category shares. The two move independently: coupling them + // would mean a breaking change in one category forced every plugin of + // every OTHER category to rebuild, because a handshake-version bump + // rejects a plugin before any category RPC is issued. + // + // Correctness does not depend on this field — a category's proto + // package version is part of its gRPC service name, so a plugin serving + // pluggableharness.model.v2.ModelService and a kernel dispensing v1 + // simply do not match. It exists so that mismatch is reported as a + // clear version error at bring-up rather than as an opaque + // "unimplemented service" on the first real call, and so a lock file + // recording it lets the kernel reject a plugin before spawning it at + // all (.claude/rules/plugin-runtime.md). uint32 protocol_version = 5; } diff --git a/api/pluggableharness/model/v1/events.proto b/api/pluggableharness/model/v1/events.proto index b0fb77f..c4ec378 100644 --- a/api/pluggableharness/model/v1/events.proto +++ b/api/pluggableharness/model/v1/events.proto @@ -34,6 +34,24 @@ message StreamEvent { // A complete, vendor-encrypted reasoning block the kernel cannot // interpret. RedactedThinking redacted_thinking = 10; + // The vendor has accepted the request and named it. + StreamStart stream_start = 11; + } + + // StreamStart carries the vendor's own identifier for this request, as + // soon as the adapter learns it — normally from response headers, + // before any content streams. + // + // It exists so a failure can be correlated with the vendor's own logs + // when asking them what went wrong. That is why it is a separate event + // emitted early rather than a field on Stop: an id that only arrives on + // successful completion is absent in exactly the case it is needed. + // MAY be omitted entirely by a vendor that publishes no such id. + message StreamStart { + // The vendor's request identifier, verbatim (an Anthropic + // `request-id` header, an OpenAI `x-request-id`). Opaque to the + // kernel: logged and surfaced, never parsed. + string provider_request_id = 1; } // TextDelta carries one incremental fragment of assistant text output. diff --git a/api/pluggableharness/model/v1/rpc_request.proto b/api/pluggableharness/model/v1/rpc_request.proto index ee76144..19b15f8 100644 --- a/api/pluggableharness/model/v1/rpc_request.proto +++ b/api/pluggableharness/model/v1/rpc_request.proto @@ -90,18 +90,70 @@ message StreamCompletionRequest { // natural stable-prefix boundaries — see // model/protocol.md#cache-breakpoint-placement-policy. repeated CacheBreakpoint cache_breakpoints = 7; + + // Vendor-specific request knobs the kernel has no semantics for, passed + // through untouched: the kernel never reads a key, validates one, or + // assigns meaning to one. This is the escape hatch that lets a + // third-party provider ship a vendor feature — a service tier, a + // sampling seed, a beta-feature flag, a conversation-retention id — + // without a change to this protocol. Values originate in the provider's + // own ConfigSchema and the operator's provider{} block, and the provider + // documents its own accepted keys; two providers MAY use the same key + // name for unrelated things. + // + // A Struct for the same reason ConfigureRequest.config is one — the + // shape is the provider's schema, not the kernel's to name — applied + // per-request rather than once at configure time (.claude/rules/proto.md's + // Struct precedent list). + // + // MUST NOT carry anything the kernel reads. Pass-through is the whole + // contract, so a value affecting routing, capability validation, cost + // computation, or replay is a typed field on this protocol or it does + // not work at all — a provider smuggling one through here gets silence, + // not kernel behavior. Promoting such a knob to a typed field in a later + // revision is the fix; teaching the kernel to read this field is not. + // See model/data-types.md#provider_options. + optional google.protobuf.Struct provider_options = 8; } -// CountTokensRequest is CountTokens' request: the raw text to count, per -// model.md §2.1. +// CountTokensRequest is CountTokens' request: the request whose input +// tokens are being counted, per model/protocol.md#counttokens. +// +// This mirrors StreamCompletionRequest's content-bearing fields, minus +// everything that only affects generation, because every vendor exposing +// exact counting counts a whole request rather than a string — Anthropic's +// /v1/messages/count_tokens takes messages plus system plus tools. An +// earlier revision carried a flat `text` field; concatenating text and +// discarding the rest undercounts by the entire tool-schema and +// system-preamble weight, which is exactly the weight that decides whether +// a turn fits in the context window. message CountTokensRequest { - // The text to count tokens for. - string text = 1; + reserved 1; + + reserved "text"; // Selects which of this provider's ModelSpec.id to count against — a // provider serving several models MAY have distinct tokenizers per // model. MUST be set. string model_id = 2; + + // The conversation to count, in emission order. MAY be empty. A caller + // with only loose content to measure (a context provider sizing its own + // contribution, kernel-callbacks.md#counttokens) passes it as a single + // user message — which is what the adapter would have had to construct + // anyway. + repeated pluggableharness.content.v1.Message messages = 3; + + // The kernel-assembled context chain that would accompany these + // messages, counted against the same vendor mechanism + // StreamCompletionRequest.assembled_context maps to. MAY be empty. + repeated pluggableharness.content.v1.ContextSection assembled_context = 4; + + // The tool declarations that would accompany these messages. MAY be + // empty. Tool schemas are frequently the largest single contributor to + // a request's input tokens, so omitting them is the main way a count + // goes badly wrong. + repeated ToolDeclaration tools = 5; } // RenderRequest carries the opaque payload to render, per model.md §7. diff --git a/api/pluggableharness/model/v1/types.proto b/api/pluggableharness/model/v1/types.proto index 4a8c695..3d1e000 100644 --- a/api/pluggableharness/model/v1/types.proto +++ b/api/pluggableharness/model/v1/types.proto @@ -110,30 +110,33 @@ message ModelSpec { bool supports_documents = 12; } -// ThinkingMode enumerates the shapes of extended-reasoning control found -// across researched vendors (model.md §2) — a plain supports_thinking -// bool would lose information the kernel needs to build a correct -// request, since some vendors (e.g. Anthropic) expose more than one mode -// across their own model lineup. -enum ThinkingMode { +// ThinkingDisableSupport describes whether a model's reasoning can be +// turned off, per model/data-types.md#thinkingspec. A plain bool cannot +// express the real answer for every model: Anthropic's Opus 5 accepts an +// explicit disable at effort "high" or below and returns a 400 at "xhigh" +// or "max", so both true and false are wrong for it. +enum ThinkingDisableSupport { // Zero value. Never valid when ThinkingSpec.supported is true; its // presence on the wire means a caller forgot to set the field. - THINKING_MODE_UNSPECIFIED = 0; - // The model has no extended-reasoning capability. Pairs with - // ThinkingSpec.supported == false. - THINKING_MODE_NONE = 1; - // The model always reasons, adaptively, with no caller-selectable - // effort level or budget. - THINKING_MODE_ALWAYS_ON_ADAPTIVE = 2; - // The caller selects one of a fixed set of named effort levels - // (ThinkingSpec.effort_levels). - THINKING_MODE_DISCRETE_EFFORT = 3; - // The caller selects a token budget within ThinkingSpec.budget_range. - THINKING_MODE_CONTINUOUS_BUDGET = 4; + THINKING_DISABLE_SUPPORT_UNSPECIFIED = 0; + // Reasoning cannot be turned off in any configuration — a researched + // Grok model defaults reasoning on with no off switch, and Anthropic's + // Fable 5 returns a 400 for an explicit disable. Also the correct value + // when ThinkingSpec.supported is false: there is nothing to disable. + THINKING_DISABLE_SUPPORT_NEVER = 1; + // An explicit disable is accepted in every configuration. + THINKING_DISABLE_SUPPORT_ALWAYS = 2; + // An explicit disable is accepted in some configurations and rejected in + // others. This protocol deliberately does not model WHICH: the condition + // is vendor-specific and expressing it would need a general constraint + // language. What this value buys the kernel is knowing that a failed + // disable attempt is a vendor policy response, not an adapter bug. + THINKING_DISABLE_SUPPORT_CONDITIONAL = 3; } -// ThinkingBudgetRange bounds the token budget a caller may request when -// ThinkingMode is THINKING_MODE_CONTINUOUS_BUDGET. +// ThinkingBudgetRange bounds the token budget a caller may request on a +// model whose ThinkingSpec declares a BudgetControl. Both bounds are +// inclusive. message ThinkingBudgetRange { // The smallest thinking-token budget this model accepts. int64 min = 1; @@ -141,66 +144,105 @@ message ThinkingBudgetRange { int64 max = 2; } +// EffortControl declares that a model accepts a named reasoning-effort +// level, and which levels, per model/data-types.md#thinkingspec. +message EffortControl { + // The selectable effort levels, e.g. ["low","medium","high","xhigh", + // "max"]. MUST be non-empty — a model with no selectable levels omits + // the whole EffortControl instead. + repeated string levels = 1; + + // The level the vendor applies when a request omits effort entirely. + // MUST be set, and MUST be one of `levels` — makes the vendor's actual + // default behavior visible/auditable via GetCapabilities rather than + // hidden in adapter code, so a kernel wanting deterministic behavior can + // always send an explicit override. + string default = 2; +} + +// BudgetControl declares that a model accepts an explicit reasoning-token +// budget, and its bounds, per model/data-types.md#thinkingspec. +message BudgetControl { + // The accepted token-budget range. MUST be present. + ThinkingBudgetRange range = 1; + + // The budget the vendor applies when a request omits one. MAY be + // omitted, which means the vendor reasons zero tokens by default. + optional int64 default = 2; + + // Whether the vendor still honors this control but steers callers to + // effort/adaptive instead, and MAY remove it in a later model. This + // status is per-model, never per-vendor: Anthropic's Opus 4.6 and + // Sonnet 4.6 accept a deprecated budget, Haiku 4.5 accepts ONLY the + // budget, and Opus 4.7 onward reject it with a 400 (declared by omitting + // BudgetControl entirely, not by setting this flag). + bool deprecated = 3; +} + // ThinkingSpec describes one model's extended-reasoning capability, per -// model.md §2. +// model/data-types.md#thinkingspec. +// +// These are independent axes, not one-of-N modes. A model MAY reason +// adaptively AND expose an effort ladder (Anthropic Opus 4.8, Sonnet 5), +// or accept an effort level AND a deprecated token budget (Opus 4.6, +// Sonnet 4.6). An earlier revision modeled this as a single mutually- +// exclusive enum, which forced every such model to declare a half-truth. message ThinkingSpec { - // Whether this model has any extended-reasoning capability at all. + reserved 2 to 6; + + reserved "budget_range", "can_disable", "default", "effort_levels", "mode"; + + // Whether this model has any extended-reasoning capability at all. When + // false, effort and budget MUST both be absent and adaptive_by_default + // MUST be false; `disable` is meaningless (there is nothing to disable), + // so UNSPECIFIED and NEVER are equivalent and a reader MUST treat them + // identically. That equivalence is what keeps an all-zero ThinkingSpec a + // valid declaration for a model that does not reason — the common case. + // Only a positive claim that reasoning CAN be turned off (ALWAYS or + // CONDITIONAL) contradicts supported == false. bool supported = 1; - // Which reasoning-control shape this model uses. MUST be - // THINKING_MODE_NONE when supported == false. - ThinkingMode mode = 2; + // The named-effort-level control, present iff this model accepts one. + // Absent means sending GenerationParams.thinking_effort to this model is + // a kernel-level reject, not something forwarded to the vendor. + optional EffortControl effort = 7; - // The selectable effort levels, e.g. ["low","medium","high","xhigh", - // "max"]. MUST be non-empty when mode == THINKING_MODE_DISCRETE_EFFORT; - // meaningless otherwise. - repeated string effort_levels = 3; - - // The selectable token-budget range. MUST be present when mode == - // THINKING_MODE_CONTINUOUS_BUDGET; meaningless otherwise. - optional ThinkingBudgetRange budget_range = 4; - - // Whether reasoning can be turned off once enabled. MUST be set - // accurately — some vendors' reasoning cannot be disabled (e.g. a - // researched Grok model defaults reasoning on with no off switch). - bool can_disable = 5; - - // The effort level (discrete_effort) or budget-token value - // (continuous_budget), as a string, the vendor applies when a request - // omits thinking config entirely. MUST be set when mode != - // THINKING_MODE_NONE — makes the vendor's actual default behavior - // visible/auditable via GetCapabilities rather than hidden in adapter - // code, so a kernel wanting deterministic behavior can always send an - // explicit override. - optional string default = 6; -} + // The explicit-token-budget control, present iff this model accepts one. + // A model that never had one, and a model whose vendor removed it, + // both declare it absent. + optional BudgetControl budget = 8; -// CachingMode enumerates the prompt-caching mechanics found across -// researched vendors (model.md §2). -enum CachingMode { - // Zero value. Never valid when CachingSpec.supported is true; its - // presence on the wire means a caller forgot to set the field. - CACHING_MODE_UNSPECIFIED = 0; - // The model has no prompt-caching capability. Pairs with - // CachingSpec.supported == false. - CACHING_MODE_NONE = 1; - // The caller must place cache breakpoints on content blocks explicitly - // (Anthropic/Mistral-style). - CACHING_MODE_EXPLICIT_MARKERS = 2; - // The vendor applies caching transparently above a token threshold, no - // caller action required. - CACHING_MODE_IMPLICIT_AUTOMATIC = 3; + // Whether omitting every thinking control still produces reasoning. + // False means an unconfigured request reasons zero tokens. + bool adaptive_by_default = 9; + + // Whether, and when, reasoning can be turned off. MUST be set when + // supported is true. + ThinkingDisableSupport disable = 10; } // CachingSpec describes one model's prompt-caching capability, per -// model.md §2. +// model/data-types.md#cachingspec. +// +// These are independent axes, not one-of-N modes, for the same reason +// ThinkingSpec's are: Google's Gemini 2.5 and later run implicit automatic +// caching by default AND offer explicit manual declaration concurrently at +// a deeper discount. An earlier revision modeled this as a single +// mutually-exclusive enum, which forced such a model to under-declare +// itself — and, because cache_breakpoints were gated on that enum naming +// EXPLICIT_MARKERS, required it to discard breakpoints it could in fact +// have honored. message CachingSpec { - // Whether this model has any prompt-caching capability at all. - bool supported = 1; + reserved 2; - // Which caching mechanic this model uses. MUST be CACHING_MODE_NONE - // when supported == false. - CachingMode mode = 2; + reserved "mode"; + + // Whether this model has any prompt-caching capability at all. When + // false, explicit_markers and implicit_automatic MUST both be false; + // when true, at least one of them MUST be true — a model caching by a + // mechanism this protocol cannot name is not declarable, and declaring + // neither reads as "no caching" to every caller. + bool supported = 1; // Whether this provider runs its own cache-keepalive loop (e.g. a // background goroutine re-pinging before a cache TTL expires, so a long @@ -208,8 +250,22 @@ message CachingSpec { // default false. Cache TTL mechanics are vendor-specific, so per // operator decision this is a provider-owned behavior the kernel never // drives — this field only tells the kernel/operator whether a given - // provider implements the optimization (model.md §2). + // provider implements the optimization. bool keepalive_supported = 3; + + // Whether the caller may place cache breakpoints on content blocks and + // have the adapter translate them into vendor-native markers (an + // Anthropic cache_control block, a Mistral prompt_cache_key). This is + // the axis StreamCompletionRequest.cache_breakpoints is gated on: an + // adapter for a model that does not declare it MUST ignore that field + // rather than error on it. + bool explicit_markers = 4; + + // Whether the vendor caches transparently above some token threshold + // with no caller action. Declaring this requires nothing of the kernel; + // it exists so cache-hit and cost behavior are explicable rather than + // surprising. + bool implicit_automatic = 5; } // PricingTier is one time-bounded, input-size-bounded rate within a @@ -433,11 +489,59 @@ message Usage { // future Pricing revision declares a distinct reasoning rate — there is // none as of this revision. optional int64 reasoning_tokens = 5; + + // The vendor's own rate-limit state as of this completion, when it + // reports any. MAY be empty — a vendor that publishes nothing has + // nothing to declare, and an adapter MUST NOT synthesize a snapshot. + // + // Repeated because vendors publish several budgets at once and they + // exhaust independently: OpenAI and xAI return separate request and + // token headers, Anthropic reports input and output separately. Naming + // which budget is close to empty is the whole point — "you have 2% + // left" is unactionable without saying 2% of what. + repeated RateLimitSnapshot rate_limits = 6; + // Deliberately no cost field: the kernel computes and persists // cost_usd from these token counts plus the matching PricingTier, per // model.md §4.1 — the provider never computes cost itself. } +// RateLimitKind names which vendor budget a RateLimitSnapshot describes. +enum RateLimitKind { + // Zero value. Never valid on a real snapshot; its presence on the wire + // means an adapter forgot to set the field. + RATE_LIMIT_KIND_UNSPECIFIED = 0; + // Requests per window. + RATE_LIMIT_KIND_REQUESTS = 1; + // Tokens per window, undifferentiated by direction. + RATE_LIMIT_KIND_TOKENS = 2; + // Input tokens per window, where the vendor meters them separately. + RATE_LIMIT_KIND_INPUT_TOKENS = 3; + // Output tokens per window, where the vendor meters them separately. + RATE_LIMIT_KIND_OUTPUT_TOKENS = 4; +} + +// RateLimitSnapshot is one of the vendor's rate-limit budgets as of one +// completion, per model/data-types.md#streamevent. +// +// Every numeric field is optional because vendors publish different +// subsets: an adapter reports what its vendor actually returned and omits +// the rest rather than inventing a value. A snapshot with only `kind` set +// is still useful — it says the budget exists. +message RateLimitSnapshot { + // Which budget this describes. MUST be set. + RateLimitKind kind = 1; + + // How much of this budget remains. + optional int64 remaining = 2; + + // This budget's ceiling for the current window. + optional int64 limit = 3; + + // When this budget next resets. + optional google.protobuf.Timestamp reset_at = 4; +} + // ModelTarget describes the model a context or memory contribution is // being assembled for, derived from that model's ModelSpec // (model.md §2). Carried on context.md's ContextRequest and memory.md's diff --git a/cmd/providerconform/main.go b/cmd/providerconform/main.go new file mode 100644 index 0000000..1b8bd5e --- /dev/null +++ b/cmd/providerconform/main.go @@ -0,0 +1,174 @@ +// Command providerconform checks a built model-provider plugin against +// the conformance suite and reports what it found. +// +// providerconform [flags] +// +// It launches the binary the way the kernel does — a real handshake, a +// real subprocess, a real dispense — so it exercises the plugin's own +// main() wiring, and so it works on a plugin written in any language: +// it speaks nothing but the wire protocol. +// +// The assertions are pkg/model/modeltest's, identical to the ones a Go +// author gets from modeltest.Run in their own test. This binary exists +// for the two cases that cannot reach those: a plugin not written in Go, +// and an operator who wants to check a binary they did not build. +// +// Exit codes are meant to be scripted against: +// +// 0 no violations (skips may still be reported) +// 1 at least one violation +// 2 the binary could not be launched or checked at all +// +// 1 and 2 are deliberately distinct. A binary that will not start has not +// failed the suite, it has failed to be tested, and a CI job that +// conflates the two reports a conformance regression when the real +// problem is a bad path or a missing execute bit. +// +// Everything here is flag parsing and wiring, per +// .claude/rules/go-layout.md; the checking itself lives in +// pkg/model/modeltest so a Go author and this binary can never drift +// apart in what they assert. +package main + +import ( + "context" + "encoding/json" + "errors" + "flag" + "fmt" + "io" + "os" + "os/signal" + "strings" + "syscall" + + "google.golang.org/protobuf/types/known/structpb" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +// Exit codes, named so the doc comment above and the code cannot drift. +const ( + exitOK = 0 + exitViolated = 1 + exitUnusable = 2 +) + +// errUsage reports a command line the tool cannot act on. +var errUsage = errors.New("usage: providerconform [flags] ") + +// errFlagsReported marks a flag-parsing failure the flag package has +// already written to stderr. main returns it without printing, so the +// operator sees one explanation rather than two. +var errFlagsReported = errors.New("flag parsing failed") + +func main() { + // signal.NotifyContext so an interrupted run tears the plugin + // subprocess down rather than orphaning it. Released explicitly rather + // than deferred, because os.Exit below would skip a defer. + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + + code, err := run(ctx, os.Args[1:], os.Stdout) + if err != nil && !errors.Is(err, errFlagsReported) { + fmt.Fprintln(os.Stderr, "providerconform:", err) + } + + stop() + os.Exit(code) +} + +// run parses args, runs the suite, and writes the report to out. +// +// Split from main so every path returns rather than calling os.Exit, +// which would skip deferred cleanup (go-style.md). +func run(ctx context.Context, args []string, out io.Writer) (int, error) { + fs := flag.NewFlagSet("providerconform", flag.ContinueOnError) + fs.SetOutput(os.Stderr) + + configPath := fs.String("config", "", + "path to a JSON file passed to the plugin's Configure RPC. Point the provider at a recorded transcript or a local test server here — a conformance run must not make billed vendor calls.") + modelID := fs.String("model", "", + "which advertised model to exercise. Defaults to the first the plugin advertises.") + timeout := fs.Duration("timeout", modeltest.DefaultCallTimeout, + "per-RPC timeout. Lower it against a fake, where any real delay means the plugin is wedged.") + + if err := fs.Parse(args); err != nil { + return exitUnusable, errFlagsReported + } + if fs.NArg() != 1 { + fs.Usage() + return exitUnusable, errUsage + } + binary := fs.Arg(0) + + opts := []modeltest.Option{modeltest.WithCallTimeout(*timeout)} + if *modelID != "" { + opts = append(opts, modeltest.WithModelID(*modelID)) + } + if *configPath != "" { + cfg, err := loadConfig(*configPath) + if err != nil { + return exitUnusable, err + } + opts = append(opts, modeltest.WithConfig(cfg)) + } + + report, err := modeltest.CheckBinary(ctx, binary, opts...) + if err != nil { + return exitUnusable, err + } + + if _, err := io.WriteString(out, formatReport(binary, report)); err != nil { + return exitUnusable, fmt.Errorf("writing the report: %w", err) + } + if !report.OK() { + return exitViolated, nil + } + return exitOK, nil +} + +// loadConfig reads a JSON object into the Struct Configure expects. +func loadConfig(path string) (*structpb.Struct, error) { + data, err := os.ReadFile(path) // #nosec G304 -- the operator names this file; reading it is the flag's purpose + if err != nil { + return nil, fmt.Errorf("reading -config: %w", err) + } + var raw map[string]any + if err := json.Unmarshal(data, &raw); err != nil { + return nil, fmt.Errorf("parsing -config as a JSON object: %w", err) + } + cfg, err := structpb.NewStruct(raw) + if err != nil { + return nil, fmt.Errorf("converting -config: %w", err) + } + return cfg, nil +} + +// formatReport renders the findings and a one-line summary. +// +// Built as a string and written once rather than printed piecemeal, so +// the single write is the only place that can fail and its error is +// actually checked. +// +// Skips appear as prominently as failures on purpose: a skip means a +// requirement was not reached, and the whole point of reporting them is +// that an unexercised check must never read as a pass. +func formatReport(binary string, report modeltest.Report) string { + var sb strings.Builder + fmt.Fprintf(&sb, "conformance: %s\n\n", binary) + if body := report.String(); body != "" { + sb.WriteString(body) + sb.WriteString("\n") + } + + failures, skips := len(report.Failures()), len(report.Skips()) + switch { + case failures == 0 && skips == 0: + sb.WriteString("PASS — every check satisfied\n") + case failures == 0: + fmt.Fprintf(&sb, "PASS — no violations, %d check(s) not reached\n", skips) + default: + fmt.Fprintf(&sb, "FAIL — %d violation(s), %d check(s) not reached\n", failures, skips) + } + return sb.String() +} diff --git a/cmd/providerconform/main_test.go b/cmd/providerconform/main_test.go new file mode 100644 index 0000000..1be4b84 --- /dev/null +++ b/cmd/providerconform/main_test.go @@ -0,0 +1,146 @@ +package main + +import ( + "context" + "errors" + "io" + "os" + "path/filepath" + "strings" + "testing" + + "github.com/pluggableharness/agent/pkg/model/modeltest" +) + +func TestRun_exitCodesDistinguishUnusableFromViolated(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + args []string + wantCode int + }{ + // A binary that will not start has not failed the suite, it has + // failed to be tested. A CI job conflating the two reports a + // conformance regression when the real problem is a bad path. + "missing binary": { + args: []string{filepath.Join(t.TempDir(), "does-not-exist")}, + wantCode: exitUnusable, + }, + "no binary named": { + args: nil, + wantCode: exitUnusable, + }, + "too many arguments": { + args: []string{"a", "b"}, + wantCode: exitUnusable, + }, + "unreadable config": { + args: []string{"-config", filepath.Join(t.TempDir(), "absent.json"), "some-binary"}, + wantCode: exitUnusable, + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + code, _ := run(context.Background(), tt.args, io.Discard) + if code != tt.wantCode { + t.Errorf("run(%v) = %d, want %d", tt.args, code, tt.wantCode) + } + }) + } +} + +func TestRun_missingBinaryNamesTheBinary(t *testing.T) { + t.Parallel() + + missing := filepath.Join(t.TempDir(), "not-a-plugin") + _, err := run(context.Background(), []string{missing}, io.Discard) + if err == nil { + t.Fatal("run() = nil error for a missing binary") + } + // The path has to appear, or an operator with several plugins cannot + // tell which one failed to launch. + if !strings.Contains(err.Error(), missing) { + t.Errorf("error %q does not name the binary %q", err, missing) + } +} + +func TestRun_usageErrorIsDistinguishable(t *testing.T) { + t.Parallel() + + _, err := run(context.Background(), nil, io.Discard) + if !errors.Is(err, errUsage) { + t.Errorf("err = %v, want errUsage", err) + } +} + +func TestLoadConfig(t *testing.T) { + t.Parallel() + + dir := t.TempDir() + + valid := filepath.Join(dir, "valid.json") + if err := os.WriteFile(valid, []byte(`{"api_key":"sk-test","port":8080}`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + cfg, err := loadConfig(valid) + if err != nil { + t.Fatalf("loadConfig: %v", err) + } + if got := cfg.GetFields()["api_key"].GetStringValue(); got != "sk-test" { + t.Errorf("api_key = %q, want sk-test", got) + } + + malformed := filepath.Join(dir, "bad.json") + if err := os.WriteFile(malformed, []byte(`not json`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := loadConfig(malformed); err == nil { + t.Error("loadConfig accepted malformed JSON") + } + + // A JSON array is valid JSON but not a config object; rejecting it + // here beats a confusing failure inside Configure. + array := filepath.Join(dir, "array.json") + if err := os.WriteFile(array, []byte(`[1,2,3]`), 0o600); err != nil { + t.Fatalf("write: %v", err) + } + if _, err := loadConfig(array); err == nil { + t.Error("loadConfig accepted a JSON array as a config object") + } +} + +func TestFormatReport_summarizesEachOutcome(t *testing.T) { + t.Parallel() + + tests := map[string]struct { + findings []modeltest.Finding + want string + }{ + "clean": {want: "PASS — every check satisfied"}, + "skips only": { + findings: []modeltest.Finding{{Check: "a", Severity: modeltest.SeveritySkip, Message: "not reached"}}, + want: "PASS — no violations, 1 check(s) not reached", + }, + "violations": { + findings: []modeltest.Finding{{Check: "a", Severity: modeltest.SeverityFail, Message: "violated"}}, + want: "FAIL — 1 violation(s), 0 check(s) not reached", + }, + } + + for name, tt := range tests { + t.Run(name, func(t *testing.T) { + t.Parallel() + + got := formatReport("some-binary", modeltest.Report{Findings: tt.findings}) + if !strings.Contains(got, tt.want) { + t.Errorf("formatReport =\n%s\nwant a line containing %q", got, tt.want) + } + if !strings.Contains(got, "some-binary") { + t.Errorf("formatReport does not name the binary:\n%s", got) + } + }) + } +} diff --git a/docs/first-party/providers/anthropic.md b/docs/first-party/providers/anthropic.md index 2b6b844..5cd40e9 100644 --- a/docs/first-party/providers/anthropic.md +++ b/docs/first-party/providers/anthropic.md @@ -59,9 +59,15 @@ Unlike the other three first-party vendors, there is no mini/flash/codex-style t ## 6. Implications for PluggableHarness Agent -**`ThinkingSpec`** ([`data-types.md#thinkingspec`](../../specifications/model/data-types.md#thinkingspec)): Anthropic is the vendor that most directly motivated this type's sum-type shape rather than a boolean flag. A conformant adapter cannot declare one `ThinkingSpec` for the whole plugin — it must vary per `ModelSpec`: `mode: always_on_adaptive` for Opus 4.8 and Sonnet 5 (with `can_disable` reflecting that `budget_tokens` now 400s rather than silently degrading), `mode: continuous_budget` for Haiku 4.5 (which never gained adaptive support), and — for Sonnet 4.6, which genuinely supports both mechanisms simultaneously during its transitional window — the adapter author will need to pick one canonical `mode` to declare (adaptive is recommended by Anthropic) while being aware `budget_tokens` still functions underneath as an escape hatch not directly representable in a single `ThinkingSpec` value. The `default` field matters concretely here: because Sonnet 5 runs adaptive thinking even when a request omits `thinking` entirely, a kernel wanting deterministic, budget-bounded behavior must know to always send an explicit override rather than relying on omission meaning "no thinking." +**`ThinkingSpec`** ([`data-types.md#thinkingspec`](../../specifications/model/data-types.md#thinkingspec)): Anthropic is the vendor that most directly motivated this type's per-axis shape, and the one whose lineup an earlier single-mode enum could not describe. A conformant adapter declares a `ThinkingSpec` per `ModelSpec`, not one per plugin, and each model declares every control it actually accepts: -**`CachingSpec`** ([`data-types.md#cachingspec`](../../specifications/model/data-types.md#cachingspec)): every current Anthropic model sets `mode: explicit_markers` — the adapter is responsible for placing `cache_control` breakpoints on content blocks per `data-types.md`'s explicit-markers semantics, not for detecting automatic caching. Given the 5-minute standard TTL, an adapter maintaining a long-running agentic session across tool-execution gaps is a strong candidate for implementing the optional `keepalive_supported` behavior described in `data-types.md`'s cache-keepalive note, since Anthropic's own API gives the adapter no server-side keepalive to rely on. +- Opus 4.8 and Sonnet 5 set `adaptive_by_default: true` **and** an `effort` control. Both are true at once — omitting thinking config still reasons, and `output_config.effort` selects a level on top of that — which is why the adapter sends `thinking: {type: "adaptive"}` alongside the effort level rather than instead of it. +- Haiku 4.5 declares a `budget` control and no `effort` control, with `adaptive_by_default: false`: omitting the thinking parameter there means no reasoning at all, not adaptive reasoning. +- Sonnet 4.6's transitional window — where `budget_tokens` still functions underneath the effort ladder — is declarable directly: both controls present, with `budget.deprecated: true`. No canonical mode has to be picked, and nothing is lost. + +`effort.default` matters concretely: because these models reason even when a request omits thinking entirely, a kernel wanting deterministic behavior reads the declared default and sends an explicit override rather than relying on omission meaning "no thinking." Opus 5's disable is `conditional` — Anthropic accepts an explicit disable at effort `high` or below and rejects it above — which tells the kernel a failed disable is vendor policy rather than an adapter bug. + +**`CachingSpec`** ([`data-types.md#cachingspec`](../../specifications/model/data-types.md#cachingspec)): every current Anthropic model sets `explicit_markers: true` and `implicit_automatic: false` — the adapter is responsible for placing `cache_control` breakpoints on content blocks per `data-types.md`'s explicit-markers semantics, not for detecting automatic caching. Given the 5-minute standard TTL, an adapter maintaining a long-running agentic session across tool-execution gaps is a strong candidate for implementing the optional `keepalive_supported` behavior described in `data-types.md`'s cache-keepalive note, since Anthropic's own API gives the adapter no server-side keepalive to rely on. **Tool schema and `ToolCall`/`ToolResult`** ([`data-types.md#tool-schema`](../../specifications/model/data-types.md#tool-schema)): Anthropic is one of the vendors (with Google and Ollama) whose tool-call arguments already arrive as a parsed object, so the Anthropic adapter's translation at the string/object boundary is the simpler direction — it serializes the kernel's parsed-JSON internal representation directly into `input` with no encode/decode step, unlike an OpenAI-shaped adapter. The adapter does need to handle the parallel-tool-call batching rule from §4 above (all `tool_result` blocks for one turn must land in a single `user` message) since this is stricter than what the generic protocol assumes about one-result-per-message. diff --git a/docs/first-party/providers/google.md b/docs/first-party/providers/google.md index 79723e3..d9da041 100644 --- a/docs/first-party/providers/google.md +++ b/docs/first-party/providers/google.md @@ -65,9 +65,13 @@ As a general pattern: confidence tracks how flagship a model is. `gemini-3-pro` ## 6. Implications for PluggableHarness Agent -**Per-model `ThinkingSpec.mode`, not a per-vendor constant.** Google's own lineup spans three different values of the [`ThinkingSpec`](../../specifications/model/data-types.md#thinkingspec) `mode` enum on its own: `discrete_effort` for the 3.x line (with `effort_levels` populated from the LOW/MEDIUM/HIGH or MINIMAL/LOW/MEDIUM/HIGH sets and `default` set to that model's actual default, e.g. `"HIGH"` for `gemini-3-pro`), `continuous_budget` for the 2.5 line (`budget_range` populated from `thinkingBudget`'s token-count bounds), and effectively `none` for `gemini-1.5-pro`. This is exactly the scenario `data-types.md` cites as the reason `ThinkingSpec` lives on each `ModelSpec` rather than being a single vendor-level flag — a Google adapter must build a distinct `ThinkingSpec` per model, and must not assume the `thinking_level` vs `thinkingBudget` parameter name generalizes across the whole roster. +**Per-model `ThinkingSpec`, not a per-vendor constant.** Google's own lineup occupies three different positions on [`ThinkingSpec`](../../specifications/model/data-types.md#thinkingspec)'s axes: the 3.x line declares an `effort` control (levels from the LOW/MEDIUM/HIGH or MINIMAL/LOW/MEDIUM/HIGH sets, with `effort.default` set to that model's actual default, e.g. `"HIGH"` for `gemini-3-pro`), the 2.5 line declares a `budget` control (range from `thinkingBudget`'s token-count bounds), and `gemini-1.5-pro` declares neither and leaves `supported` false. This is exactly the scenario `data-types.md` cites as the reason `ThinkingSpec` lives on each `ModelSpec` rather than being a single vendor-level flag — a Google adapter must build a distinct `ThinkingSpec` per model, and must not assume the `thinking_level` vs `thinkingBudget` parameter name generalizes across the whole roster. -**`CachingSpec.mode` cannot represent Google's actual behavior as a single value.** [`CachingSpec`](../../specifications/model/data-types.md#cachingspec)'s `mode` field is a single enum per model (`none` / `explicit_markers` / `implicit_automatic`) — a sum in the "one active mode" sense, not a set. But Google's 2.5+ models genuinely run both simultaneously: implicit automatic caching is on by default (75% discount, no caller action) *and* explicit manual declaration is available concurrently for a deeper 90% discount. Neither enum value alone captures this. A plugin author building this adapter has to make a real choice here rather than treating it as a formality: the pragmatic default is `mode: implicit_automatic` (it matches what happens when the caller does nothing, which is the common case), with the explicit/manual pathway and its better discount rate exposed separately — e.g. surfaced only through documentation or a provider-specific `Configure` option rather than through `CachingSpec` itself. This is worth flagging back to the protocol's designers as a real gap: the current `CachingSpec` shape has no way to declare "this model supports two caching modes concurrently, with different discount rates," which is precisely the situation Google's own docs describe. +**Google's two concurrent caching mechanisms are declarable directly.** Its 2.5+ models genuinely run both at once: implicit automatic caching is on by default (75% discount, no caller action) *and* explicit manual declaration is available concurrently for a deeper 90% discount. [`CachingSpec`](../../specifications/model/data-types.md#cachingspec)'s axes are independent, so such a model sets `implicit_automatic: true` **and** `explicit_markers: true`, and no canonical mode has to be chosen. + +This page previously flagged the inability to express that as a gap to raise with the protocol's designers; the gap is closed. It mattered for more than accuracy: `cache_breakpoints` are gated on the explicit-markers axis, so a model forced to declare only implicit caching was thereby required to discard breakpoints it could have honored — losing the deeper discount with no error anywhere. + +One related gap remains open and is recorded in [`conformance.md`](../../specifications/model/conformance.md#open-questions): `PricingTier` carries a single cache-rate pair, so a model billing implicit and explicit hits at different rates cannot price both. Declaring both mechanisms is now correct; pricing them separately still needs a `Pricing` revision. **Tool-call arguments arrive pre-parsed.** Per the [tool schema](../../specifications/model/data-types.md#tool-schema) section's cross-vendor note, Google — like Anthropic and Ollama, unlike OpenAI and Mistral — delivers `functionCall.args` as an already-parsed object rather than a JSON-encoded string. The adapter's `ToolCall` translation should pass this straight through into the kernel's internal parsed-JSON representation without a parse step, and the reverse `function_response` submission needs no re-encoding to a string either. diff --git a/docs/first-party/providers/openai.md b/docs/first-party/providers/openai.md index 5d00ceb..428debc 100644 --- a/docs/first-party/providers/openai.md +++ b/docs/first-party/providers/openai.md @@ -57,9 +57,9 @@ Authentication is an HTTP Bearer token: `Authorization: Bearer