Bump @typesafe-ai/sdk from 0.6.0 to 0.7.0 in the runtime group across 1 directory

CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## Unreleased
 
+### Changed
+
+- A `box` question is refused with a `validation` error that names it. `@typesafe-ai/sdk` 0.7.0 added the type, which locates a subject in the request's image; this package sends `noul`, `choice`, and `score`, and its state carries no image.
+- A `null` state is refused with a `validation` error that names it, because 0.7.0 types `state` as a non-null entry.
+- A token count the API did not report (`usage.input_tokens` or `output_tokens` is `null` in 0.7.0) adds nothing to the session or daily totals and costs nothing, and the result renders it as `not reported` instead of `null`.
+- Requests are sent uncompressed. 0.7.0 compresses any body of 1 KiB or more with zstd by default, and none of this package's five backends is measured for that encoding.
+
 ## 0.12.0
 
 ### Added

README.md

@@ -74,7 +74,7 @@ The [TypeSafe docs](https://docs.typesafe.ai/primitives) cover each primitive in
 An enabled extension with no usable key looks exactly like a working one — one evaluation ran keyless for hours before anyone noticed. Three answers exist now, and `/typesafe status` prints all of them:
 
 - **Is Jev reachable?** `authState()` reports the key source, whether it has been accepted, and the last failure that degraded it; `describeAuth()` turns that into a level and one safe line. The extension calls both at session start and after an authentication rejection, so a headless run says judgments are skipped instead of quietly falling back to its offline path.
-- **What has this run cost?** `getSpend()` returns session counters, today's persisted counters, and the cap currently reached. Cost is estimated from input tokens only, because output is free.
+- **What has this run cost?** `getSpend()` returns session counters, today's persisted counters, and the cap currently reached. Cost is estimated from input tokens only, because output is free; a count the API did not report counts as nothing.
 - **When does it stop?** `maxRequests` bounds a client instance. Three caps bound a local day, survive restarts, and stop a request before it is submitted:
 
 | Option | Environment | Bounds |

docs/api.md

@@ -69,7 +69,7 @@ Every backend takes its key from its own environment variable first, then from t
 
 ## Admission
 
-`prepareEvaluationRequest(value, { maxInputBytes, limits })` is the one admission rule, used by the tool, the playground, and `evaluate`. It normalizes the near-miss aliases a model produces (`options` / `levels` / `choices` for `criteria`, a string Noul criterion, a label array for a Choice), validates the schema and JSON-safety, checks `limits` when given, then enforces the byte budget. `DEFAULT_MAX_INPUT_BYTES`, `DEFAULT_MAX_QUESTIONS`, and `DEFAULT_MAX_REQUESTS` hold the shared defaults.
+`prepareEvaluationRequest(value, { maxInputBytes, limits })` is the one admission rule, used by the tool, the playground, and `evaluate`. It normalizes the near-miss aliases a model produces (`options` / `levels` / `choices` for `criteria`, a string Noul criterion, a label array for a Choice), validates the schema and JSON-safety, checks `limits` when given, then enforces the byte budget. Two shapes it refuses by name: a `box` question — 0.7.0's own question type for locating a subject in the request's image, while this package sends only `noul`, `choice`, and `score` — and a `null` state, which 0.7.0 no longer accepts. Both raise a `validation` error that names the field. `DEFAULT_MAX_INPUT_BYTES`, `DEFAULT_MAX_QUESTIONS`, and `DEFAULT_MAX_REQUESTS` hold the shared defaults.
 
 ## Limits
 
@@ -109,7 +109,7 @@ Day caps live in `~/.pi/agent/pi-typesafe/usage.json` (owner-only, atomic, best-
 | `maxInputTokensPerDay` | `PI_TYPESAFE_MAX_INPUT_TOKENS_PER_DAY` | input tokens |
 | `maxUsdPerDay` | `PI_TYPESAFE_MAX_USD_PER_DAY` | estimated spend |
 
-The environment may lower an explicit cap, never raise it. A reached cap raises a `budget` error that names the cap, the amount used, and the day, before anything is submitted. Cost is estimated from input tokens only, because output is free. Each request is priced when it is recorded, at its own model's price, so one ledger can hold requests to backends and models with different prices and the day's spend, and the `maxUsdPerDay` cap, add up what each request cost; requests to a free model add nothing. A ledger file written before costs were recorded keeps working: days without a recorded cost are estimated at the ledger's `usdPerMTok` (default `0.042`).
+The environment may lower an explicit cap, never raise it. A reached cap raises a `budget` error that names the cap, the amount used, and the day, before anything is submitted. Cost is estimated from input tokens only, because output is free. Each request is priced when it is recorded, at its own model's price, so one ledger can hold requests to backends and models with different prices and the day's spend, and the `maxUsdPerDay` cap, add up what each request cost; requests to a free model add nothing. A token count the API did not report (0.7.0 reports `null`) adds nothing to the session or daily totals and is priced at zero, because there is nothing to count or price; the request still counts as an attempt against the request cap. A ledger file written before costs were recorded keeps working: days without a recorded cost are estimated at the ledger's `usdPerMTok` (default `0.042`).
 
 `openUsageLedger(options)`, `usagePath()`, `estimateUsd(tokens, usdPerMTok)`, `capsFromEnvironment(env)`, and `mergeCaps(explicit, environment)` expose the same arithmetic for your own display.
 

package.json

@@ -64,7 +64,7 @@
     "prepack": "npm run build"
   },
   "dependencies": {
-    "@typesafe-ai/sdk": "^0.6.0"
+    "@typesafe-ai/sdk": "^0.7.0"
   },
   "peerDependencies": {
     "@earendil-works/pi-coding-agent": ">=0.85.1 <1",

src/batch.ts

@@ -75,7 +75,7 @@ export interface BatchEvaluation<Q extends Questions = Questions> {
   readonly answers: Partial<Evaluation<Q>["answers"]>;
   /** The model of the first successful request, when there is one. */
   readonly model?: string;
-  /** Usage summed over the requests that succeeded. */
+  /** Usage summed over the requests that succeeded; a token count the API did not report adds nothing. */
   readonly usage: Usage;
   /** Wall-clock time for the whole batch. */
   readonly elapsedMs: number;
@@ -95,8 +95,8 @@ function summarize<Q extends Questions>(results: readonly Settled<Evaluation<Q>>
     }
     succeeded++;
     Object.assign(answers, result.value.answers);
-    inputTokens += result.value.usage.input_tokens;
-    outputTokens += result.value.usage.output_tokens;
+    inputTokens += result.value.usage.input_tokens ?? 0;
+    outputTokens += result.value.usage.output_tokens ?? 0;
     model ??= result.value.model;
   }
   return {

src/client.ts

@@ -151,8 +151,9 @@ function positiveNumber(value: number, label: string): number {
 function validResult<Q extends Questions>(result: SystemOneResult<Q>, questions: Q): boolean {
   const probability = (value: number) => Number.isFinite(value) && value >= 0 && value <= 1;
   if (!result || typeof result.model !== "string" || !result.model || !result.usage || !result.answers) return false;
+  // 0.7.0 reports a token count it did not measure as null; that is not malformed, it is unreported.
   for (const count of [result.usage.input_tokens, result.usage.output_tokens]) {
-    if (!Number.isSafeInteger(count) || count < 0) return false;
+    if (count !== null && (!Number.isSafeInteger(count) || count < 0)) return false;
   }
   if (Object.keys(result.answers).length !== Object.keys(questions).length) return false;
   for (const [id, question] of Object.entries(questions)) {
@@ -162,13 +163,17 @@ function validResult<Q extends Questions>(result: SystemOneResult<Q>, questions:
       if (!probability(answer.noul)) return false;
       continue;
     }
+    // A box answer answers a question this package refuses to send, so it never validates.
+    if (answer.type === "box") return false;
     if (!probability(answer.confidence) || !answer.probabilities) return false;
+    // Rubric length, not its elements: 0.7.0's content type is recursive and mapping it exhausts the compiler.
     const keys = question.type === "choice" ? Object.keys(question.criteria)
-      : question.type === "score" ? question.criteria.map((_, index) => String(index)) : [];
+      : question.type === "score" ? Array.from({ length: question.criteria.length }, (_, index) => String(index)) : [];
     const probabilities = new Map(Object.entries(answer.probabilities));
     if (probabilities.size !== keys.length || keys.some(key => !probability(probabilities.get(key) ?? NaN))) return false;
     if (answer.type === "choice" && !keys.includes(answer.choice)) return false;
-    if (answer.type === "score" && (!Number.isFinite(answer.score) || answer.score < 0 || answer.score > keys.length - 1 || !answer.legend)) return false;
+    // Presence of own `legend`, not its value: 0.7.0's legend type carries the recursive content type, which exhausts the compiler.
+    if (answer.type === "score" && (!Number.isFinite(answer.score) || answer.score < 0 || answer.score > keys.length - 1 || !Object.hasOwn(answer, "legend"))) return false;
   }
   return true;
 }
@@ -233,6 +238,8 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe {
     baseURL,
     timeout,
     retry: { maxRetries: 0 },
+    // 0.7.0 zstd-compresses any body of 1 KiB or more by default; this package's five backends are unmeasured for it.
+    compression: false,
     logLevel: "off",
     ...(transport ? { fetch: transport } : {}),
   });
@@ -301,12 +308,15 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe {
         const result = await client.systemOne(request, callOptions);
         if (!validResult(result, request.questions)) throw new TypeSafeIntegrationError("response", "TypeSafe returned an unexpected answer or usage format.");
         usage.requestsSucceeded += 1;
-        usage.inputTokens += result.usage.input_tokens;
-        usage.outputTokens += result.usage.output_tokens;
+        // An unreported count is not a count this package can add, price, or charge against a token cap.
+        const inputTokens = result.usage.input_tokens ?? 0;
+        const outputTokens = result.usage.output_tokens ?? 0;
+        usage.inputTokens += inputTokens;
+        usage.outputTokens += outputTokens;
         // Unrounded: a small request must still count toward the cap.
-        const cost = (result.usage.input_tokens * rateFor(sentModel)) / 1e6;
+        const cost = (inputTokens * rateFor(sentModel)) / 1e6;
         sessionUsd += cost;
-        ledger.recordSuccess(result.usage.input_tokens, result.usage.output_tokens, cost);
+        ledger.recordSuccess(inputTokens, outputTokens, cost);
         if (!verificationRecorded) {
           verificationRecorded = true;
           recordAuthVerified();

src/extension.ts

@@ -44,16 +44,20 @@ const sample = {
   },
 };
 
+/** 0.7.0 reports a count it did not measure as null; name it instead of printing "null". */
+const formatNumber = (count: number | null): string => (count === null ? "not reported" : String(count));
+
 function format(result: Evaluation<Questions>, expanded = false): string {
   const lines = [`TypeSafe · ${JSON.stringify(result.model)} · ${result.elapsedMs} ms`];
   for (const [id, answer] of Object.entries(result.answers)) {
     const label = JSON.stringify(id);
     if (answer.type === "noul") lines.push(`${label}: P(yes) = ${answer.noul.toFixed(3)}`);
     else if (answer.type === "choice") lines.push(`${label}: ${JSON.stringify(answer.choice)} · confidence ${answer.confidence.toFixed(3)}`);
-    else lines.push(`${label}: ${answer.score.toFixed(3)} · confidence ${answer.confidence.toFixed(3)}`);
-    if (expanded && answer.type !== "noul") lines.push(`  ${JSON.stringify(answer.probabilities)}`);
+    else if (answer.type === "score") lines.push(`${label}: ${answer.score.toFixed(3)} · confidence ${answer.confidence.toFixed(3)}`);
+    else lines.push(`${label}: box ${JSON.stringify(answer.box)} · confidence ${formatNumber(answer.confidence)}`);
+    if (expanded && (answer.type === "choice" || answer.type === "score")) lines.push(`  ${JSON.stringify(answer.probabilities)}`);
   }
-  lines.push(`${result.usage.input_tokens} input / ${result.usage.output_tokens} output tokens`);
+  lines.push(`${formatNumber(result.usage.input_tokens)} input / ${formatNumber(result.usage.output_tokens)} output tokens`);
   lines.push("Confidence is distribution concentration, not proof of correctness.");
   return lines.join("\n");
 }

src/schema.ts

@@ -26,12 +26,10 @@ export const DEFAULT_BACKEND_LIMITS: BackendLimits = Object.freeze({ maxQuestion
 
 // The API accepts structured descriptions, not only strings. The schema is the only shape guidance the model gets
 // before its first call, so every field the agent authors says what it means.
-const entry = (description?: string) => Type.Union([
-  Type.String(),
-  Type.Null(),
-  Type.Array(Type.Unknown()),
-  Type.Record(Type.String(), Type.Unknown()),
-], description === undefined ? {} : { description });
+const content = () => [Type.String(), Type.Array(Type.Unknown()), Type.Record(Type.String(), Type.Unknown())];
+const entry = (description?: string) => Type.Union([...content(), Type.Null()], description === undefined ? {} : { description });
+/** `state` alone is non-null: @typesafe-ai/sdk 0.7.0 types it as `NonNullable<EntryType>`. */
+const state = Type.Union(content(), { description: "What to judge: text, or an object whose fields the questions name. Not null." });
 const instructions = Type.Optional(entry("One judgment about the whole state, phrased as a question or a statement."));
 const question = Type.Union([
   Type.Object({
@@ -60,7 +58,7 @@ const question = Type.Union([
 
 /** The JSON schema used by both the Pi tool and the programmatic interface. */
 export const evaluationSchema = Type.Object({
-  state: entry("What to judge: text, or an object whose fields the questions name."),
+  state,
   questions: Type.Record(Type.String({ minLength: 1, maxLength: 100 }), question, {
     minProperties: 1,
     maxProperties: DEFAULT_MAX_QUESTIONS,
@@ -71,6 +69,29 @@ export const evaluationSchema = Type.Object({
 
 const usage = `Expected { state, questions: { <id>: { type: "choice", instructions, criteria: { label: description|null } } | { type: "score", instructions, criteria: [level0, level1, ...] } | { type: "noul", instructions } } }; 1–${DEFAULT_MAX_QUESTIONS} questions, Choice 1–64 options, Score 2–32 levels.`;
 
+/** Question types this package sends. The SDK also offers `box`, which locates a subject in the request's image. */
+const SUPPORTED_QUESTION_TYPES = new Set(["noul", "choice", "score"]);
+
+/**
+ * Name the two refusals the schema can otherwise only report as a union mismatch: a `box` question, and a `null`
+ * state. Both are shapes @typesafe-ai/sdk 0.7.0 added or tightened; neither is something this package can send.
+ */
+function refuseUnsupported(value: unknown): void {
+  if (!value || typeof value !== "object" || Array.isArray(value)) return;
+  const request = value as Record<string, unknown>;
+  if (request.state === null) {
+    throw new TypeSafeIntegrationError("validation", `state must not be null: send what to judge as text, an object, or an array. ${usage}`);
+  }
+  const questions = request.questions;
+  if (!questions || typeof questions !== "object" || Array.isArray(questions)) return;
+  for (const [id, question] of Object.entries(questions as Record<string, unknown>)) {
+    const type = question && typeof question === "object" && !Array.isArray(question) ? (question as Record<string, unknown>).type : undefined;
+    if (typeof type === "string" && !SUPPORTED_QUESTION_TYPES.has(type)) {
+      throw new TypeSafeIntegrationError("validation", `Invalid evaluation request at questions.${id.slice(0, 100)}.type: ${JSON.stringify(type.slice(0, 40))} is not a question type this package sends. ${usage}`);
+    }
+  }
+}
+
 /** Paths and messages only; never the submitted values. */
 function describeSchemaErrors(value: unknown): string {
   const details: string[] = [];
@@ -87,6 +108,7 @@ export function parseEvaluationRequest(value: unknown): SystemOneRequest {
   if (value !== null && typeof value === "object" && !Array.isArray(value) && !isJsonSafe(value)) {
     throw new TypeSafeIntegrationError("validation", `Invalid evaluation request: state and questions must be plain JSON. ${usage}`);
   }
+  refuseUnsupported(value);
   if (!Check(evaluationSchema, value)) {
     throw new TypeSafeIntegrationError("validation", `Invalid evaluation request at ${describeSchemaErrors(value)}. ${usage}`);
   }

tests/backends.test.ts

@@ -19,6 +19,7 @@ function responseFor(questions: Questions): Response {
       const keys = Object.keys(q.criteria);
       return [id, { type: "choice", choice: keys[0], confidence: 1, probabilities: Object.fromEntries(keys.map((key, i) => [key, i === 0 ? 1 : 0])) }];
     }
+    if (q.type !== "score") throw new Error(`the test reply has no answer for a ${q.type} question`);
     return [id, { type: "score", score: 0, confidence: 1, probabilities: Object.fromEntries(q.criteria.map((_, i) => [i, i === 0 ? 1 : 0])), legend: Object.fromEntries(q.criteria.map((level, i) => [i, level])) }];
   }));
   return Response.json({ model: "jev-test", answers, usage: { input_tokens: 42, output_tokens: 0 } });

tests/batch.test.ts

@@ -172,3 +172,15 @@ test("a daily cap stops the batch with the cap named", async () => {
   assert.ok(second.error instanceof TypeSafeIntegrationError && second.error.code === "budget");
   assert.ok(second.error.message.includes("daily request cap"));
 });
+
+test("an unreported token count adds nothing to the batch usage", async () => {
+  const questions = manyQuestions(2);
+  const client = createTypeSafe({
+    ledger: ledgerFor("null-usage"),
+    fetch: async () => Response.json({ model: "jev-test", answers: Object.fromEntries(Object.keys(questions).map(id => [id, { type: "noul", noul: 0.5 }])), usage: { input_tokens: null, output_tokens: null } }),
+  });
+  const batch = await client.evaluateAll({ state: "many", questions });
+  assert.equal(batch.ok, true);
+  assert.deepEqual(batch.usage, { input_tokens: 0, output_tokens: 0 });
+  assert.equal(client.getSpend().today.inputTokens, 0);
+});