diff --git a/CHANGELOG.md b/CHANGELOG.md index 54ee30d..116a6b4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,6 +2,13 @@ ## Unreleased +### Changed + +- A `box` question is refused with a `validation` error that names it. `@typesafe-ai/sdk` 0.7.0 added the type, which locates a subject in the request's image; this package sends `noul`, `choice`, and `score`, and its state carries no image. +- A `null` state is refused with a `validation` error that names it, because 0.7.0 types `state` as a non-null entry. +- A token count the API did not report (`usage.input_tokens` or `output_tokens` is `null` in 0.7.0) adds nothing to the session or daily totals and costs nothing, and the result renders it as `not reported` instead of `null`. +- Requests are sent uncompressed. 0.7.0 compresses any body of 1 KiB or more with zstd by default, and none of this package's five backends is measured for that encoding. + ## 0.12.0 ### Added diff --git a/README.md b/README.md index 2d0ace4..dd76d82 100644 --- a/README.md +++ b/README.md @@ -74,7 +74,7 @@ The [TypeSafe docs](https://docs.typesafe.ai/primitives) cover each primitive in An enabled extension with no usable key looks exactly like a working one — one evaluation ran keyless for hours before anyone noticed. Three answers exist now, and `/typesafe status` prints all of them: - **Is Jev reachable?** `authState()` reports the key source, whether it has been accepted, and the last failure that degraded it; `describeAuth()` turns that into a level and one safe line. The extension calls both at session start and after an authentication rejection, so a headless run says judgments are skipped instead of quietly falling back to its offline path. -- **What has this run cost?** `getSpend()` returns session counters, today's persisted counters, and the cap currently reached. Cost is estimated from input tokens only, because output is free. +- **What has this run cost?** `getSpend()` returns session counters, today's persisted counters, and the cap currently reached. Cost is estimated from input tokens only, because output is free; a count the API did not report counts as nothing. - **When does it stop?** `maxRequests` bounds a client instance. Three caps bound a local day, survive restarts, and stop a request before it is submitted: | Option | Environment | Bounds | diff --git a/docs/api.md b/docs/api.md index 8737297..318ecde 100644 --- a/docs/api.md +++ b/docs/api.md @@ -69,7 +69,7 @@ Every backend takes its key from its own environment variable first, then from t ## Admission -`prepareEvaluationRequest(value, { maxInputBytes, limits })` is the one admission rule, used by the tool, the playground, and `evaluate`. It normalizes the near-miss aliases a model produces (`options` / `levels` / `choices` for `criteria`, a string Noul criterion, a label array for a Choice), validates the schema and JSON-safety, checks `limits` when given, then enforces the byte budget. `DEFAULT_MAX_INPUT_BYTES`, `DEFAULT_MAX_QUESTIONS`, and `DEFAULT_MAX_REQUESTS` hold the shared defaults. +`prepareEvaluationRequest(value, { maxInputBytes, limits })` is the one admission rule, used by the tool, the playground, and `evaluate`. It normalizes the near-miss aliases a model produces (`options` / `levels` / `choices` for `criteria`, a string Noul criterion, a label array for a Choice), validates the schema and JSON-safety, checks `limits` when given, then enforces the byte budget. Two shapes it refuses by name: a `box` question — 0.7.0's own question type for locating a subject in the request's image, while this package sends only `noul`, `choice`, and `score` — and a `null` state, which 0.7.0 no longer accepts. Both raise a `validation` error that names the field. `DEFAULT_MAX_INPUT_BYTES`, `DEFAULT_MAX_QUESTIONS`, and `DEFAULT_MAX_REQUESTS` hold the shared defaults. ## Limits @@ -109,7 +109,7 @@ Day caps live in `~/.pi/agent/pi-typesafe/usage.json` (owner-only, atomic, best- | `maxInputTokensPerDay` | `PI_TYPESAFE_MAX_INPUT_TOKENS_PER_DAY` | input tokens | | `maxUsdPerDay` | `PI_TYPESAFE_MAX_USD_PER_DAY` | estimated spend | -The environment may lower an explicit cap, never raise it. A reached cap raises a `budget` error that names the cap, the amount used, and the day, before anything is submitted. Cost is estimated from input tokens only, because output is free. Each request is priced when it is recorded, at its own model's price, so one ledger can hold requests to backends and models with different prices and the day's spend, and the `maxUsdPerDay` cap, add up what each request cost; requests to a free model add nothing. A ledger file written before costs were recorded keeps working: days without a recorded cost are estimated at the ledger's `usdPerMTok` (default `0.042`). +The environment may lower an explicit cap, never raise it. A reached cap raises a `budget` error that names the cap, the amount used, and the day, before anything is submitted. Cost is estimated from input tokens only, because output is free. Each request is priced when it is recorded, at its own model's price, so one ledger can hold requests to backends and models with different prices and the day's spend, and the `maxUsdPerDay` cap, add up what each request cost; requests to a free model add nothing. A token count the API did not report (0.7.0 reports `null`) adds nothing to the session or daily totals and is priced at zero, because there is nothing to count or price; the request still counts as an attempt against the request cap. A ledger file written before costs were recorded keeps working: days without a recorded cost are estimated at the ledger's `usdPerMTok` (default `0.042`). `openUsageLedger(options)`, `usagePath()`, `estimateUsd(tokens, usdPerMTok)`, `capsFromEnvironment(env)`, and `mergeCaps(explicit, environment)` expose the same arithmetic for your own display. diff --git a/package-lock.json b/package-lock.json index 4b816e4..580f42c 100644 --- a/package-lock.json +++ b/package-lock.json @@ -9,7 +9,7 @@ "version": "0.12.0", "license": "MIT", "dependencies": { - "@typesafe-ai/sdk": "^0.6.0" + "@typesafe-ai/sdk": "^0.7.0" }, "devDependencies": { "@earendil-works/pi-coding-agent": "0.86.1", @@ -44,7 +44,6 @@ "resolved": "https://registry.npmjs.org/@earendil-works/pi-coding-agent/-/pi-coding-agent-0.86.1.tgz", "integrity": "sha512-vZBuNfJnruxZyemZ3O05V0S/Ylze08ahFTIQ1Mik++gVdOevPl89gt/Uv0U97BPAJaj9cj6Vf9rcIgKtUrd0BA==", "dev": true, - "hasShrinkwrap": true, "license": "MIT", "dependencies": { "@earendil-works/chord": "^0.86.1", @@ -2551,12 +2550,12 @@ } }, "node_modules/@typesafe-ai/sdk": { - "version": "0.6.0", - "resolved": "https://registry.npmjs.org/@typesafe-ai/sdk/-/sdk-0.6.0.tgz", - "integrity": "sha512-IddX+Q0XM+VagOUZFeP7wZjaO4SHMdvnh2zEBdrZZnXedWI3BNK1lKhMx3ayrkFWvVLbVcUHJy6AVZlY+e6Jaw==", + "version": "0.7.0", + "resolved": "https://registry.npmjs.org/@typesafe-ai/sdk/-/sdk-0.7.0.tgz", + "integrity": "sha512-Yg3OZArCQMJx98ut8QVkhx3rFo9ON7pz8Hjzv8c3FGz69Sk8jYftLOlz+Wf6kq6jol/1m08aBnjRAzJr2x3GoQ==", "license": "MIT", "engines": { - "node": ">=20" + "node": ">=22" } }, "node_modules/@typescript/typescript-aix-ppc64": { diff --git a/package.json b/package.json index 33ccc2c..537bba8 100644 --- a/package.json +++ b/package.json @@ -64,7 +64,7 @@ "prepack": "npm run build" }, "dependencies": { - "@typesafe-ai/sdk": "^0.6.0" + "@typesafe-ai/sdk": "^0.7.0" }, "peerDependencies": { "@earendil-works/pi-coding-agent": ">=0.85.1 <1", diff --git a/src/batch.ts b/src/batch.ts index 83717f0..141aa23 100644 --- a/src/batch.ts +++ b/src/batch.ts @@ -75,7 +75,7 @@ export interface BatchEvaluation { readonly answers: Partial["answers"]>; /** The model of the first successful request, when there is one. */ readonly model?: string; - /** Usage summed over the requests that succeeded. */ + /** Usage summed over the requests that succeeded; a token count the API did not report adds nothing. */ readonly usage: Usage; /** Wall-clock time for the whole batch. */ readonly elapsedMs: number; @@ -95,8 +95,8 @@ function summarize(results: readonly Settled> } succeeded++; Object.assign(answers, result.value.answers); - inputTokens += result.value.usage.input_tokens; - outputTokens += result.value.usage.output_tokens; + inputTokens += result.value.usage.input_tokens ?? 0; + outputTokens += result.value.usage.output_tokens ?? 0; model ??= result.value.model; } return { diff --git a/src/client.ts b/src/client.ts index ca1737a..e3a00ac 100644 --- a/src/client.ts +++ b/src/client.ts @@ -151,8 +151,9 @@ function positiveNumber(value: number, label: string): number { function validResult(result: SystemOneResult, questions: Q): boolean { const probability = (value: number) => Number.isFinite(value) && value >= 0 && value <= 1; if (!result || typeof result.model !== "string" || !result.model || !result.usage || !result.answers) return false; + // 0.7.0 reports a token count it did not measure as null; that is not malformed, it is unreported. for (const count of [result.usage.input_tokens, result.usage.output_tokens]) { - if (!Number.isSafeInteger(count) || count < 0) return false; + if (count !== null && (!Number.isSafeInteger(count) || count < 0)) return false; } if (Object.keys(result.answers).length !== Object.keys(questions).length) return false; for (const [id, question] of Object.entries(questions)) { @@ -162,13 +163,17 @@ function validResult(result: SystemOneResult, questions: if (!probability(answer.noul)) return false; continue; } + // A box answer answers a question this package refuses to send, so it never validates. + if (answer.type === "box") return false; if (!probability(answer.confidence) || !answer.probabilities) return false; + // Rubric length, not its elements: 0.7.0's content type is recursive and mapping it exhausts the compiler. const keys = question.type === "choice" ? Object.keys(question.criteria) - : question.type === "score" ? question.criteria.map((_, index) => String(index)) : []; + : question.type === "score" ? Array.from({ length: question.criteria.length }, (_, index) => String(index)) : []; const probabilities = new Map(Object.entries(answer.probabilities)); if (probabilities.size !== keys.length || keys.some(key => !probability(probabilities.get(key) ?? NaN))) return false; if (answer.type === "choice" && !keys.includes(answer.choice)) return false; - if (answer.type === "score" && (!Number.isFinite(answer.score) || answer.score < 0 || answer.score > keys.length - 1 || !answer.legend)) return false; + // Presence of own `legend`, not its value: 0.7.0's legend type carries the recursive content type, which exhausts the compiler. + if (answer.type === "score" && (!Number.isFinite(answer.score) || answer.score < 0 || answer.score > keys.length - 1 || !Object.hasOwn(answer, "legend"))) return false; } return true; } @@ -233,6 +238,8 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { baseURL, timeout, retry: { maxRetries: 0 }, + // 0.7.0 zstd-compresses any body of 1 KiB or more by default; this package's five backends are unmeasured for it. + compression: false, logLevel: "off", ...(transport ? { fetch: transport } : {}), }); @@ -301,12 +308,15 @@ export function createTypeSafe(options: TypeSafeOptions = {}): TypeSafe { const result = await client.systemOne(request, callOptions); if (!validResult(result, request.questions)) throw new TypeSafeIntegrationError("response", "TypeSafe returned an unexpected answer or usage format."); usage.requestsSucceeded += 1; - usage.inputTokens += result.usage.input_tokens; - usage.outputTokens += result.usage.output_tokens; + // An unreported count is not a count this package can add, price, or charge against a token cap. + const inputTokens = result.usage.input_tokens ?? 0; + const outputTokens = result.usage.output_tokens ?? 0; + usage.inputTokens += inputTokens; + usage.outputTokens += outputTokens; // Unrounded: a small request must still count toward the cap. - const cost = (result.usage.input_tokens * rateFor(sentModel)) / 1e6; + const cost = (inputTokens * rateFor(sentModel)) / 1e6; sessionUsd += cost; - ledger.recordSuccess(result.usage.input_tokens, result.usage.output_tokens, cost); + ledger.recordSuccess(inputTokens, outputTokens, cost); if (!verificationRecorded) { verificationRecorded = true; recordAuthVerified(); diff --git a/src/extension.ts b/src/extension.ts index a9b3566..935098d 100644 --- a/src/extension.ts +++ b/src/extension.ts @@ -44,16 +44,20 @@ const sample = { }, }; +/** 0.7.0 reports a count it did not measure as null; name it instead of printing "null". */ +const formatNumber = (count: number | null): string => (count === null ? "not reported" : String(count)); + function format(result: Evaluation, expanded = false): string { const lines = [`TypeSafe · ${JSON.stringify(result.model)} · ${result.elapsedMs} ms`]; for (const [id, answer] of Object.entries(result.answers)) { const label = JSON.stringify(id); if (answer.type === "noul") lines.push(`${label}: P(yes) = ${answer.noul.toFixed(3)}`); else if (answer.type === "choice") lines.push(`${label}: ${JSON.stringify(answer.choice)} · confidence ${answer.confidence.toFixed(3)}`); - else lines.push(`${label}: ${answer.score.toFixed(3)} · confidence ${answer.confidence.toFixed(3)}`); - if (expanded && answer.type !== "noul") lines.push(` ${JSON.stringify(answer.probabilities)}`); + else if (answer.type === "score") lines.push(`${label}: ${answer.score.toFixed(3)} · confidence ${answer.confidence.toFixed(3)}`); + else lines.push(`${label}: box ${JSON.stringify(answer.box)} · confidence ${formatNumber(answer.confidence)}`); + if (expanded && (answer.type === "choice" || answer.type === "score")) lines.push(` ${JSON.stringify(answer.probabilities)}`); } - lines.push(`${result.usage.input_tokens} input / ${result.usage.output_tokens} output tokens`); + lines.push(`${formatNumber(result.usage.input_tokens)} input / ${formatNumber(result.usage.output_tokens)} output tokens`); lines.push("Confidence is distribution concentration, not proof of correctness."); return lines.join("\n"); } diff --git a/src/schema.ts b/src/schema.ts index 6eb3086..ae73fd1 100644 --- a/src/schema.ts +++ b/src/schema.ts @@ -26,12 +26,10 @@ export const DEFAULT_BACKEND_LIMITS: BackendLimits = Object.freeze({ maxQuestion // The API accepts structured descriptions, not only strings. The schema is the only shape guidance the model gets // before its first call, so every field the agent authors says what it means. -const entry = (description?: string) => Type.Union([ - Type.String(), - Type.Null(), - Type.Array(Type.Unknown()), - Type.Record(Type.String(), Type.Unknown()), -], description === undefined ? {} : { description }); +const content = () => [Type.String(), Type.Array(Type.Unknown()), Type.Record(Type.String(), Type.Unknown())]; +const entry = (description?: string) => Type.Union([...content(), Type.Null()], description === undefined ? {} : { description }); +/** `state` alone is non-null: @typesafe-ai/sdk 0.7.0 types it as `NonNullable`. */ +const state = Type.Union(content(), { description: "What to judge: text, or an object whose fields the questions name. Not null." }); const instructions = Type.Optional(entry("One judgment about the whole state, phrased as a question or a statement.")); const question = Type.Union([ Type.Object({ @@ -60,7 +58,7 @@ const question = Type.Union([ /** The JSON schema used by both the Pi tool and the programmatic interface. */ export const evaluationSchema = Type.Object({ - state: entry("What to judge: text, or an object whose fields the questions name."), + state, questions: Type.Record(Type.String({ minLength: 1, maxLength: 100 }), question, { minProperties: 1, maxProperties: DEFAULT_MAX_QUESTIONS, @@ -71,6 +69,29 @@ export const evaluationSchema = Type.Object({ const usage = `Expected { state, questions: { : { type: "choice", instructions, criteria: { label: description|null } } | { type: "score", instructions, criteria: [level0, level1, ...] } | { type: "noul", instructions } } }; 1–${DEFAULT_MAX_QUESTIONS} questions, Choice 1–64 options, Score 2–32 levels.`; +/** Question types this package sends. The SDK also offers `box`, which locates a subject in the request's image. */ +const SUPPORTED_QUESTION_TYPES = new Set(["noul", "choice", "score"]); + +/** + * Name the two refusals the schema can otherwise only report as a union mismatch: a `box` question, and a `null` + * state. Both are shapes @typesafe-ai/sdk 0.7.0 added or tightened; neither is something this package can send. + */ +function refuseUnsupported(value: unknown): void { + if (!value || typeof value !== "object" || Array.isArray(value)) return; + const request = value as Record; + if (request.state === null) { + throw new TypeSafeIntegrationError("validation", `state must not be null: send what to judge as text, an object, or an array. ${usage}`); + } + const questions = request.questions; + if (!questions || typeof questions !== "object" || Array.isArray(questions)) return; + for (const [id, question] of Object.entries(questions as Record)) { + const type = question && typeof question === "object" && !Array.isArray(question) ? (question as Record).type : undefined; + if (typeof type === "string" && !SUPPORTED_QUESTION_TYPES.has(type)) { + throw new TypeSafeIntegrationError("validation", `Invalid evaluation request at questions.${id.slice(0, 100)}.type: ${JSON.stringify(type.slice(0, 40))} is not a question type this package sends. ${usage}`); + } + } +} + /** Paths and messages only; never the submitted values. */ function describeSchemaErrors(value: unknown): string { const details: string[] = []; @@ -87,6 +108,7 @@ export function parseEvaluationRequest(value: unknown): SystemOneRequest { if (value !== null && typeof value === "object" && !Array.isArray(value) && !isJsonSafe(value)) { throw new TypeSafeIntegrationError("validation", `Invalid evaluation request: state and questions must be plain JSON. ${usage}`); } + refuseUnsupported(value); if (!Check(evaluationSchema, value)) { throw new TypeSafeIntegrationError("validation", `Invalid evaluation request at ${describeSchemaErrors(value)}. ${usage}`); } diff --git a/tests/backends.test.ts b/tests/backends.test.ts index 433169f..28a0895 100644 --- a/tests/backends.test.ts +++ b/tests/backends.test.ts @@ -19,6 +19,7 @@ function responseFor(questions: Questions): Response { const keys = Object.keys(q.criteria); return [id, { type: "choice", choice: keys[0], confidence: 1, probabilities: Object.fromEntries(keys.map((key, i) => [key, i === 0 ? 1 : 0])) }]; } + if (q.type !== "score") throw new Error(`the test reply has no answer for a ${q.type} question`); return [id, { type: "score", score: 0, confidence: 1, probabilities: Object.fromEntries(q.criteria.map((_, i) => [i, i === 0 ? 1 : 0])), legend: Object.fromEntries(q.criteria.map((level, i) => [i, level])) }]; })); return Response.json({ model: "jev-test", answers, usage: { input_tokens: 42, output_tokens: 0 } }); diff --git a/tests/batch.test.ts b/tests/batch.test.ts index d36793e..29e277e 100644 --- a/tests/batch.test.ts +++ b/tests/batch.test.ts @@ -172,3 +172,15 @@ test("a daily cap stops the batch with the cap named", async () => { assert.ok(second.error instanceof TypeSafeIntegrationError && second.error.code === "budget"); assert.ok(second.error.message.includes("daily request cap")); }); + +test("an unreported token count adds nothing to the batch usage", async () => { + const questions = manyQuestions(2); + const client = createTypeSafe({ + ledger: ledgerFor("null-usage"), + fetch: async () => Response.json({ model: "jev-test", answers: Object.fromEntries(Object.keys(questions).map(id => [id, { type: "noul", noul: 0.5 }])), usage: { input_tokens: null, output_tokens: null } }), + }); + const batch = await client.evaluateAll({ state: "many", questions }); + assert.equal(batch.ok, true); + assert.deepEqual(batch.usage, { input_tokens: 0, output_tokens: 0 }); + assert.equal(client.getSpend().today.inputTokens, 0); +}); diff --git a/tests/client.test.ts b/tests/client.test.ts index f752776..141f7fa 100644 --- a/tests/client.test.ts +++ b/tests/client.test.ts @@ -3,7 +3,7 @@ import { mkdtempSync } from "node:fs"; import { tmpdir } from "node:os"; import { join } from "node:path"; import { after, before, test } from "node:test"; -import { createTypeSafe, choice, noul, score, normalizeEvaluationRequest, parseEvaluationRequest, authState, clearAuthState, TypeSafeIntegrationError } from "../src/index.js"; +import { createTypeSafe, choice, noul, score, normalizeEvaluationRequest, parseEvaluationRequest, authState, clearAuthState, openUsageLedger, TypeSafeIntegrationError } from "../src/index.js"; import { safeError } from "../src/errors.js"; import { APIError } from "@typesafe-ai/sdk"; import type { Questions, SystemOneRequest } from "../src/index.js"; @@ -15,6 +15,7 @@ export function responseFor(questions: Questions): Response { const keys = Object.keys(q.criteria); return [id, { type: "choice", choice: keys[0], confidence: 1, probabilities: Object.fromEntries(keys.map((key, i) => [key, i === 0 ? 1 : 0])) }]; } + if (q.type !== "score") throw new Error(`the test reply has no answer for a ${q.type} question`); return [id, { type: "score", score: 0, confidence: 1, probabilities: Object.fromEntries(q.criteria.map((_, i) => [i, i === 0 ? 1 : 0])), legend: Object.fromEntries(q.criteria.map((level, i) => [i, level])) }]; })); return Response.json({ model: "jev-test", answers, usage: { input_tokens: 42, output_tokens: 0 } }); @@ -66,12 +67,40 @@ test("official SDK helpers, typed answers, metadata, and one batched network cal assert.equal(spend.blocked, undefined); }); -test("structured state, descriptions, and null values are supported", () => { - assert.doesNotThrow(() => parseEvaluationRequest({ state: null, questions: { +test("an unreported token count is counted as nothing, not priced, and not printed as garbage", async () => { + const ledger = openUsageLedger({ path: join(mkdtempSync(join(tmpdir(), "pi-typesafe-null-usage-")), "usage.json") }); + const client = createTypeSafe({ apiKey: "test-key", ledger, fetch: async () => + Response.json({ model: "jev-test", answers: { yes: { type: "noul", noul: 0.9 } }, usage: { input_tokens: null, output_tokens: null } }) }); + const result = await client.evaluate(sample()); + assert.equal(result.answers.yes.noul, 0.9); + assert.equal(result.usage.input_tokens, null, "the caller still sees what the API reported"); + assert.deepEqual(client.getUsage(), { requestsStarted: 1, requestsSucceeded: 1, requestsFailed: 0, inputTokens: 0, outputTokens: 0, estimatedUsd: 0 }); + const spend = client.getSpend(); + assert.equal(spend.today.inputTokens, 0); + assert.equal(spend.today.outputTokens, 0); + assert.equal(spend.today.estimatedUsd, 0); + assert.equal(spend.blocked, undefined); + assert.equal(/NaN|null|undefined/.test(ledger.describe({})), false, ledger.describe({})); +}); + +test("a box question is refused by name before any request leaves the process", async () => { + let calls = 0; + const client = createTypeSafe({ apiKey: "test-key", fetch: async () => { calls++; return Response.json({}); } }); + await assert.rejects(client.evaluate({ state: "synthetic", questions: { target: { type: "box", instructions: "Where is the logo?" } } }), (error: unknown) => + error instanceof TypeSafeIntegrationError && error.code === "validation" && error.message.includes('questions.target.type: "box" is not a question type this package sends')); + assert.equal(calls, 0); + assert.equal(client.getUsage().requestsStarted, 0); +}); + +test("structured state, descriptions, and null values are supported, but a null state is refused", () => { + assert.doesNotThrow(() => parseEvaluationRequest({ state: { goal: "Check" }, questions: { yes: noul({ goal: "Check" }, { true: ["a"], false: null }), pick: choice(["Choose"], { a: { detail: "a" }, b: null }), rating: score(null, [null, { level: "high" }]), } })); + // 0.7.0 types state as non-null, so a null state is refused by name instead of sent. + assert.throws(() => parseEvaluationRequest({ state: null, questions: { yes: noul("Is this synthetic?") } }), (error: unknown) => + error instanceof TypeSafeIntegrationError && error.code === "validation" && error.message.startsWith("state must not be null")); }); test("invalid questions are rejected before network submission without echoing state", async () => { diff --git a/tests/extension.test.ts b/tests/extension.test.ts index 55c91ab..c198e37 100644 --- a/tests/extension.test.ts +++ b/tests/extension.test.ts @@ -561,3 +561,17 @@ test("an invalid backend name declines login and logout without touching the sto clearStoredApiKey(); } }); + +test("an unreported token count is named in the result, not printed as null", async () => { + await withFetch(async () => Response.json({ model: "jev-test", answers: { yes: { type: "noul", noul: 0.9 } }, usage: { input_tokens: null, output_tokens: null } }), async () => { + confirmResult = true; + await runCommand("enable"); + const result = await runTool(); + const renderer = tool.definition.renderResult; + assert.ok(renderer); + const component = Reflect.apply(renderer, tool.definition, [result, { expanded: true, isPartial: false }, {}]); + const text = (component.render(80) as string[]).join("\n"); + assert.ok(text.includes("not reported input / not reported output tokens"), text); + assert.equal(text.includes("null"), false, text); + }); +}); diff --git a/tests/schema.test.ts b/tests/schema.test.ts index fa64c26..bbe6e7a 100644 --- a/tests/schema.test.ts +++ b/tests/schema.test.ts @@ -83,3 +83,12 @@ test("every field the agent authors carries a description, so a bare union is no // `true` and `false` inside noul criteria are the only fields whose parent already explains them. assert.deepEqual(undescribed.filter(name => name !== "true" && name !== "false"), [], "every authored field must say what it means"); }); + +test("a box question is refused by name, and a null state is refused, before the schema reports a bare union", () => { + const names = (pattern: RegExp) => (error: unknown) => error instanceof TypeSafeIntegrationError && error.code === "validation" && pattern.test(error.message); + assert.throws(() => prepareEvaluationRequest({ state: "s", questions: { target: { type: "box", instructions: "Where is the logo?" } } }), + names(/questions\.target\.type: "box" is not a question type this package sends/)); + // 0.7.0 requires a state that is not null, so the request is refused instead of sent with a null state. + assert.throws(() => prepareEvaluationRequest({ state: null, questions: { yes: { type: "noul", instructions: "?" } } }), names(/^state must not be null/)); + assert.doesNotThrow(() => prepareEvaluationRequest({ state: "s", questions: { yes: { type: "noul", instructions: "?" } } })); +});