Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,9 @@ TYPESAFE_API_KEY=
# Optional, explicit consent for automated/headless Pi tool calls:
# PI_TYPESAFE_ENABLED=1
# Optional, route judgments through another backend that serves the same protocol.
# One of: typesafe (default), openrouter, commandcode, liquid, subq. The key then comes
# from that backend's own variable (OPENROUTER_API_KEY, COMMANDCODE_API_KEY, LIQUID_API_KEY,
# SUBQ_API_KEY), not TYPESAFE_API_KEY, and /typesafe login does not apply.
# One of: typesafe (default), openrouter, commandcode, liquid, subq. The key comes from that
# backend's own variable (OPENROUTER_API_KEY, COMMANDCODE_API_KEY, LIQUID_API_KEY,
# SUBQ_API_KEY), or from /typesafe login <backend>; a set variable wins over a stored key.
# PI_TYPESAFE_BACKEND=openrouter
# Optional, override the backend's default model. Mapped to the backend's id form, so
# jev-1.14 becomes typesafe/jev-1.14 on OpenRouter and jev-latest becomes ~typesafe/jev-latest.
Expand Down
26 changes: 26 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,32 @@

## Unreleased

## 0.12.0

### Added

- The login store keeps one key per judgment backend: a registry name, or an endpoint's `keyEnv`. `keySituation`, `authState`, `ensureApiKey`, and `createTypeSafe` read the key stored for the backend in play, so a key stored for one backend is never sent to another. A backend's own environment variable still wins over its stored key.
- `/typesafe login [backend]` prompts with that backend's name, proves the key before saving, and stores it under that backend. TypeSafe and Liquid AI are proved with their model list, which checks the key; SubQ, OpenRouter, and Command Code are proved with one judge request, because their public model list accepts any key. A failed proof stores nothing.
- `/typesafe logout [backend]` removes one backend's key and leaves the others. With no backend named, both commands act on the session's own backend (`PI_TYPESAFE_BACKEND`), and a logout of that backend turns judgments off for the session.

### Changed

- `/typesafe login` and `/typesafe logout` no longer decline on a backend other than TypeSafe. One rule now covers every backend: the stored key is used unless the backend's environment variable is set, and then the variable wins. On a backend whose variable is set, `login` says so instead of prompting, and `logout` removes the stored key while saying the variable still wins.
- The README and `.env.example` no longer say that `/typesafe login` does not apply to another backend.

### Removed

- `usesTypesafeKey` is no longer exported: every backend has a login store now, so no caller needs to ask. This is a minor release, per CONTRIBUTING.

### Fixed

- `ask`'s deadline fired only while some other handle kept the Node event loop alive. `AbortSignal.timeout`'s timer is unref'd, so a call whose only pending work was the deadline never settled and CI reported it as `Promise resolution is still pending but the event loop has already resolved` on Node 22.19.0. The deadline is now a ref'd timer cleared when the call settles, so the documented `timeoutMs` bounds the call on every supported Node version.

### Storage and downgrade

- The store file is now `{ "version": 1, "keys": { "<backend>": "<key>" } }`, written under a temporary name and renamed whole. A pre-0.12 file `{ "apiKey": "..." }` reads as the TypeSafe key and is rewritten in the new shape on the next save.
- A 0.11.0 release run against a 0.12.0 store reads no TypeSafe key (it sees `missing`, not corruption) and does not touch another backend's key. If that older release saves a key, it rewrites the file in the old shape, and the other backends' stored keys are dropped; those backends fall back to their environment variables.

## 0.11.0

### Added
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ pi install npm:pi-typesafe

Then, inside Pi:

1. `/typesafe login` and paste a key from [console.typesafe.ai](https://console.typesafe.ai). Input is hidden; the key is verified against the API and saved to `~/.pi/agent/pi-typesafe/auth.json` with owner-only permissions.
1. `/typesafe login` and paste a key from [console.typesafe.ai](https://console.typesafe.ai). Input is hidden; the key is verified against the API and saved to `~/.pi/agent/pi-typesafe/auth.json` with owner-only permissions. `/typesafe login <backend>` does the same for OpenRouter, Command Code, Liquid AI, SubQ, or an endpoint you name, keeping each key under its own name in the same store.
2. `/typesafe test` sends one built-in sample request and shows the answers.
3. `/typesafe enable` lets the agent call the tool for this session, after you confirm the data notice.

Expand Down Expand Up @@ -91,8 +91,8 @@ Need more than one request? `evaluateAll(request)` asks any number of questions

| Command | Effect |
| --- | --- |
| `/typesafe login` | Enter and verify an API key (hidden input), then store it |
| `/typesafe logout` | Delete the stored key and disable the tool |
| `/typesafe login [backend]` | Enter and verify a backend's API key (hidden input), then store it under that backend; no argument means the session's backend. Refuses while that backend's environment variable is set, because the variable wins |
| `/typesafe logout [backend]` | Delete that backend's stored key, and turn judgments off when it was the session's backend; no argument means the session's backend |
| `/typesafe setup` | Check which key is in use; starts login if none |
| `/typesafe status` | Opt-in state, key state, session and today's counters, cost estimate, any reached cap |
| `/typesafe enable` | Confirm the data notice and allow agent tool calls this session |
Expand Down Expand Up @@ -142,7 +142,7 @@ if (!answer.ok) return { skipped: answer.errorCode === "budget" }; // never thr

Your extension owns its own user consent and budget; `/typesafe enable` applies only to this package's tool. Check `authState()` rather than your own consent flag before you report that judgments are on.

Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. Command Code serves the same protocol: `createTypeSafe({ backend: "commandcode" })` sends them to `api.commandcode.ai` with the key from `COMMANDCODE_API_KEY` and the model `typesafe/jev`. So does Liquid AI: `createTypeSafe({ backend: "liquid" })` sends them to `api.liquid.ai` with the key from `LIQUID_API_KEY` and the model `d1:free`, which is free; pass `model: "d1"` for the paid model, priced at $0.040 per million input tokens. Liquid counts the state once per question, where TypeSafe counts it once per request, so a request with N questions costs about N times the state on Liquid. Speed, measured on 2026-10-03 with one request at a time: `d1:free` took a median of about 14 seconds and sometimes failed with HTTP 504 after about 46 seconds, while `d1` took about 0.5 seconds. With `d1:free`, raise `timeoutMs` above its 15-second default, or pass `model: "d1"`. SubQ too: `createTypeSafe({ backend: "subq" })` sends them to `api.subq.ai` with the key from `SUBQ_API_KEY` and the model `subq-snap-preview`, priced at $0.05 per million input tokens. SubQ takes at most 16 questions in one request, a Choice of at least 2 options, and a Score of at most 10 levels: `evaluate` refuses a larger request before it sends it, `evaluateAll` splits to 16, and `client.limits` tells you the limit without a request. Or pass a `BackendEndpoint` object as `backend` to name any host that serves the protocol: it brings its own key variable, never reads `TYPESAFE_API_KEY`, and `backendHost()` reports its destination host for your consent text. These backends have no login store, so `/typesafe login` does not apply to them. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool follow `PI_TYPESAFE_BACKEND` (`typesafe` by default) and name that backend's host, key variable, and model in their consent and status text — so the extension sends judgments where it says it sends them. A name the registry does not know is a `configuration` error at session start, not a silent fallback. `PI_TYPESAFE_MODEL` overrides the backend's default model, written as that backend's own id is written (`jev-1.14`; it is mapped to the backend's id form, so OpenRouter sends `typesafe/jev-1.14`). `/typesafe login` and `/typesafe logout` apply only to a backend that uses the TypeSafe key store and say so on any other. Every export — the client, `ask`, batching, the usage ledger, auth state, the `PI_TYPESAFE_BACKEND` and `PI_TYPESAFE_MODEL` readers, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md).
Judgments can also go through OpenRouter: `createTypeSafe({ backend: "openrouter" })` sends them to `openrouter.ai` with the key from `OPENROUTER_API_KEY`. Command Code serves the same protocol: `createTypeSafe({ backend: "commandcode" })` sends them to `api.commandcode.ai` with the key from `COMMANDCODE_API_KEY` and the model `typesafe/jev`. So does Liquid AI: `createTypeSafe({ backend: "liquid" })` sends them to `api.liquid.ai` with the key from `LIQUID_API_KEY` and the model `d1:free`, which is free; pass `model: "d1"` for the paid model, priced at $0.040 per million input tokens. Liquid counts the state once per question, where TypeSafe counts it once per request, so a request with N questions costs about N times the state on Liquid. Speed, measured on 2026-10-03 with one request at a time: `d1:free` took a median of about 14 seconds and sometimes failed with HTTP 504 after about 46 seconds, while `d1` took about 0.5 seconds. With `d1:free`, raise `timeoutMs` above its 15-second default, or pass `model: "d1"`. SubQ too: `createTypeSafe({ backend: "subq" })` sends them to `api.subq.ai` with the key from `SUBQ_API_KEY` and the model `subq-snap-preview`, priced at $0.05 per million input tokens. SubQ takes at most 16 questions in one request, a Choice of at least 2 options, and a Score of at most 10 levels: `evaluate` refuses a larger request before it sends it, `evaluateAll` splits to 16, and `client.limits` tells you the limit without a request. Or pass a `BackendEndpoint` object as `backend` to name any host that serves the protocol: it brings its own key variable, never reads `TYPESAFE_API_KEY`, and `backendHost()` reports its destination host for your consent text. Each backend's key can also be stored with `/typesafe login <backend>`: it prompts once, proves the key with the backend's model list where that list checks the key and otherwise with one judge request, and saves it under that backend's name, so a key stored for one backend is never sent to another. A backend's own environment variable still wins over its stored key. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey`, or the status you report describes the TypeSafe key while the requests use another one. The `/typesafe` commands and the `typesafe_evaluate` tool follow `PI_TYPESAFE_BACKEND` (`typesafe` by default) and name that backend's host, key variable, and model in their consent and status text — so the extension sends judgments where it says it sends them. A name the registry does not know is a `configuration` error at session start, not a silent fallback. `PI_TYPESAFE_MODEL` overrides the backend's default model, written as that backend's own id is written (`jev-1.14`; it is mapped to the backend's id form, so OpenRouter sends `typesafe/jev-1.14`). `/typesafe login [backend]` and `/typesafe logout [backend]` act on the backend they name, or on the session's own backend when they name none, and on a backend whose environment variable is set they say that the variable wins. Every export — the client, `ask`, batching, the usage ledger, auth state, the `PI_TYPESAFE_BACKEND` and `PI_TYPESAFE_MODEL` readers, and the `pi-typesafe/calibrate` and `pi-typesafe/ui` entry points — is in [docs/api.md](docs/api.md).

## Development

Expand Down
4 changes: 2 additions & 2 deletions docs/api.md
Original file line number Diff line number Diff line change
Expand Up @@ -63,7 +63,7 @@ SubQ (`subq`) serves the same protocol at `api.subq.ai` under the SDK's own `/v1

Anything that is neither a registry name nor such an object is refused with `backend must be a registry name or a backend object.`

The TypeSafe backend takes its key from `TYPESAFE_API_KEY`, then the `/typesafe login` store. Every other backend — registry or endpoint — reads only its own `keyEnv` variable: the store holds a TypeSafe key, and a login verifies against api.typesafe.ai, so neither applies elsewhere, and the TypeSafe key is only ever sent to the typesafe backend. `createTypeSafe` with no `apiKey` and no key in the backend's variable fails with `No API key. Set <keyEnv> in the environment.` before any request is built. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send.
Every backend takes its key from its own environment variable first, then from the `/typesafe login` store. The store keeps one key per backend: a registry key under the backend name, an endpoint key under its `keyEnv`. A key stored for one backend is never read for another, and the TypeSafe key is only ever sent to the typesafe backend. `createTypeSafe` with no `apiKey` and no key for the backend fails with `No API key. Run /typesafe login <backend> in Pi, or set <keyEnv> in the environment.` before any request is built. Pass the same `backend` to `authState`, `keySituation`, and `ensureApiKey` so what you report matches what you send.

`evaluate(request, { signal })` validates before sending and rejects with `TypeSafeIntegrationError`. `code` is one of `configuration`, `validation`, `budget`, `aborted`, `timeout`, `http`, `connection`, `response`; messages never contain upstream bodies, keys, or your submitted state, and no header value except a numeric `Retry-After` count in seconds (quoted by the 429 advice as `Retry after <n> seconds.`). The advice is backend-aware: a 401 names the backend's own key variable (`Check TYPESAFE_API_KEY.`, `Check OPENROUTER_API_KEY.`, `Check COMMANDCODE_API_KEY.`, or `Check <keyEnv>.` for an endpoint), and a 402 says `Check your account balance.` except on OpenRouter, which says `Insufficient credits. Add credits at https://openrouter.ai/credits.` `listModels()` verifies the key without counting toward `maxRequests`, except on a backend whose model list is public (`modelsVerifyKey: false`, or an endpoint that does not set `modelsVerifyKey: true`), which accepts any key and leaves the auth state unverified.

Expand Down Expand Up @@ -156,7 +156,7 @@ console.log(formatCalibration(calibrate("action guard", samplesOf(results).sampl

## Login helpers: `pi-typesafe/ui`

`ensureApiKey(ctx, { backend })`, `loginWithPrompt(ctx)`, and `promptForApiKey(ctx)` use the same hidden input as `/typesafe login`. `ensureApiKey(ctx)` returns the existing key source, or prompts, verifies, and stores a new TypeSafe key (`undefined` when the user cancels). For any other backend it returns the environment source or throws `configuration` naming the variable to set; it never opens the prompt, because the prompt verifies against api.typesafe.ai and writes the TypeSafe store. These need Pi's TUI, so call them only from extension command handlers.
`ensureApiKey(ctx, { backend })`, `loginWithPrompt(ctx, { backend })`, and `promptForApiKey(ctx, label)` use the same hidden input as `/typesafe login`. `ensureApiKey` returns the existing key source, or prompts, proves, and stores a key for that backend (`undefined` when the user cancels). The proof is the backend's model list when that list checks the key; otherwise it is one judge request, because a public list accepts any key. `loginWithPrompt` resolves to `{ path, models? }` — `models` is absent when one request proved the key. These need Pi's TUI, so call them only from extension command handlers.

## One agent tool

Expand Down
4 changes: 2 additions & 2 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"name": "pi-typesafe",
"version": "0.11.0",
"version": "0.12.0",
"description": "TypeSafe AI (Jev) decisions for Pi: batched Choice/Score/Noul evaluation tool, terminal playground, and a typed API other extensions build on.",
"type": "module",
"license": "MIT",
Expand Down
10 changes: 8 additions & 2 deletions src/ask.ts
Original file line number Diff line number Diff line change
Expand Up @@ -30,13 +30,19 @@ export type AskAnswer<Q extends Questions> =
* share `prepareEvaluationRequest`, so what one accepts the other accepts.
*/
export async function ask<Q extends Questions>(judge: Judge, request: SystemOneRequest<Q>, options: AskOptions = {}): Promise<AskAnswer<Q>> {
const timeout = AbortSignal.timeout(options.timeoutMs ?? DEFAULT_ASK_TIMEOUT_MS);
const signal = options.signal ? AbortSignal.any([options.signal, timeout]) : timeout;
// AbortSignal.timeout's timer is unref'd, so a deadline that is the only pending work never fires: node:test on
// Node 22.19 reported this promise as never settling (CI run 39945731438). A ref'd timer, cleared when the ask
// settles, makes the documented deadline real and leaves no handle behind.
const deadline = new AbortController();
const timer = setTimeout(() => deadline.abort(), options.timeoutMs ?? DEFAULT_ASK_TIMEOUT_MS);
const signal = options.signal ? AbortSignal.any([options.signal, deadline.signal]) : deadline.signal;
try {
const result = await judge.evaluate(request, { signal });
return { ok: true, answers: result.answers, model: result.model, usage: result.usage, elapsedMs: result.elapsedMs };
} catch (error) {
if (error instanceof TypeSafeIntegrationError) return { ok: false, error: error.message, errorCode: error.code };
return { ok: false, error: FALLBACK_MESSAGE };
} finally {
clearTimeout(timer);
}
}
9 changes: 5 additions & 4 deletions src/auth.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { chmodSync, mkdirSync, readFileSync, renameSync, rmSync, writeFileSync } from "node:fs";
import { join } from "node:path";
import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, resolveBackend, usesTypesafeKey } from "./backends.js";
import { DEFAULT_BACKEND, TYPESAFE_KEY_ENV, resolveBackend } from "./backends.js";
import type { BackendSpec } from "./backends.js";
import { credentialsPath, keySituation, keySourceLabel, piTypesafeDir } from "./credentials.js";
import type { KeySource } from "./credentials.js";
Expand Down Expand Up @@ -40,8 +40,9 @@ export interface AuthState {
/** Short human label for the key source: `TYPESAFE_API_KEY`, `OPENROUTER_API_KEY`, `/typesafe login`, `no key`, `unusable key`. */
readonly keyName: string;
/**
* True when the key in effect was accepted by the backend (login verifies a TypeSafe key; a successful request proves
* any key). The record is shared across backends: switching backends keeps the last outcome until the next request.
* True when the key in effect was accepted by the backend (a login proves the backend it was asked for; a successful
* request proves any key). The record is shared across backends: switching backends keeps the last outcome until the
* next request.
*/
readonly verified: boolean;
readonly verifiedAt?: string;
Expand Down Expand Up @@ -154,7 +155,7 @@ export function describeAuth(state: AuthState = authState()): AuthReport {
const label = `${config.label} key`;
const since = state.lastFailure ? ` Last failure: ${state.lastFailure.message}${state.lastFailure.at ? ` (${state.lastFailure.at})` : ""}` : "";
if (state.kind === "missing") {
const how = usesTypesafeKey(config) ? `a key is configured (/typesafe login or ${TYPESAFE_KEY_ENV})` : `${config.keyEnv} is set in the environment`;
const how = `a key is configured (/typesafe login${config.name && config.name !== DEFAULT_BACKEND ? ` ${config.name}` : ""} or ${config.keyEnv})`;
return { level: "error", text: `${label}: missing — every Jev judgment is skipped until ${how}.${since}` };
}
if (state.kind === "unusable") {
Expand Down
Loading
Loading