From 5faf0869efcdfe28f0843498a70609c9857eecd2 Mon Sep 17 00:00:00 2001 From: Universe Date: Wed, 14 Oct 2026 05:47:14 +0900 Subject: [PATCH 01/17] Ratify the AI provider contract and its plan The plan for gridaco/fx#69 and gridaco/fx#79: decisions R1 to R15, the amendments to the overview's decisions 1, 4 and 9, decision 15, and the milestones this branch follows. GG and the grida fx umbrella leave the overview's Later list, since both shipped. AGENT_READINESS gains AR-20, proposed. --- AGENT_READINESS.md | 1 + docs/wg/overview.md | 9 +- docs/wg/provider-contract.md | 343 +++++++++++++++++++++++++++++++++++ 3 files changed, 348 insertions(+), 5 deletions(-) create mode 100644 docs/wg/provider-contract.md diff --git a/AGENT_READINESS.md b/AGENT_READINESS.md index b52ed294..5644355e 100644 --- a/AGENT_READINESS.md +++ b/AGENT_READINESS.md @@ -351,6 +351,7 @@ promise through documentation alone. | AR-17 | "Tidy up the layout; place X below Y." | Ratified design: [spec/layout.md](spec/layout.md). The automatic grid, decks and the layout report with automatic cells are released (0.2.0); the adjacent `fx-layout-v1` file the scenario edits is not implemented; visual editing deferred. | Agent discovers exact scope/references, edits and validates layout, verifies the renderer loaded the intended revision, and reports the actual URL or actionable diagnostics. No builder execution, provider calls, execution-history changes or identity/resume changes; repeats, nested scopes and stale references have provider-free evidence. | | AR-18 | "Clean up last week's exploratory runs, but keep my named ones." | Implemented in source: `runs list` and `runs remove` (CLI, Python, JavaScript), and `cache prune` (CLI). | Agent lists, previews and then removes exactly the previewed runs; named, `--run` and external runs survive filters; an active run, the last pick holder and a non-run selection are refused with a code and nothing else changes; catalog and run index forget removed folders; pruning keeps every remaining run resumable and refuses a shared, busy or unreadable cache, removing nothing; provider-free CLI, conformance and SDK evidence. | | AR-19 | "Share this result so somebody else can inspect it without my local service." | Implemented in source: terminal, inactive run export through CLI, Python and JavaScript; shared static browser reader and canvas. | Select the exact run, choose disclosure, export to a new directory, and review the receipt, omissions and concrete content. Verified selected bytes, non-overwriting placement, source stability and explicit reader failures preserve the evidence; publication remains a separate authorized consumer operation. | +| AR-20 | "Use this new model", or "use this provider FX does not ship." | Proposed: ratified as the [AI provider contract](docs/wg/provider-contract.md), not yet implemented. Today a project route may name another model on a provider FX ships; the one adapter FX has for that capability and provider serves it, with the request it builds for FX's own routes. A provider FX does not ship can be neither keyed nor served. | A new model on GG, OpenRouter or fal runs from a project route alone (`extends`, a new model id, `provider_options`, a cap when it has no price); a provider that speaks the OpenAI chat wire runs from a declaration; any other provider runs from a declaration and a Python or TypeScript adapter. Plans price or cap every call, refusals name the missing piece, keys stay redacted, and every path has provider-free evidence. | For AR-19, [the export contract](spec/exports.md), [CLI conformance](conformance/portable-export/), exporter regression tests, diff --git a/docs/wg/overview.md b/docs/wg/overview.md index ee559b7d..b0ad621e 100644 --- a/docs/wg/overview.md +++ b/docs/wg/overview.md @@ -20,20 +20,21 @@ It distinguishes implemented behavior, ratified contracts and proposed capabilit | | Decision | Status | |---|---|---| -| 1 | **FX is standalone.** It owns its provider adapters, and users bring their own keys from the environment (`GG_API_KEY`, `OPENROUTER_API_KEY`, `FAL_KEY`, …). FX depends on no Grida package. | ratified | +| 1 | **FX is standalone.** It owns its provider adapters, and users bring their own keys from the environment (`GG_API_KEY`, `OPENROUTER_API_KEY`, `FAL_KEY`, …). FX depends on no Grida package.
• Amended 2026-10-14 by the [AI provider contract](provider-contract.md): FX owns the adapter contract and its own adapters, and users may add providers and adapters. It may pin Grida Models' public data, which it never needs in order to run. (This decision predates Models' first release, 2026-10-11.) | ratified; amended 2026-10-14 | | 2 | **The engine is written in Rust.** TypeScript users must not need Python, and Python users must not need Node. A native binary is the one runtime both can carry. | ratified | | 3 | **A protocol, not FFI.** There are no napi, PyO3 or WASM bindings.
• SDKs and node bodies talk to the engine through a node protocol: JSON-RPC 2.0 over stdio, framed as in LSP.
• An `initialize` handshake carries the protocol version, because a project's SDK and the installed engine will drift. | ratified | -| 4 | **The engine is the only place with engine logic.** Expressions, identity, prompt rendering, the agent loop and file facts all live in the binary. SDKs only build documents, author nodes and host node bodies, so every SDK produces the same cache keys for the same request. | proposal | +| 4 | **The engine is the only place with engine logic.** Expressions, identity, prompt rendering, the agent loop and file facts all live in the binary. SDKs only build documents, author nodes and host node bodies, so every SDK produces the same cache keys for the same request.
• Amended 2026-10-14 by the [AI provider contract](provider-contract.md): adapter code, FX's or a project's, maps requests and answers; the checks stay in the engine. | proposal; amendment ratified 2026-10-14 | | 5 | **Paid work needs a priced plan and a ceiling.** A workflow file can spend the user's money, so nothing paid runs without a priced plan and a `--max-usd` ceiling. A run whose every call is on the user's own plan (the Codex and Grok Build routes, which bill no dollars) is not paid work in this sense: it runs without one, under a ceiling of $0 that refuses any other route (proposed with gridaco/fx#46; awaiting ratification). | ratified | | 6 | **The workflow registry copies; it never references.**
• This repo later doubles as the default registry: `grida-fx add ` copies a workflow or node into the user's project (like shadcn registries or `npx skills`) and stamps where it came from.
• Nothing is fetched while a run is being planned. | ratified direction | | 7 | **The Grida umbrella comes last.** `grida fx` (in the `grida` CLI, with this repo as a submodule) hands off to the FX engine, and GG becomes a provider. | ratified | | 8 | **The standard library's node bodies are in Python** (`grida.fx.std`, with Pillow). A Rust image resize that differs from Pillow's by one byte changes every request downstream of it, so the bodies stay in Python until Rust bodies come, with a planned rekey. The engine still owns `select` and every fact. | in the approved plan | -| 9 | **No third-party OpenAI client.** Adapters are thin clients on FX's own injected transport. The canonical request then *is* the wire body, and every exchange can be replayed in tests. | in the approved plan | +| 9 | **No third-party OpenAI client.** Adapters are thin clients on FX's own injected transport. The canonical request then *is* the wire body, and every exchange can be replayed in tests.
• Amended 2026-10-14 by the [AI provider contract](provider-contract.md): this covers FX's own adapters, and a project's adapter reaches the network as it likes. The canonical request is what an adapter maps to the wire, not the wire body itself. | in the approved plan; amended 2026-10-14 | | 10 | **Document versions start at v1 in the `fx` namespace** (`fx: workflow/v1`, `fx-graph-v1`, …). Authored YAML documents carry `fx: /v1`; machine-written JSON carries `"kind": "fx--v1"`. | in the approved plan | | 11 | **Initial publication waited for trusted publishing; that bootstrap is complete.**
• The original decision required the owner to set up trusted publishing (OIDC) before publication. npm sets that up only on a package that already exists, so [RELEASING.md](../../RELEASING.md) records the first-publish bootstrap. Stable releases, from `0.1.0`, publish to both registries.
• For source development, `tools/build_engine.py` builds the engine into a checkout's Python SDK, and [examples/](../../examples/) prove that path in CI. | the owner's direction (2026-10-06) | | 12 | **No cache replay.** A cache is never carried across an identity change by replaying old runs against it. Records keep the canonical request, so a rekey can compute the new keys from them (decision 8). | the owner's decision (2026-10-06) | | 13 | **Run observation is a public contract.** FX will expose versioned, read-only execution events for independent consumers; the built-in viewer must use the same observation boundary. Recorded events remain authoritative. Ordered reads, cursor/replay semantics, compatibility and observer independence are specified before a follow interface is advertised as stable. [Run observation](../../spec/observation.md) records the wire interface, guarantees, lifecycle and verification. | v1 ratified and implemented (2026-10-08); released in 0.2.0 | | 14 | **Canvas layout is engine-computed presentation.** The viewer lays a workflow out as a left-to-right grid of cells and stacks repeated instances into decks. An optional `.layout.json` (`fx-layout-v1`) beside the workflow's takes file places declared steps by column and row. The engine computes the cells and the member order and serves them on the scoped layout route; the browser places its own projection's cards in them, so `inspect` (once it reads the layout file) and the browser agree. The file never enters planning, identities, caches or resume. [Canvas layout](../../spec/layout.md) holds the design. | ratified design (2026-10-09); the grid and decks released in 0.2.0; the layout file not implemented | +| 15 | **The AI provider contract.** Capabilities stay FX's and the engine checks them. One adapter contract serves FX's own adapters in Rust and a project's in Python or TypeScript. Providers are declared, a route gives a price or a per-call cap, and any model on a provider FX speaks runs from a route alone. [AI provider contract](provider-contract.md) holds the decisions and the plan. | ratified (2026-10-14) | The [local service contract](../../spec/service.md) is ratified and implemented (2026-10-08; released in 0.2.0): optional project initialization, a native foreground or @@ -102,8 +103,6 @@ are still Python (decision 8). ## Later - The workflow registry. -- GG as a provider. -- The `grida fx` umbrella. - Views: the [canvas layout](../../spec/layout.md) file (decision 14), [custom views and optional OS supervision](../../TODO.md#workflow-and-run-viewer). Already released: the [local workflow-plan and run viewer](../guide/07-viewing.md), the read-only node canvas with diff --git a/docs/wg/provider-contract.md b/docs/wg/provider-contract.md new file mode 100644 index 00000000..f47f8988 --- /dev/null +++ b/docs/wg/provider-contract.md @@ -0,0 +1,343 @@ +# AI provider contract + +Status: ratified by the owner, 2026-10-14. It is built in one pull request that closes +[gridaco/fx#69](https://github.com/gridaco/fx/issues/69) and covers +[gridaco/fx#79](https://github.com/gridaco/fx/issues/79) as far as Grida Models' schema allows. +Once it lands, [`spec/`](../../spec/) holds the contracts; this page keeps the decisions and +the record of the plan. + +## Why + +- New models and providers arrive faster than FX can ship adapters, so FX will always be + behind. A user must be able to run any model offered by a provider FX already speaks + (GG, OpenRouter, fal), and to add a provider FX does not know, without waiting for an FX + release. +- FX must stay useful on its own, with its engine and its contracts, if its maintainers are away + for weeks or months. +- FX is a general workflow engine with first-class AI support. The contract covers AI + capabilities only. It is not a generic meter for paid HTTP calls: a user who needs one writes + a node and meters it. +- The fix is the fundamental one: FX's own providers use the same contract that users build on. + +## Decisions + +Ratified 2026-10-14. + +| | Decision | +|---|---| +| R1 | **Capabilities stay FX's.** The engine checks every request against its capability and the route's contract before any adapter sees it, and runs the capability's every-route checks on every answer. An adapter maps requests and answers, and adds only its route's own judgements. | +| R2 | **One adapter contract, two bindings.** FX's own adapters stay in Rust and in process, so AI calls need neither Python nor Node. A project's adapters run out of process over the node protocol, in Python or TypeScript. The contract (outcomes, long jobs, costs, reasons, handles, redaction) is its own spec; each FX adapter's wire details move to a reference kept beside its code. | +| R3 | **Network access belongs to the adapter.** A project adapter reaches the network however it likes: its own HTTP client or a vendor's SDK. The contract asks only for what FX's promises need: one send per attempt with retries off, an honest outcome (received or not, billed or not), and credentials named in its declaration, so the engine can redact them. The SDKs offer an optional `fetch` through the engine's transport, which adds the never-sent classification, response caps and offline replay. FX's own adapters keep the transport. | +| R4 | **Adapters are project code, like node modules.** They live in the project or in a package under `sources:`, are pinned by `fx.lock`, and are identified by a version or a source digest. | +| R5 | **Providers are declared.** A declaration names the provider's credential variables and how they are sent, its base-URL variable and default, and the hosts its results may be downloaded from. FX's providers become built-in declarations. Keys still come only from the environment or the project's `.env`, under the existing rules, now for the declared names. FX never writes credentials ([gridaco/fx#45](https://github.com/gridaco/fx/issues/45)). A declared provider bills dollars; only FX's own runtimes bill a plan. | +| R6 | **An OpenAI-compatible family by declaration alone**, for the chat capabilities (`structured.generate`, `agent.turn`). | +| R7 | **Routes.** `extends` lets a route take another route's price, features, limits and contract and restate only what differs. Namespaced `provider_options`, like the Vercel AI SDK's `providerOptions`, pass parameters FX does not expose; only the matching adapter reads them, and they are part of identity. The engine always supplies the serving adapter's identity to the route fingerprint: no built-in key moves, and a project route declared without a contract recomputes once. | +| R8 | **A route gives a price or a per-call cap.** A capped route is unpriced: each call holds the cap and is charged what the provider reports, else the whole cap. A route with neither is refused, and an empty price no longer means $0. | +| R9 | **Any model on a provider FX speaks.** A new model id, with `extends`, `provider_options` and a cap where needed, is enough to run a model FX does not list on GG, OpenRouter and fal. | +| R10 | **Grida Models is used when FX is built, never when it runs.** FX pins a released snapshot of [Grida Models](https://github.com/gridaco/models), keeps a generated projection of the facts Models' JSON Schema defines, and CI checks the built-in route table against it. A route's effective features are the provider's documented facts that its adapter implements; an unknown fact never enables one. A project-level pin is a follow-up. | +| R11 | **GG's planning prices are FX's allowances, and say so.** GG's image and video routes keep allowances taken from the same model elsewhere until GG's own rates can be planned from. A GG call is booked at GG's settled charge. | +| R12 | **Request fields become provider-neutral later,** in the one identity break before 1.0, with the naming review, so users migrate once. `provider_options` lands now, since it only adds a member. | +| R13 | **One definition per capability is deferred.** A capability is consolidated when it next changes. | +| R14 | **No model aliases.** The capability default in `fx.yaml` is already a one-line bump. | +| R15 | **Nothing is spent.** Everything is proven over replay fixtures, stand-ins and synthetic adapters. | + +**Amendments to the [overview](overview.md)'s decisions:** + +| Decision | Amendment | +|---|---| +| 1 | FX owns the adapter contract and its own adapters; users may add providers and adapters. "FX depends on no Grida package" gains: "It may pin Grida Models' public data, which it never needs in order to run." The decision predates Models' first release (2026-10-11). | +| 4 | Adapter code maps requests and answers; the checks stay in the engine (R1). | +| 9 | Scoped to FX's own adapters. The canonical request is what an adapter maps to the wire, not the wire body itself. | + +## What users can extend + +| Job | Needs | +|---|---| +| A model FX does not list, on a provider it speaks | a route: `extends` and the new model id, with a cap when it has no price | +| A parameter FX does not expose | `provider_options` | +| A provider that speaks the OpenAI chat wire | a declaration | +| Any other provider | a declaration and an adapter in Python or TypeScript | + +## Working rules + +- **One draft PR,** from the branch `provider-contract`. The owner reviews and merges it. + Commits stay local and are pushed once, when the plan is done or the session must stop; the + draft PR opens then. +- **Commits follow the milestones in order.** Each commit builds and passes the checks for what + it touches, and each milestone ends with its gate green. A milestone ticks its boxes here in + its last commit. +- **Spec first.** A milestone that changes a contract changes `spec/` before the code. +- **Users' surfaces move together.** Every user-visible change updates `llms.txt` in the same + milestone, and public changes move `AGENT_READINESS.md` and both SDKs with them (AGENTS.md). +- **Never stop to ask.** A question this page does not answer takes the default under + [Defaults](#defaults); anything left out becomes a follow-up issue, and the work goes on. +- **Code facts win.** Where this page and the code disagree on a fact, the page is corrected in + the milestone that finds it. +- **No release.** The PR bumps no version; a release follows RELEASING.md separately. + +## Milestones + +### M0. Ratify + +- [x] This page. +- [x] The overview: amend decisions 1, 4 and 9; add decision 15, this contract, ratified + 2026-10-14; drop "GG as a provider" and "The `grida fx` umbrella" from Later, since both shipped. +- [x] `AGENT_READINESS.md`: scenario AR-20, "Use a model or provider FX does not ship", + Proposed. + +Gate: the docs' relative links resolve; `check_spec`. + +### M1. The adapter contract, split out (no behaviour change) + +- [ ] `spec/adapters.md`: the adapter contract, version 1, made from `providers.md`'s + introduction and §1, §2 and §4 to §8, unchanged in substance. +- [ ] `spec/providers.md`: providers, keys and endpoints; the built-in adapters as a normative + table (provider, capability, contract `adapter`, `adapter_behavior` as exact JSON, reference); + the built-in route table; renamed features. fal and Tripo get their behaviour named, and + `background.remove` on fal gets one. +- [ ] Each FX adapter's wire details move from §9 to + `crates/grida-fx-providers/src//README.md`, with its contract names and behaviour. +- [ ] Every reference follows: spec cross-references, the guide's anchors (`#96-codex`, + `#97-grida-gateway`, `#98-grok-build`, `#10-the-built-in-route-table`), `llms.txt`, Rust + comments, and capabilities.md's "the route's, in providers.md §9". +- [ ] The gates follow the text: `check_spec` (j), (k) and its forbidden-name roots; + `tests/routes.rs`'s sentence and contract checks; `bind.rs`'s renamed-features anchor; + `tests/capabilities.rs`. A new check: every "`` §N" reference names a heading + that exists. + +Gate: `check_spec`, `cargo test`, `check_llms`. + +### M2. The engine owns the checks (no behaviour change) + +- [ ] Spec: the engine checks the route's contract and the request's fit before dispatch, and + runs the every-route checks after the adapter's own (`adapters.md` §5, `capabilities.md` §1). +- [ ] Runtime: the contract check and `check_request` before dispatch, and + `checks::every_route` on every answer, for adapters and stand-ins alike. Adapters drop their + copies and keep their route-specific checks. +- [ ] Refusal sentences and their order are unchanged; every adapter test passes as written. + +Gate: `cargo test`, conformance. + +### M3. Route identity and `extends` + +- [ ] Spec (`identity.md` §7): where a route's contract leaves out `adapter` or + `adapter_behavior`, the engine supplies them from the built-in adapter table, for a provider + that has an adapter. Routes on providers without one are untouched. `extends` is resolved + before the fingerprint and is not part of it. +- [ ] `fx-routes-v1`: `extends`; `price` is not required beside it. +- [ ] Core: a static (capability, provider) → identity table, promoted from `tests/routes.rs`; + filling in `load_catalog`; resolving `extends`. +- [ ] A test pins all 29 built-in fingerprints before the change and proves that none moves. +- [ ] `tools/digest.py` fills and resolves exactly as the engine does; vectors for a filled + contract and an extended route. +- [ ] Drop `jpeg_output`, `webp_output` and `hosted_url_reference_input` from the vocabulary + (capabilities.md, `features.json`, `capabilities.rs`) and from the routes. +- [ ] `llms.txt`; guide pages 05 and 13. + +Gate: `cargo test`, conformance with no expected file moved, `check_spec`, `pytest`. + +### M4. Capped routes + +- [ ] Spec: `price: {cap_usd: N}` stands alone; `price: {}` is refused; the hold is the cap + (`protocol.md` §6.1); costs (`adapters.md`); decimals (`identity.md` §12). +- [ ] Schemas: `fx-routes-v1`; in `fx-graph-v1`, an instance's `price` gains an optional + `capped: true`. +- [ ] Core: the route price's flag, the refusals in `parse_price`, and a plan note per capped + route. +- [ ] Browser and SDKs: `web/fx-web` accepts `capped`; the viewer shows "up to $N"; the + TypeScript document types. +- [ ] Tests: the parser, the plan's text and graph, and settlement with and without a reported + cost. A conformance case plans a capped route. +- [ ] `llms.txt`; guide page 04. + +Gate: `cargo test`, conformance, the viewer's checks, the SDKs' tests. + +### M5. `provider_options` + +- [ ] Spec: every paid capability takes an optional object `provider_options`, keyed by + provider. An adapter reads only its route's provider's member; members FX sets itself are + refused; it never holds a credential, since requests are recorded verbatim. A route's + `contract.provider_options` is merged under the step's. +- [ ] Each paid built-in gets the optional param, with no default, so no key moves. +- [ ] Adapters merge their member into the body: OpenAI, OpenRouter, fal, GG, Tripo and + ElevenLabs. Codex and Grok Build refuse a non-empty one. +- [ ] Replay tests per adapter, for the pass-through and for the refusal of members FX sets. +- [ ] `examples/movie-sprite`: its prompt-expansion setting goes through `provider_options`; + README and pasted plan output updated. +- [ ] `llms.txt`; guide pages 03 and 13. + +Gate: `cargo test`, conformance, `check_examples`, `check_llms`. + +### M6. Any model on GG, OpenRouter and fal + +- [ ] The registry keys adapters by capability, provider and contract `adapter`. A route + without one takes its provider's default for the capability. +- [ ] OpenRouter images: a generic adapter, the default for routes FX does not ship, sends what + the request and `provider_options` say, with no forced quality, moderation or size list. An + answer that is not a PNG becomes one, as Grok Build's do. The built-in routes keep + `fx-openrouter-image-v1` unchanged. +- [ ] fal images: the same, with fal's common request and answer shapes taken from the fal + endpoint parameters that Models records. The GPT Image envelope stays with the GPT Image + adapter. +- [ ] fal video (`fal-queue`): durations, resolutions and aspect ratios come from the route (its + price units and tiers), not constants; the first frame is sent only when given. No byte it + sends today changes. +- [ ] GG images: the picture limit comes from the route; the built-in GG image routes declare + `max_input_pictures: 16`. +- [ ] A replay test per generic adapter with a made-up model; `doctor` and `plan` report the + route as servable. +- [ ] `llms.txt`; guide page 13. + +Gate: `cargo test`, conformance, `check_llms`. + +### M7. Grida Models facts and OpenRouter transparency + +- [ ] `tools/make_models.py --release `, run by hand with the network: + - downloads `models.v1.json` and verifies its SHA-256; + - validates it against that release's schema; + - keeps only the schema-defined members that FX's routes use; + - writes `crates/grida-fx-providers/routes/models.json` with a `source` block (repository, + release, asset, bytes, SHA-256, schema `$id`). + `--check` verifies it offline. The full release is never committed. +- [ ] Pinned to `v1.20261013.6`. +- [ ] `crates/grida-fx-providers/tests/models.rs` checks every route's public facts against the + projection: + - wire ids and endpoint ids; + - features from documented parameters, intersected with adapter support, both ways: a + documented and supported feature must be declared; + - picture limits and size envelopes; + - GG's 3D endpoints; + - text token prices as bounds, never equality. + + Each exception sits in one table with its reason, and an exception no longer needed fails. +- [ ] OpenRouter GPT Image: `alpha` on both routes; the adapter sends `background: + transparent`; replay tests for the body and the answer's check. `adapter_behavior` stays + `"3"`, since no body it sent before changes. +- [ ] `providers.md`: GG's prices are said to be FX's allowances; GG's image picture limit. +- [ ] Drift: `llms.txt` and guide page 05 on masks, since fal's edit route takes one. +- [ ] CI: `make_models.py --check` in the `spec` job; AGENTS.md's verification list. + +Gate: `cargo test`, `check_spec`, `make_models.py --check`, `check_llms`. + +### M8. Provider declarations + +- [ ] Spec (`providers.md`): a declaration names the provider (the route grammar, not a built-in + name), its credential (variable, header, scheme), its base URL (variable and default; https + unless loopback), its download hosts, and its family or adapter. Refused: a built-in + provider's name; a variable a built-in or another declaration claims; `GRIDA_FX_*`, proxy and + certificate variables. +- [ ] `fx-project-v1`: `providers:`. +- [ ] Built-in declarations replace the fixed key, variable, endpoint and provider lists. + - Credential headers and prefixes are owned values. + - The `.env` loader reads the declared names, and the redactor learns their values. + - The environment scrubs (Codex and Grok Build children, the service, the browser opener) use + the computed list. + - Codex and Grok Build stay runtimes. +- [ ] One download-host allowlist helper, used by fal and Tripo. +- [ ] `doctor` lists declared providers with their key state and base-URL host. +- [ ] The tools' and tests' key lists read the engine's list, or stay in step with it. +- [ ] A conformance case declares `acme`: plan, `doctor`, and the refused declarations. +- [ ] `llms.txt` (it says "only these eleven variables"); guide pages 05 and 13. + +Gate: the whole verification list. + +### M9. OpenAI-compatible providers + +- [ ] Spec: the `openai-compatible` family serves `structured.generate` and `agent.turn` over + `{base}/chat/completions`. + - Errors map through the default status table, never `Some(0)`. + - Cost is `None`, so the hold or the cap is charged. + - Contract adapters are `openai-compatible-structured` and `openai-compatible-tool-loop`, + behaviour `1`. +- [ ] Registered from declarations, on the shared chat module. +- [ ] A replay test: a made-up declared `acme` answers both capabilities. +- [ ] An `llms.txt` recipe: a declared provider, planned offline. + +Gate: `cargo test`, `check_llms`. + +### M10. Project adapters in Python and TypeScript + +- [ ] Spec: the out-of-process binding (`adapters.md`); in `protocol.md`, the methods + `adapter.load`, `adapter.send`, `adapter.submit` and `adapter.collect`, and the host-to-engine + `fetch`, added within protocol v1; their schemas. +- [ ] A declaration's `adapter: #`; `fx.lock` `adapters`. Identity: the contract + `adapter` is the declared adapter, and `adapter_behavior` is its version, or `source:`. +- [ ] Engine: + - an adapter host started lazily in a live run, outside the pools, and shut down with the run; + - an RPC adapter for send, submit and collect; + - a host lost mid-send is `Failed { retryable }`, mid-submit `Uncertain`, mid-collect + `Unreachable`; + - handles must be objects; + - cancellation at the invocation; + - dollars only; + - declared credentials passed into the host's environment. +- [ ] `fetch`: the engine's transport, with the credential attached by the engine, the network + switch, caps and deadlines. +- [ ] SDKs: an adapter authoring API in `grida.fx` and `@grida/fx`, with outcome types and + concurrent calls; tests; READMEs; `protocol.md` §8. +- [ ] Conformance: `adapter-py` and `adapter-ts`, each a synthetic `acme` adapter run live with + the network off: an answer, a refusal and a long job. +- [ ] `llms.txt`; guide pages 03 and 13. + +Gate: the whole verification list, `check_host` included. + +### M11. Docs, readiness and the PR + +- [ ] Guide page 13, "Use a model or provider FX does not ship"; caps in page 04; routes in + page 05. +- [ ] `llms.txt`: recipes for a new model and a declared provider; the errors table. +- [ ] `skills/grida-fx` and its onboarding reference. +- [ ] `AGENT_READINESS.md`: AR-20 Current, with its evidence. +- [ ] The follow-up issues filed, each linked from this page. +- [ ] The whole verification list; push; open the draft PR. + +## Defaults + +| Question | Default | +|---|---| +| Where a project declares providers | `providers:` in `fx.yaml` | +| A cap's form | `price: {cap_usd: N}`, alone in `price` | +| What `extends` may name | a route of the same capability and provider, from the built-in table (even under `--routes`) or an entry already read. It takes a snapshot and is never hashed | +| Route-level options | `contract.provider_options`, already in the fingerprint | +| Step-level options | an optional `provider_options` param on each paid built-in, without a version bump. If `identity.md` requires a bump, route-level options only, and a follow-up | +| The adapter of a route without a contract | its provider's default for the capability, from the built-in table (the generic one after M6) | +| The spec's layout | `spec/adapters.md` and `spec/providers.md`, renumbered, with every reference updated; references in `crates/grida-fx-providers/src//README.md` | +| `adapter_behavior` bumps | none where no byte an adapter sent before changes | +| A generic adapter's wire | only what published facts support (Models' endpoint parameters, the provider's own published schema). Otherwise ship the selection and `provider_options` only, and file a follow-up | +| An answer that is not a PNG | converted to PNG in the adapter, as Grok Build's are | +| A declared provider's billing | dollars only; no declaration can claim a plan | +| A declared provider's errors | the default status table; never `Some(0)` | +| A declared provider's cost | none read; the hold or the cap is charged | +| New protocol methods | added within protocol v1; hosts answer unknown methods with -32601 | +| Project adapters in conformance | run live with the network off; the synthetic adapter makes no request | +| The Models file | one generated projection; the full release is never committed | +| Prices against Models | bounds, never equality | +| A test that resists | fix the code, not the test, unless the test pins wording this plan changes | + +## Follow-ups + +Filed as issues in M11, or as soon as a milestone leaves something out, with only verified facts. + +**gridaco/fx:** +- a project-level Models pin and model import (#69's P5); +- the media routes' facts from Models, once its media prices are in its schema; +- the identity break before 1.0 (R12); +- OpenAI-compatible images; +- a generic adapter for OpenRouter music; +- pacing shared across a provider's routes: an extended GG image route gets its own slot; +- ElevenLabs speech's allowance for long texts: its high is $0.05 a call, and its answers + report no cost; +- a live check that OpenRouter's transparent answers carry alpha, which needs the owner's go + and a cap; +- Tripo's face limit: 25,000 in FX, 50,000 in Models. + +**gridaco/models**, gaps found against `v1.20261013.6`: +- no card for Gemini Omni Flash; +- no endpoint facts on the OpenAI binding; +- no mask facts; +- GG's meters have no quantity per request; +- media prices sit outside the schema, and `token_pricing` cannot express image models; +- no ElevenLabs binding or US-dollar rate; +- `input_references` is typed as an integer on OpenRouter; +- no `size` on OpenRouter; +- `image_constraints` only on fal. From 8242da7840ae14e0246771f7ecb2a9f3d2c07fbe Mon Sep 17 00:00:00 2001 From: Universe Date: Wed, 14 Oct 2026 06:55:01 +0900 Subject: [PATCH 02/17] Split the adapter contract out of the providers spec MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit spec/adapters.md is the adapter contract, version 1: where adapters run, the transport, outcomes, refusals before sending, costs, long jobs, downloads and reasons, moved from spec/providers.md unchanged in substance. spec/providers.md keeps keys and endpoints, gains the built-in adapters as a normative table with each contract's adapter and behaviour in exact JSON, and keeps the route table and renamed features. Each adapter's wire details move beside its code, in crates/grida-fx-providers/src//README.md. Every reference follows, and check_spec gains (l): every ' §N' reference names a heading that exists. No behaviour changes. --- Cargo.toml | 2 +- ISSUES.md | 10 +- TODO.md | 15 +- conformance/README.md | 2 +- conformance/run.py | 2 +- crates/grida-fx-core/src/expand/bind.rs | 12 +- .../grida-fx-core/tests/expand_semantics.rs | 2 +- crates/grida-fx-providers/Cargo.toml | 2 +- crates/grida-fx-providers/routes/default.yaml | 21 +- crates/grida-fx-providers/src/acp/mod.rs | 4 +- crates/grida-fx-providers/src/adapter.rs | 17 +- crates/grida-fx-providers/src/capabilities.rs | 4 +- crates/grida-fx-providers/src/chat/mod.rs | 34 +- crates/grida-fx-providers/src/chat/schema.rs | 2 +- crates/grida-fx-providers/src/checks.rs | 4 +- crates/grida-fx-providers/src/clock.rs | 2 +- crates/grida-fx-providers/src/codex/README.md | 52 ++ crates/grida-fx-providers/src/codex/mod.rs | 2 +- .../grida-fx-providers/src/codex/protocol.rs | 6 +- .../grida-fx-providers/src/codex/runtime.rs | 6 +- .../src/elevenlabs/README.md | 27 + .../grida-fx-providers/src/elevenlabs/mod.rs | 22 +- .../src/elevenlabs/sound.rs | 8 +- .../src/elevenlabs/speech.rs | 8 +- crates/grida-fx-providers/src/fal/README.md | 57 ++ .../grida-fx-providers/src/fal/background.rs | 6 +- crates/grida-fx-providers/src/fal/image.rs | 20 +- crates/grida-fx-providers/src/fal/mod.rs | 47 +- crates/grida-fx-providers/src/fal/output.rs | 4 +- crates/grida-fx-providers/src/fal/video.rs | 34 +- crates/grida-fx-providers/src/gg/README.md | 86 +++ crates/grida-fx-providers/src/gg/chat.rs | 8 +- crates/grida-fx-providers/src/gg/image.rs | 8 +- crates/grida-fx-providers/src/gg/job.rs | 12 +- crates/grida-fx-providers/src/gg/mesh.rs | 14 +- crates/grida-fx-providers/src/gg/mod.rs | 22 +- crates/grida-fx-providers/src/gg/video.rs | 11 +- crates/grida-fx-providers/src/grok/README.md | 50 ++ crates/grida-fx-providers/src/grok/mod.rs | 2 +- .../grida-fx-providers/src/grok/protocol.rs | 4 +- crates/grida-fx-providers/src/grok/runtime.rs | 6 +- crates/grida-fx-providers/src/installed.rs | 6 +- crates/grida-fx-providers/src/keys.rs | 2 +- crates/grida-fx-providers/src/lib.rs | 3 +- crates/grida-fx-providers/src/live.rs | 4 +- .../grida-fx-providers/src/openai/README.md | 31 + crates/grida-fx-providers/src/openai/mod.rs | 35 +- .../src/openrouter/README.md | 68 ++ .../src/openrouter/agent.rs | 8 +- .../src/openrouter/image.rs | 22 +- .../grida-fx-providers/src/openrouter/mod.rs | 32 +- .../src/openrouter/music.rs | 12 +- .../src/openrouter/structured.rs | 21 +- .../grida-fx-providers/src/process_group.rs | 2 +- crates/grida-fx-providers/src/redact.rs | 2 +- crates/grida-fx-providers/src/routes.rs | 6 +- crates/grida-fx-providers/src/setup.rs | 4 +- crates/grida-fx-providers/src/stand_in.rs | 6 +- .../grida-fx-providers/src/transport/http.rs | 6 +- .../grida-fx-providers/src/transport/mod.rs | 12 +- crates/grida-fx-providers/src/tripo/README.md | 54 ++ crates/grida-fx-providers/src/tripo/mesh.rs | 8 +- crates/grida-fx-providers/src/tripo/mod.rs | 48 +- crates/grida-fx-providers/src/tripo/rig.rs | 8 +- crates/grida-fx-providers/src/wire.rs | 16 +- crates/grida-fx-providers/tests/codex.rs | 2 +- crates/grida-fx-providers/tests/elevenlabs.rs | 4 +- .../tests/fal_background.rs | 2 +- crates/grida-fx-providers/tests/fal_image.rs | 2 +- crates/grida-fx-providers/tests/fal_video.rs | 19 +- crates/grida-fx-providers/tests/gg.rs | 2 +- crates/grida-fx-providers/tests/grok.rs | 2 +- crates/grida-fx-providers/tests/keys.rs | 2 +- crates/grida-fx-providers/tests/openai.rs | 21 +- .../tests/openrouter_agent.rs | 4 +- .../tests/openrouter_image.rs | 4 +- .../tests/openrouter_music.rs | 4 +- .../tests/openrouter_structured.rs | 6 +- crates/grida-fx-providers/tests/routes.rs | 213 ++++--- crates/grida-fx-providers/tests/stand_in.rs | 4 +- crates/grida-fx-providers/tests/transport.rs | 10 +- crates/grida-fx-providers/tests/tripo_mesh.rs | 46 +- crates/grida-fx-providers/tests/tripo_rig.rs | 16 +- crates/grida-fx-runtime/src/calls/retry.rs | 6 +- crates/grida-fx-runtime/src/host/process.rs | 3 +- crates/grida-fx-runtime/tests/calls.rs | 2 +- .../grida-fx-runtime/tests/live_adapters.rs | 4 +- crates/grida-fx-runtime/tests/retry.rs | 2 +- crates/grida-fx/src/cli.rs | 2 +- crates/grida-fx/src/engine.rs | 4 +- crates/grida-fx/src/verbs/doctor.rs | 2 +- crates/grida-fx/src/verbs/run.rs | 2 +- crates/grida-fx/tests/cli.rs | 2 +- docs/guide/05-running.md | 8 +- docs/guide/12-codex.md | 2 +- docs/guide/13-providers.md | 2 +- docs/guide/14-grok.md | 2 +- docs/wg/overview.md | 2 +- docs/wg/provider-contract.md | 15 +- llms.txt | 2 +- python/tests/test_stand_in_engine.py | 2 +- python/tests/test_tools.py | 104 ++- spec/README.md | 3 +- spec/adapters.md | 153 +++++ spec/capabilities.md | 24 +- spec/distribution.md | 2 +- spec/protocol.md | 8 +- spec/providers.md | 591 ++---------------- spec/schemas/fx-call-record-v1.schema.json | 2 +- spec/service.md | 2 +- spec/store.md | 4 +- tools/check_examples.py | 2 +- tools/check_llms.py | 2 +- tools/check_spec.py | 228 ++++++- 114 files changed, 1526 insertions(+), 1099 deletions(-) create mode 100644 crates/grida-fx-providers/src/codex/README.md create mode 100644 crates/grida-fx-providers/src/elevenlabs/README.md create mode 100644 crates/grida-fx-providers/src/fal/README.md create mode 100644 crates/grida-fx-providers/src/gg/README.md create mode 100644 crates/grida-fx-providers/src/grok/README.md create mode 100644 crates/grida-fx-providers/src/openai/README.md create mode 100644 crates/grida-fx-providers/src/openrouter/README.md create mode 100644 crates/grida-fx-providers/src/tripo/README.md create mode 100644 spec/adapters.md diff --git a/Cargo.toml b/Cargo.toml index b7897b51..3d77802e 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -35,7 +35,7 @@ chrono = { version = "0.4", default-features = false, features = ["clock", "std" base64 = "0.22" getrandom = "0.3" unicode-general-category = "1.1" -# rustls trusting the bundled roots and the platform's certificate store (spec/providers.md §2). +# rustls trusting the bundled roots and the platform's certificate store (spec/adapters.md §2). reqwest = { version = "0.12", default-features = false, features = [ "rustls-tls-webpki-roots", "rustls-tls-native-roots", diff --git a/ISSUES.md b/ISSUES.md index 799d32a1..291abd68 100644 --- a/ISSUES.md +++ b/ISSUES.md @@ -6,7 +6,7 @@ Known problems that are not fixed yet. Each says what was seen, what is known, a **Status:** open. A design gap, not a bug. -An answer holds only its files and its `data`: "no request id, no usage, no provider URL" ([providers.md](spec/providers.md) §4.4, and the §11 change notes). That rule is right for the answer, since a call record is replayed to node bodies and must not depend on which provider made it. But nothing else keeps the provider's usage either: not the call record, and not the run's events. FX reads the cost it needs and drops the rest. +An answer holds only its files and its `data`: "no request id, no usage, no provider URL" ([adapters.md](spec/adapters.md) §3.4). That rule is right for the answer, since a call record is replayed to node bodies and must not depend on which provider made it. But nothing else keeps the provider's usage either: not the call record, and not the run's events. FX reads the cost it needs and drops the rest. So a bill cannot be explained after the fact: @@ -14,9 +14,9 @@ So a bill cannot be explained after the fact: - OpenAI reports token usage and no cost, so FX books an OpenAI image at its whole hold and keeps none of the tokens. Nothing then shows whether the hold covered the call. - No request or generation id is kept, so a call cannot be looked up at the provider afterwards. -The per-size image prices of [providers.md](spec/providers.md) §10 could only be measured from an earlier engine's records, which kept the usage block. Issue 2 is open because FX's own run kept nothing. +The per-size image prices of [providers.md](spec/providers.md) §3 could only be measured from an earlier engine's records, which kept the usage block. Issue 2 is open because FX's own run kept nothing. -**Proposal.** Keep answers as they are. Record what the provider reported on the run's `call` event: the token counts and the cost breakdown, as allowlisted numbers, and the provider's request id when it is a safe id ([providers.md](spec/providers.md) §8). None of it is a secret, none of it enters a call key, and a replay would carry no usage, since nothing was billed. Spec: [protocol.md](spec/protocol.md) (the `call` event), [providers.md](spec/providers.md) §4.4 and each provider's section of §9, and the events schema. +**Proposal.** Keep answers as they are. Record what the provider reported on the run's `call` event: the token counts and the cost breakdown, as allowlisted numbers, and the provider's request id when it is a safe id ([adapters.md](spec/adapters.md) §7). None of it is a secret, none of it enters a call key, and a replay would carry no usage, since nothing was billed. Spec: [protocol.md](spec/protocol.md) (the `call` event), [adapters.md](spec/adapters.md) §3.4 and each adapter's reference ([providers.md](spec/providers.md) §2), and the events schema. ## 2. OpenRouter billed twice the image output for one call @@ -39,7 +39,7 @@ The per-size image prices of [providers.md](spec/providers.md) §10 could only b **Open candidates:** -1. **`moderation: low`.** FX always passes it to OpenAI through `provider.options` ([providers.md](spec/providers.md) §9.2). The earlier calls that billed the single count did not send it. It is the only request difference found, and no mechanism is known for it to double the output. +1. **`moderation: low`.** FX always passes it to OpenAI through `provider.options` ([OpenRouter's adapter reference](crates/grida-fx-providers/src/openrouter/README.md)). The earlier calls that billed the single count did not send it. It is the only request difference found, and no mechanism is known for it to double the output. 2. **An unannounced change since late September**, at OpenRouter (counting the output twice) or at OpenAI (the model drawing twice as much at `max`). **Seen since.** On 2026-10-12, a 1024x1024 `max` generate (background `opaque`) on @@ -54,4 +54,4 @@ does not send `moderation: low`; it does not say which upstream GG used. - **Two 1024x1024 `max` calls on OpenRouter**, one with `moderation: low` and one without (about $0.85), separate candidate 1 from candidate 2. - **Once issue 1 is fixed**, any new call shows its breakdown in the run. -**What depends on it.** FX's OpenRouter image highs cover twice the output as a precaution. They are only holds, since OpenRouter reports its cost. If OpenAI itself now bills twice, the OpenAI and fal highs under-count by up to one output: a 1024x1024 generate would cost about $0.42 and be booked at $0.29 ([providers.md](spec/providers.md) §10, "An open risk"). +**What depends on it.** FX's OpenRouter image highs cover twice the output as a precaution. They are only holds, since OpenRouter reports its cost. If OpenAI itself now bills twice, the OpenAI and fal highs under-count by up to one output: a 1024x1024 generate would cost about $0.42 and be booked at $0.29 ([providers.md](spec/providers.md) §3, "An open risk"). diff --git a/TODO.md b/TODO.md index 8663bf63..b3d85b74 100644 --- a/TODO.md +++ b/TODO.md @@ -369,9 +369,9 @@ implementation; installation alone does not authorize invocation. ## Codex-backed image generation for local workflows **Status: in review, 2026-10-12** ([gridaco/fx#46](https://github.com/gridaco/fx/pull/46)), as -the route `imagegen@codex` of `image.generate` -([spec/providers.md §9.6](spec/providers.md#96-codex)). Users signed into Codex with ChatGPT pick -it in `fx.yaml` and run with no provider key. `codex` bills the user's plan, not dollars: a live run whose +the route `imagegen@codex` of `image.generate` ([the Codex adapter's +reference](crates/grida-fx-providers/src/codex/README.md)). Users signed into Codex with ChatGPT +pick it in `fx.yaml` and run with no provider key. `codex` bills the user's plan, not dollars: a live run whose calls are all on it needs no `--max-usd`, runs under a ceiling of $0 and refuses any other provider. Codex is a *runtime* the engine owns: one `codex app-server` per invocation, started on the first Codex call, shared by the rest (a thread each) and shut down with the run; its @@ -431,7 +431,7 @@ Follow-ups: **Grok Build: in review, 2026-10-13** ([gridaco/fx#72](https://github.com/gridaco/fx/issues/72)), as the route `imagine@grok` of `image.generate` and `image.edit` -([spec/providers.md §9.8](spec/providers.md#98-grok-build)): Grok Build's own ACP server +([the Grok Build adapter's reference](crates/grida-fx-providers/src/grok/README.md)): Grok Build's own ACP server (`grok agent --no-leader stdio`) through FX's ACP client ([`acp`](crates/grida-fx-providers/src/acp/mod.rs)), a session per call whose agent offers only Grok Build's two image tools, billed to the user's subscription (`Billing::Plan`). Verified with fake Grok Build programs in tests and real runs on @@ -442,7 +442,8 @@ Grok Build 1.0.52. Follow-ups: client is provider-neutral. Still per runtime: the JSON-RPC connection (Codex's app-server client and [`acp`](crates/grida-fx-providers/src/acp/mod.rs) differ only in routing keys and request answers), the start-once runtime state, the version check (gridaco/fx#73 for Codex), - and spec/providers.md §9.6 and §9.8, which restate the same rules; share them spec first. + and the Codex and Grok Build adapters' references, which restate the same rules; share them spec + first. - **An aspect ratio.** Grok Build's tools take one (`1:1`, `16:9`, `9:16`, …); FX leaves it unset until `image.generate` has a shape setting (gridaco/fx#57). - **Retrying a failed image tool.** A passing network error between Grok Build and xAI fails the @@ -461,8 +462,8 @@ gridaco/fx#72. ## Grida Gateway follow-ups -**Status: in review, 2026-10-12.** Grida Gateway (`@gg`, [spec/providers.md -§9.7](spec/providers.md#97-grida-gateway)) serves text, agent turns, images, video, 3D models and +**Status: in review, 2026-10-12.** Grida Gateway (`@gg`, [its adapter +reference](crates/grida-fx-providers/src/gg/README.md)) serves text, agent turns, images, video, 3D models and rigs with one key, `GG_API_KEY`, and is what the docs recommend first. FX books a GG call at GG's settled charge, read from the response or its billing readback; one that has not settled within a short wait is booked at its whole hold. diff --git a/conformance/README.md b/conformance/README.md index ede2b71c..36ff6004 100644 --- a/conformance/README.md +++ b/conformance/README.md @@ -231,7 +231,7 @@ Each step runs with a minimal environment, never the caller's: - `HOME`, a fresh empty folder for the case; - `NO_COLOR=1`, `LANG=C.UTF-8` and `PYTHONDONTWRITEBYTECODE=1`; - `GRIDA_FX_NETWORK=off` and `GRIDA_FX_DISABLE_DOTENV=1`, so no provider is reached and no `.env` - is read ([providers.md](../spec/providers.md) §2, §3); + is read ([adapters.md](../spec/adapters.md) §2; [providers.md](../spec/providers.md) §1); - `GRIDA_FX_PYTHON` and `GRIDA_FX_NODE`, each passed through when set (a path made absolute, above); - `RUSTUP_HOME` and `CARGO_HOME`, from the caller, else `~/.rustup` and `~/.cargo` when those diff --git a/conformance/run.py b/conformance/run.py index 24aeeea5..a10f6612 100644 --- a/conformance/run.py +++ b/conformance/run.py @@ -230,7 +230,7 @@ def step_environment(home: Path, passed: dict[str, str]) -> dict[str, str]: No provider key, tool override, FX cache location or locale can change what a case prints, and a case that runs `--live` reaches no provider: the network is off and no `.env` is read - (spec/providers.md sections 2 and 3). + (spec/adapters.md section 2; spec/providers.md section 1). """ return { diff --git a/crates/grida-fx-core/src/expand/bind.rs b/crates/grida-fx-core/src/expand/bind.rs index 68a78c57..ea7c7e1a 100644 --- a/crates/grida-fx-core/src/expand/bind.rs +++ b/crates/grida-fx-core/src/expand/bind.rs @@ -15,11 +15,11 @@ //! `.requires`: `{name} is not an image.generate feature (spec/capabilities.md)` (several //! capabilities: `an image.generate or structured.generate feature`), followed by `; FX calls it //! {feature}` when the name is a renamed one of their features (spec/providers.md -//! §11). It is then left out of every route's missing features. A type that calls a capability +//! §4). It is then left out of every route's missing features. A type that calls a capability //! FX does not define (spec/capabilities.md §13, or a project's own) has no vocabulary to check //! against, so its routes alone decide. The vocabulary is data, `features.json`: each //! capability's features and the renamed ones; `tools/check_spec.py` and this module's tests hold -//! it to spec/capabilities.md and spec/providers.md §11. +//! it to spec/capabilities.md and spec/providers.md §4. //! //! A capability step's own request (spec/capabilities.md §1): for `image.generate` and //! `image.edit`, the with-values imply features ([`implied_features`]), each one the route lacks @@ -66,7 +66,7 @@ pub fn capability_features(capability: &str) -> Option<&'static [String]> { vocabulary().capabilities.get(capability).map(Vec::as_slice) } -/// FX's name for a renamed feature (spec/providers.md §11). +/// FX's name for a renamed feature (spec/providers.md §4). pub fn renamed_feature(name: &str) -> Option<&'static str> { vocabulary().renamed.get(name).map(String::as_str) } @@ -660,7 +660,9 @@ mod tests { #[test] fn every_renamed_feature_is_one_providers_md_drops_for_a_feature_of_fx() { let providers = spec_text("providers.md"); - let changes = &providers[providers.find("## 11. ").expect("providers.md §11")..]; + let changes = &providers[providers + .find("## 4. Renamed features") + .expect("providers.md §4")..]; let all: BTreeSet<&str> = vocabulary() .capabilities .values() @@ -671,7 +673,7 @@ mod tests { for (old, new) in &vocabulary().renamed { assert!( changes.contains(&format!("`{old}`")), - "{old} in providers.md §11" + "{old} in providers.md §4" ); assert!(!all.contains(old.as_str()), "{old} is still a feature"); assert!(all.contains(new.as_str()), "{new} is a feature"); diff --git a/crates/grida-fx-core/tests/expand_semantics.rs b/crates/grida-fx-core/tests/expand_semantics.rs index 10dc32f5..abafcb1f 100644 --- a/crates/grida-fx-core/tests/expand_semantics.rs +++ b/crates/grida-fx-core/tests/expand_semantics.rs @@ -1331,7 +1331,7 @@ steps: ", ); // A name no capability the type calls defines is refused as such, once, with FX's name for - // a renamed one (spec/providers.md §11); a known feature a route lacks is the route's. + // a renamed one (spec/providers.md §4); a known feature a route lacks is the route's. assert_eq!( problems(&expansion), vec![ diff --git a/crates/grida-fx-providers/Cargo.toml b/crates/grida-fx-providers/Cargo.toml index 0f2fce8a..2f2fd655 100644 --- a/crates/grida-fx-providers/Cargo.toml +++ b/crates/grida-fx-providers/Cargo.toml @@ -25,7 +25,7 @@ url.workspace = true reqwest.workspace = true sha2.workspace = true jsonschema.workspace = true -# Picture reduction for structured output (spec/providers.md §9.2). +# Picture reduction for structured output (src/openrouter/README.md). image.workspace = true png = { workspace = true, optional = true } # The scratch folder a Codex conversation runs in. diff --git a/crates/grida-fx-providers/routes/default.yaml b/crates/grida-fx-providers/routes/default.yaml index 48e73cc2..685f9097 100644 --- a/crates/grida-fx-providers/routes/default.yaml +++ b/crates/grida-fx-providers/routes/default.yaml @@ -1,36 +1,35 @@ -# The built-in default route table (spec/identity.md §7; spec/providers.md §10). +# The built-in default route table (spec/identity.md §7; spec/providers.md §3). # # Embedded in the grida-fx binary and used unless a --routes file is given; a project's # route_tables override any entry. Prices are planning allowances in US dollars, not provider # quotes: what a call costs is settled from what the provider reports, else the whole hold -# (spec/providers.md §6). +# (spec/adapters.md §5). # # Grida Gateway (`@gg`) is Grida's own provider: one key, GG_API_KEY, serves its # text, image, video and 3D routes. It charges a model's regular tariff, so its routes take the # prices of the same model elsewhere. A call is booked at GG's settled charge, else at its whole -# hold (spec/providers.md §9.7, §10). +# hold (src/gg/README.md; spec/providers.md §3). # # The Codex routes (imagegen@codex, for image.generate and image.edit) run on the user's own # signed-in Codex, which counts them against their plan: they are free to FX, $0, take at most 5 # input pictures, and share one route id, so at most three of their calls run at once on the -# invocation's one Codex (spec/providers.md §9.6, §10). +# invocation's one Codex (src/codex/README.md; spec/providers.md §3). # # The Grok Build routes (imagine@grok, likewise) run on the user's own signed-in Grok Build, on # their Grok subscription: free to FX, $0, at most 3 input pictures, three calls at once, opaque -# images only (spec/providers.md §9.8, §10). +# images only (src/grok/README.md; spec/providers.md §3). # # The other image routes are priced per call, in tiers by `size`. A tier's low is the output tokens at # quality max for that size, measured or derived from a measured bill; its high adds an allowance # for the prompt and the input pictures, and on OpenRouter covers twice the output -# (spec/providers.md §10). A size the tiers do not list (`auto`, none, or any other WxH) takes the +# (spec/providers.md §3). A size the tiers do not list (`auto`, none, or any other WxH) takes the # whole range. A project restates a route in its own route table for tighter prices. # # The contract object of each entry is part of its route fingerprint, and so of every cache key -# the route serves: editing one changes those keys. Its `adapter` (and, for the Grida Gateway, -# OpenAI and OpenRouter routes, `adapter_behavior`) is the one spec/providers.md §9 names for the -# capability's adapter on that provider. The image routes carry `adapter_behavior` as a string, -# every other route as the integer 1. Price, features, max_input_pictures, concurrency and -# requests_per_minute are not identity. +# the route serves: editing one changes those keys. Its `adapter` and `adapter_behavior` are the +# ones spec/providers.md §2 names for the capability's adapter on that provider. The image routes +# carry `adapter_behavior` as a string, every other route as the integer 1. Price, features, +# max_input_pictures, concurrency and requests_per_minute are not identity. # # Features are the names spec/capabilities.md lists for each capability. tests/routes.rs holds # this file to both specs and to the adapters of grida-fx-providers. diff --git a/crates/grida-fx-providers/src/acp/mod.rs b/crates/grida-fx-providers/src/acp/mod.rs index f39142b1..f4e95263 100644 --- a/crates/grida-fx-providers/src/acp/mod.rs +++ b/crates/grida-fx-providers/src/acp/mod.rs @@ -1,5 +1,5 @@ //! A client of the Agent Client Protocol (ACP v1) over a program's stdin and stdout -//! (spec/providers.md §9.8): JSON-RPC 2.0, one message per line. It keeps ACP's own vocabulary at +//! (src/grok/README.md): JSON-RPC 2.0, one message per line. It keeps ACP's own vocabulary at //! this boundary; what a session means is the driver's (Grok Build's, [`crate::grok`]). //! //! Two tasks serve a connection: a writer, which owns the program's stdin and writes the lines it @@ -18,7 +18,7 @@ //! //! A `session/new` response opens its session's notifications before the reader reads on, so //! none of a prompt's is missed. Reasons are fixed sentences: an agent's own text can name private -//! paths (spec/providers.md §8). +//! paths (spec/adapters.md §7). use crate::installed::{locked, read_line}; use crate::process_group::Group; diff --git a/crates/grida-fx-providers/src/adapter.rs b/crates/grida-fx-providers/src/adapter.rs index 50f79f86..674ffbaa 100644 --- a/crates/grida-fx-providers/src/adapter.rs +++ b/crates/grida-fx-providers/src/adapter.rs @@ -71,7 +71,7 @@ pub struct AnsweredFile { pub bytes: Vec, } -/// How a provider bills its calls (spec/providers.md §6). +/// How a provider bills its calls (spec/adapters.md §5). #[derive(Debug, Clone, Copy, PartialEq, Eq)] pub enum Billing { /// In US dollars, to the account of the key FX sends: what the ceiling bounds. @@ -178,7 +178,7 @@ pub enum Sent { /// nothing. The engine may send the same request again under the same attempt, at $0. /// `retry_after` is the wait the provider asked for before that ([`retry_after`]; `None` when /// it asked for none): the engine waits at least its own backoff, and at least `retry_after` - /// up to its cap (spec/providers.md §4.3). + /// up to its cap (spec/adapters.md §3.3). NotReceived { reason: String, retry_after: Option, @@ -346,9 +346,10 @@ pub enum Collected { Answered(Answer), /// The job ended without a result: the record becomes `settled`, the call fails /// `call_failed`, and a later run submits it anew. `cost` is what the ended job costs - /// (spec/providers.md §7): the provider's own figure for it, as for an answer when the job - /// finished with an unusable result; `Some(0)` only where spec/providers.md §9 says so; `None` - /// otherwise, and the run that submitted the job then charges its whole hold. + /// (spec/adapters.md §6): the provider's own figure for it, as for an answer when the job + /// finished with an unusable result; `Some(0)` only where its reference says so + /// (spec/providers.md §2); `None` otherwise, and the run that submitted the job then charges + /// its whole hold. Ended { reason: String, cost: Option }, /// Collecting failed and the job may still be running: the record stays `submitted`, the /// call fails `call_failed`, and a later run collects again. @@ -371,8 +372,8 @@ pub trait RequestAdapter: Send + Sync { Billing::Dollars } - /// The redactor every reason of this adapter's calls passes through last (spec/providers.md - /// §8). The engine's retry owner redacts with it each reason it reports: the adapter's own, + /// The redactor every reason of this adapter's calls passes through last (spec/adapters.md + /// §7). The engine's retry owner redacts with it each reason it reports: the adapter's own, /// its check's, and the engine's checks of the answer. A live invocation's adapters answer /// with the redactor over every key of the invocation ([`crate::live::adapters`]); the /// default masks the known secret shapes only. @@ -407,7 +408,7 @@ pub trait LongJob: Send + Sync { } /// A runtime some adapters share: a program the user installed and signed into (Codex), started -/// on an invocation's first call that needs it and kept for the others (spec/providers.md §1). The +/// on an invocation's first call that needs it and kept for the others (spec/adapters.md §1). The /// engine owns its life: it is shut down with the run ([`crate::Adapters::shutdown`]). pub trait Runtime: Send + Sync { /// Ends the runtime and everything it started. Whether everything is verifiably ended; a diff --git a/crates/grida-fx-providers/src/capabilities.rs b/crates/grida-fx-providers/src/capabilities.rs index ecc67ac2..75b2d8ca 100644 --- a/crates/grida-fx-providers/src/capabilities.rs +++ b/crates/grida-fx-providers/src/capabilities.rs @@ -1,6 +1,6 @@ //! The capabilities FX ships, as data (spec/capabilities.md): each one's canonical request members, //! the features a route of it may declare, and what its answer carries. Adapters call -//! [`check_request`] first (spec/providers.md §5 step 2), so every route of a capability refuses +//! [`check_request`] first (spec/adapters.md §4 step 2), so every route of a capability refuses //! the same malformed request with the same sentence. `tests/capabilities.rs` holds this table to //! spec/capabilities.md (every member table and feature list) and to the standard library's paid //! built-ins; `tests/routes.rs` holds the built-in route table's features to [`Capability::features`]. @@ -19,7 +19,7 @@ //! object`). //! //! Value checks beyond the type (non-blank text, ranges, enums) are the adapter's, with the -//! sentences spec/providers.md §9 gives per provider. +//! sentences of its reference (spec/providers.md §2). use serde_json::Value; diff --git a/crates/grida-fx-providers/src/chat/mod.rs b/crates/grida-fx-providers/src/chat/mod.rs index 658c3c1c..b55944b0 100644 --- a/crates/grida-fx-providers/src/chat/mod.rs +++ b/crates/grida-fx-providers/src/chat/mod.rs @@ -1,7 +1,7 @@ //! Chat Completions as FX speaks it, whichever provider serves the route: the structured-output //! prompt and answer (spec/capabilities.md §4) and the agent turn's transcript, tools and reply -//! (§5), with the pictures they carry (spec/providers.md §9.2; Grida Gateway's adapters, §9.7, -//! send the same messages). +//! (§5), with the pictures they carry (src/openrouter/README.md; Grida Gateway's adapters, +//! src/gg/README.md, send the same messages). //! //! What a provider adds is its own module's: the endpoint and credential, provider members of the //! body (OpenRouter's routing and reasoning), how a status classifies, and what a call costs. @@ -114,11 +114,12 @@ pub struct StructuredPrompt { pub max_tokens: u64, } -/// Steps 4 and 5 of a structured call (spec/providers.md §5, §9.2), for a request that passed its -/// capability's check: the values (a non-blank `prompt`; `matte` a colour; `max_tokens` at least -/// 1; an inline schema whose references inline and which compiles as a draft 2020-12 JSON -/// Schema), then the files (the schema file, then each context file, a picture or UTF-8 text). -/// Pictures go as they are when `pictures_unchanged`, else reduced; each part carries `detail`. +/// Steps 4 and 5 of a structured call (spec/adapters.md §4; src/openrouter/README.md), for a +/// request that passed its capability's check: the values (a non-blank `prompt`; `matte` a colour; +/// `max_tokens` at least 1; an inline schema whose references inline and which compiles as a draft +/// 2020-12 JSON Schema), then the files (the schema file, then each context file, a picture or +/// UTF-8 text). Pictures go as they are when `pictures_unchanged`, else reduced; each part carries +/// `detail`. pub fn structured_prompt( call: &CallRequest, pictures_unchanged: bool, @@ -296,7 +297,7 @@ fn over(src: [u8; 4], matte: [u8; 4]) -> [u8; 3] { [channel(0), channel(1), channel(2)] } -/// A picture reduced for a structured call (spec/providers.md §9.2): decoded, flattened on +/// A picture reduced for a structured call (src/openrouter/README.md): decoded, flattened on /// `matte`, fitted into [`PICTURE_EDGE`] × [`PICTURE_EDGE`] with Lanczos3 when larger, and /// encoded as an RGB PNG data URL. `None` when the bytes do not decode. /// @@ -355,7 +356,7 @@ fn sent_schema(original: &Value) -> Result { } /// Unwraps the `completionState` envelopes a provider sometimes puts around an answer -/// (spec/providers.md §9.2): a string holding such an object is parsed; a complete envelope +/// (src/openrouter/README.md): a string holding such an object is parsed; a complete envelope /// gives its `entries` (as an object), `items` (as a list) or `value`; an incomplete or /// malformed one is left as it is; anything else is unwrapped member by member. fn unwrap_completion(value: Value) -> Value { @@ -421,12 +422,13 @@ pub struct AgentPrompt { pub max_tokens: Option, } -/// Steps 4 and 5 of an agent turn (spec/providers.md §5, §9.2), for a request that passed its -/// capability's check: the values (each tool's name `^[a-z][a-z0-9_]{0,63}$`, non-blank -/// description and object `parameters`; `max_tokens` at least 1; the transcript: known roles, text -/// contents, an id on every assistant call and every tool message, pictures as file values), then -/// the files: every picture's kind before any file is read (`messages[].images[] is , -/// not a picture`), then each as a data URL of its kind. Each picture part carries `detail`. +/// Steps 4 and 5 of an agent turn (spec/adapters.md §4; src/openrouter/README.md), for a request +/// that passed its capability's check: the values (each tool's name `^[a-z][a-z0-9_]{0,63}$`, +/// non-blank description and object `parameters`; `max_tokens` at least 1; the transcript: known +/// roles, text contents, an id on every assistant call and every tool message, pictures as file +/// values), then the files: every picture's kind before any file is read +/// (`messages[].images[] is , not a picture`), then each as a data URL of its kind. +/// Each picture part carries `detail`. pub fn agent_prompt(call: &CallRequest, detail: Option<&str>) -> Result { // 4. Values. let request = &call.request; @@ -677,7 +679,7 @@ fn wire_messages( Ok(out) } -/// The calls of a reply (spec/providers.md §9.2), or the structural failure's sentence. +/// The calls of a reply (src/openrouter/README.md), or the structural failure's sentence. fn reply_calls(value: Option<&Value>) -> Result, &'static str> { let entries = match value { None | Some(Value::Null) => return Ok(Vec::new()), diff --git a/crates/grida-fx-providers/src/chat/schema.rs b/crates/grida-fx-providers/src/chat/schema.rs index b48e12ba..ea8cffcf 100644 --- a/crates/grida-fx-providers/src/chat/schema.rs +++ b/crates/grida-fx-providers/src/chat/schema.rs @@ -1,4 +1,4 @@ -//! JSON Schemas as the strict mode of Chat Completions takes them (spec/providers.md §9.2, +//! JSON Schemas as the strict mode of Chat Completions takes them (src/openrouter/README.md, //! "Schemas"; OpenRouter and Grida Gateway alike). //! //! - [`inline_refs`]: when the root has a `$defs` object, every object whose `$ref` starts with diff --git a/crates/grida-fx-providers/src/checks.rs b/crates/grida-fx-providers/src/checks.rs index 41d91e75..a5270509 100644 --- a/crates/grida-fx-providers/src/checks.rs +++ b/crates/grida-fx-providers/src/checks.rs @@ -1,4 +1,4 @@ -//! The answer checks more than one route makes (spec/providers.md §4.4; spec/capabilities.md). +//! The answer checks more than one route makes (spec/adapters.md §3.4; spec/capabilities.md). //! //! Adapters' `check`s call these, so two routes never disagree on what "an exact size", "an MP3" //! or "meets the schema" means, and each keeps its own sentences where they differ (a missing @@ -8,7 +8,7 @@ //! (spec/protocol.md §6.1, *Stand-in answers*; [`crate::stand_in`]). //! //! A check judges the call (its route, request, files and take) and the answer alone, never -//! anything its adapter's send or collect kept (spec/providers.md §4.4), so a stand-in's answer can +//! anything its adapter's send or collect kept (spec/adapters.md §3.4), so a stand-in's answer can //! be held to it. //! //! [`ClipFacts`] are the facts a clip's answer carries as `data` (spec/capabilities.md §6), which diff --git a/crates/grida-fx-providers/src/clock.rs b/crates/grida-fx-providers/src/clock.rs index 593d7d97..497469f0 100644 --- a/crates/grida-fx-providers/src/clock.rs +++ b/crates/grida-fx-providers/src/clock.rs @@ -1,4 +1,4 @@ -//! Time for long-job polling (spec/providers.md §7): adapters never sleep except while a +//! Time for long-job polling (spec/adapters.md §6): adapters never sleep except while a //! `collect` (or a free pre-check inside a `submit`) polls a provider job, and they do it through //! an injected [`Clock`] so tests poll instantly. diff --git a/crates/grida-fx-providers/src/codex/README.md b/crates/grida-fx-providers/src/codex/README.md new file mode 100644 index 00000000..a90cb5f8 --- /dev/null +++ b/crates/grida-fx-providers/src/codex/README.md @@ -0,0 +1,52 @@ +# Codex + +The reference for FX's adapters on `codex`, under the adapter contract, [spec/adapters.md](../../../../spec/adapters.md), whose sections it cites; it is normative for these adapters. They have no key or endpoint: they work through a runtime ([adapters.md](../../../../spec/adapters.md) §1). [spec/providers.md](../../../../spec/providers.md) §2 and §3 hold the built-in adapters and the routes FX ships. + +| Capability | Contract `adapter` | `adapter_behavior` | +|---|---|---| +| `image.generate`, `image.edit` | `fx-codex-image-v1` | `"2"` | + +- **Capabilities:** `image.generate` and `image.edit`, on the user's own Codex: the `codex` command, signed in with the user's ChatGPT account. Contract `adapter`: `fx-codex-image-v1`, behaviour `"2"`, the string. The behaviour names the turn's instructions and the conversation's settings below; a change to either is a new behaviour. Behaviour `"1"` named Codex's image skill and took no pictures; it is refused ([adapters.md](../../../../spec/adapters.md) §4 step 1). +- **The model is Codex's.** The route's model, `imagegen`, names Codex's image tool (`image_gen.imagegen`), not a model: Codex chooses the model and the quality, and FX neither picks nor records them, so the route's cache keys outlive a change of either. `independent_of` cannot reason about this route. +- **No key, no transport.** The adapter starts `codex app-server` and speaks its JSON-RPC, one JSON message per line on stdin and stdout. Codex owns the sign-in, the model, the network and the usage: FX never reads Codex's credential files, and books nothing in dollars. Codex retries its own requests to OpenAI within a turn; that is Codex's, and FX never resends a Codex call (below). +- **The program:** `GRIDA_FX_TOOL_CODEX` when set and not blank, made absolute against the working directory, and then that program only: none when it names no executable file. Else the first `codex` in an absolute `PATH` entry (empty and relative entries are skipped), else `/Applications/ChatGPT.app/Contents/Resources/codex-cli/bin/codex`. Each must be an executable file. A live invocation looks for it once, and not at all with the network off; `grida-fx doctor` looks for it too, only to report it. Where it was found is named `GRIDA_FX_TOOL_CODEX`, `PATH` or `the ChatGPT app`, never by its path ([adapters.md](../../../../spec/adapters.md) §7). +- **The version.** Codex 0.160.1 or later is needed, the version this behaviour was verified on. FX asks `codex --version` (standard input closed), within 30 s; the version is the first word of what it prints that is `major.minor.patch`, perhaps with a `-` or `+` suffix (`codex-cli 0.160.1`; a leading `v` is passed over), compared by its three numbers. A Codex that exits with a failure, prints none, or prints none within 30 s names no version. `grida-fx doctor` asks the same, only to report it. +- **The environment** is FX's, without the eleven variables of [providers.md](../../../../spec/providers.md) §1 and without `CODEX_API_KEY`, `OPENAI_ORG_ID`, `OPENAI_ORGANIZATION`, `OPENAI_PROJECT` and `OPENAI_PROJECT_ID`. Codex's stderr is dropped. +- **The user's tools are off, and so are Codex's own.** Every Codex FX starts gets `-c notify=[]`, `-c features.memories=false`, `-c features.hooks=false`, `-c features.apps=false` and `-c features.plugins=false`; then `-c features.=false` for `shell_tool`, `unified_exec`, `multi_agent`, `browser_use`, `computer_use`, `in_app_browser`, `view_image`, `goals`, `sleep_tool`, `tool_suggest` and `skill_search`, and `-c web_search="disabled"`, so a turn is offered its image tool and little else (Codex ignores a feature it does not know, and a turn's context is about half as long); and `-c mcp_servers..enabled=false` for each enabled server `codex mcp list --json` reports under those settings (`` bare when it is letters, digits, `_` and `-`, else a TOML basic string). A server only a plugin defines is not named: Codex refuses a setting for it, and `features.plugins=false` already turns it off. A list Codex will not give within 30 s is a refusal: `Codex's MCP servers could not be listed, so they cannot be turned off`. +- **The runtime.** One `codex app-server` serves an invocation: started on its first Codex call, kept for the others, and shut down with the run ([adapters.md](../../../../spec/adapters.md) §1). Starting it: the version above, the MCP list below, then Codex in a scratch folder of its own, in a process group of its own (its MCP servers and helpers join it), then `initialize`, the `initialized` notification and `account/read`. A Codex that cannot draw on the user's plan (the refusals below) is refused for every call of the invocation, and is not asked again; a Codex that exits is replaced by the next call. Shutting down closes Codex's stdin, gives it 2 s to exit, sends `SIGTERM` to the group and 2 s more, then kills the group; a command that exits on an interrupt kills the group at once. Process groups are Unix's: elsewhere only Codex itself is ended. +- **Pipes.** One task writes Codex's stdin, and one reads its stdout: each response goes to the request waiting for it, each notification to the thread it names (a thread's notifications open with its `thread/start` response, so none is missed), and a request Codex sends (an approval, a question) is answered with error `-32601`: FX grants nothing. +- **Pictures.** A call's input pictures are `image` (an edit's), then `references`, in order: at most 5, Codex's own limit (its image tool draws nothing for more). Each is PNG, JPEG or WebP. FX copies each one's store copy into the call's scratch folder as `picture-.` (`n` from 1, two digits), and names them in the turn's text by absolute path, in order, as the image tool's `referenced_image_paths`: Codex reads them through its read-only sandbox, at their own resolution, and sends them to its image endpoint for edits. Nothing is attached to the turn: a picture attached to the conversation is, in Codex's own words, a best-effort reference. A mask is refused: Codex's image tool takes none. +- **Shapes.** Codex's image tool takes no size: its images come back about 1.57 megapixels. Without input pictures their shape follows the prompt's wording, between 1:3 and 3:1 (asked for about 4:1, about 1:3 and 16:9, it drew 2172x724, 724x2172 and 1672x941; asked for 1024x1024 it drew 1254x1254). With them, an image takes about the first picture's shape unless the prompt asks for another (a 1000x400 picture, edited, came back 1984x793). +- **Limits:** 60 s for Codex to accept the turn (starting the runtime too, on the invocation's first call), and 600 s for the whole send; a message from Codex is at most 64 MiB, refused as soon as it passes that. +- **Refusals ([adapters.md](../../../../spec/adapters.md) §4), in this order:** + - the route ([adapters.md](../../../../spec/adapters.md) §4 step 1) and the request ([adapters.md](../../../../spec/adapters.md) §4 step 2); + - a blank `prompt`: `an image call needs its prompt as text`; + - a `size` other than `auto`: `Codex image generation takes no exact size: use size auto, and fx/image.resize@1 after it` (Shapes, above: an exact size would only fail the check); + - a `background` other than `auto`, `opaque` or `transparent`: `background must be auto, opaque or transparent`; + - a `mask`: `Codex image edits take no mask`; + - more than 5 pictures: `Codex takes at most 5 input pictures`; + - a picture the request does not carry, or whose store copy is empty or unreadable: ` has no bytes to send`; one of another kind: ` is ; Codex takes PNG, JPEG or WebP` (`` is `image` or `references[]`); + - **the usage limit**, once a call of the invocation found it reached (an image item's failure `usageLimitExceeded`, or a turn's `codexErrorInfo`): `the Codex usage limit is reached; it resets at unix time , and no Codex call starts before then`, until that time; without a time, `the Codex usage limit is reached, and no Codex call starts again in this run`. Nothing is started for such a call, so a run with many Codex steps stops spending turns on a plan that cannot draw; + - no program: `Codex is not installed: install it with npm install -g @openai/codex and sign in with codex login, or set GRIDA_FX_TOOL_CODEX`; with the network off: `Codex was not started: GRIDA_FX_NETWORK is off`. A program that cannot be started: `Codex image generation cannot start codex: `; + - an older Codex (the version, above): `Codex (from ) is older than 0.160.1: update it, or set GRIDA_FX_TOOL_CODEX to a newer one`; one that names no version: `Codex (from ) did not name its version: update it to 0.160.1 or later, or set GRIDA_FX_TOOL_CODEX to a newer one`. `` is where the program was found (above); + - the MCP list above; + - **an account that does not draw on the user's plan**, from `account/read` when the runtime starts: no account, `Codex is not signed in: run codex login and sign in with ChatGPT`; an API key, `Codex is signed in with an API key, which bills an API account: run codex login and sign in with ChatGPT`; any other, `Codex is signed in with , not ChatGPT: run codex login and sign in with ChatGPT`; + - a thread whose `modelProvider` is not `openai`: `Codex uses the model provider , not OpenAI: image generation needs Codex signed in with ChatGPT`. +- **One send:** + 1. the runtime's Codex, started if need be (above); + 2. a scratch folder for this call alone, holding only its pictures (above), which FX removes after it; then `thread/start` in it, with `sandbox: read-only`, `approvalPolicy: never` and `ephemeral: true`; + 3. `turn/start` with `effort: low` and `serviceTierForTurn: default` (the user's own settings could spend their plan faster for nothing an image needs), and one text input: `Call the image_gen.imagegen tool exactly once. Pass the text between the tags as its prompt, exactly as written. Set transparent_background to true.` for `background: transparent`, else the same ending `false.` (Codex then draws an opaque image, so `auto` is opaque); then, with pictures, ` Set referenced_image_paths to these input images, in this order: ` and each path between backticks, separated by `, `, and `.`; then ` Do not run commands, read or edit files, use any other tool or ask questions. When the image is made, reply with one short sentence.`; a blank line; then ``, the prompt verbatim, and ``, each on its own line. The skill mention `$imagegen` is not used: it brings a skill written for a person's project, twice as long as the tool's own description; + 4. the thread's notifications of this turn (`turnId`, `turn.id`), until the first `imageGeneration` item that completed with an image, or `turn/completed`; `thread/tokenUsage/updated` is read on the way. + + A send that is dropped (its run was cancelled) or runs out of time asks Codex to stop its turn (`turn/interrupt`); the runtime goes on. One dropped before Codex answered its `turn/start` cannot name the turn yet: the runtime's reader, which receives that answer, interrupts the turn it names. +- **Answer.** The first `imageGeneration` item that completed: its `result` must be strict base64 of the image. FX then asks Codex to stop the rest of the turn (`turn/interrupt`) and reads its token usage until the turn ends, or 3 s. The answer is file `image` (its kind sniffed from the bytes; the check wants `image/png`) and `data: null`. Cost: `None`, since Codex reports none: the route's hold, $0, is what is charged, and the record says the cost was not reported. Usage ([adapters.md](../../../../spec/adapters.md) §5): billing `plan`, `runtime` `codex ` (from `initialize`'s `userAgent`), `agent_model` (the thread's `model`), and the turn's total `input_tokens`, `cached_input_tokens` and `output_tokens`, each when Codex gave it. A version or model is kept only when it is letters, digits, `.`, `-`, `_` and `+` (a version) at most 40, or letters, digits, `.`, `-` and `_` (a model) at most 60. +- **Outcomes.** One error is provably not received: `-32001` (Codex's app-server is overloaded and rejected the request on arrival) in answer to any of FX's requests is `NotReceived` ([adapters.md](../../../../spec/adapters.md) §3.1), `Codex image generation was not started: Codex is overloaded (error -32001)`, and the engine may send the call again. Every other failure is `Failed { cost: None, retryable: false }`, and that call is never sent again by the engine. Before `turn/start` is written, a failure is most likely to repeat (a Codex that will not start, a setting it refuses); once it is written, an image may already have been made and counted against the user's plan. The reason starts `Codex image generation was not started: ` before `turn/start` is written, and `Codex image generation failed: ` after, followed by: + - `Codex exited`, `Codex's output is unreadable`, `Codex sent a line that is not JSON`, `Codex sent a message over 64 MiB`, `Codex could not read a request`, `Codex started no thread`, `Codex has no pipes`, `the command is stopping`; + - `Codex refused (error )` for an error response; + - `Codex did not start the turn within s`, or `Codex did not finish within s` (FX first asks Codex to stop the turn); + - for an image item that failed: `the Codex usage limit is reached; it resets at unix time ` (without the time when Codex gives none), or `Codex's image generation failed ()`; one that did not complete: `Codex's image generation ended `; a `result` that is not base64: `Codex's image is not valid base64`. A later good image of the same turn wins over these; + - for a turn that completed without an image: by its error's `codexErrorInfo`, `the Codex usage limit is reached` (`usageLimitExceeded`), `Codex is not signed in: run codex login and sign in with ChatGPT` (`unauthorized`), or `Codex's turn failed ()`; an error without one, `Codex's turn failed`; no error, `Codex answered without an image`, or `Codex's turn ended without an image`. + + ``, ``, `` and `` are Codex's words only when they are letters, digits and `_`, at most 40 of them, else `unknown`. Codex's own messages are never shown: they can name private paths ([adapters.md](../../../../spec/adapters.md) §7). +- **Check:** the image checks of [capabilities.md](../../../../spec/capabilities.md) §2. An answer that fails them is a billed attempt like any other ([adapters.md](../../../../spec/adapters.md) §3.4): the engine may make a new attempt, and each is a new image on the user's plan. +- **Reasons:** the label is `Codex image generation`. diff --git a/crates/grida-fx-providers/src/codex/mod.rs b/crates/grida-fx-providers/src/codex/mod.rs index 772c25ed..605de1c3 100644 --- a/crates/grida-fx-providers/src/codex/mod.rs +++ b/crates/grida-fx-providers/src/codex/mod.rs @@ -1,5 +1,5 @@ //! Codex: `image.generate` and `image.edit` through the user's own Codex, signed in with ChatGPT -//! (spec/providers.md §9.6). +//! (src/codex/README.md). //! //! No key and no HTTP: the adapter draws through a [`CodexRuntime`], one `codex app-server` an //! invocation starts on its first Codex call, keeps for the others and shuts down with the run. diff --git a/crates/grida-fx-providers/src/codex/protocol.rs b/crates/grida-fx-providers/src/codex/protocol.rs index 1b8548a6..fd6a9761 100644 --- a/crates/grida-fx-providers/src/codex/protocol.rs +++ b/crates/grida-fx-providers/src/codex/protocol.rs @@ -1,7 +1,7 @@ -//! What `codex app-server` says, read (spec/providers.md §9.6): JSON-RPC 2.0 messages, the +//! What `codex app-server` says, read (src/codex/README.md): JSON-RPC 2.0 messages, the //! account, image items, turn failures and token usage. Reasons are fixed sentences with bounded //! tokens from Codex's closed vocabularies, never Codex's own text, which can name private paths -//! (spec/providers.md §8). +//! (spec/adapters.md §7). use super::Stop; use crate::adapter::Usage; @@ -84,7 +84,7 @@ pub(super) fn check_account(read: &Value) -> Result<(), Stop> { } /// The error code of a request Codex's app-server rejected on arrival because it is overloaded: -/// the request was provably not received (spec/providers.md §9.6). +/// the request was provably not received (src/codex/README.md). pub(super) const OVERLOADED: i64 = -32001; /// The reason of a request rejected with [`OVERLOADED`]. diff --git a/crates/grida-fx-providers/src/codex/runtime.rs b/crates/grida-fx-providers/src/codex/runtime.rs index 2c8343ec..3a8752d9 100644 --- a/crates/grida-fx-providers/src/codex/runtime.rs +++ b/crates/grida-fx-providers/src/codex/runtime.rs @@ -1,6 +1,6 @@ -//! The user's Codex as a runtime (spec/providers.md §1, §9.6): one `codex app-server` an -//! invocation starts on its first Codex call and keeps for the others, and shuts down with the run -//! ([`crate::adapter::Runtime`]). +//! The user's Codex as a runtime (spec/adapters.md §1; src/codex/README.md): one `codex app-server` +//! an invocation starts on its first Codex call and keeps for the others, and shuts down with the +//! run ([`crate::adapter::Runtime`]). //! //! Starting it, once: check its version ([`super::version`]), list the user's MCP servers to turn //! them off, start Codex in a process group of its own ([`crate::process_group`]), `initialize`, diff --git a/crates/grida-fx-providers/src/elevenlabs/README.md b/crates/grida-fx-providers/src/elevenlabs/README.md new file mode 100644 index 00000000..1ecc561c --- /dev/null +++ b/crates/grida-fx-providers/src/elevenlabs/README.md @@ -0,0 +1,27 @@ +# ElevenLabs + +The reference for FX's adapters on `elevenlabs`, under the adapter contract, [spec/adapters.md](../../../../spec/adapters.md), whose sections it cites; it is normative for these adapters. [spec/providers.md](../../../../spec/providers.md) §1, §2 and §3 hold their key and endpoint, the built-in adapters and the routes FX ships. + +| Capability | Contract `adapter` | `adapter_behavior` | +|---|---|---| +| `sound.generate` | `elevenlabs-sound-effect` | `1` | +| `speech.generate` | `elevenlabs-speech` | `1` | + +- **Capabilities:** + - `sound.generate`: `POST {base}/sound-generation?output_format=mp3_44100_192`. Contract `adapter`: `elevenlabs-sound-effect`, behaviour `1`. + - `speech.generate`: `POST {base}/text-to-speech/?output_format=mp3_44100_192`. Contract `adapter`: `elevenlabs-speech`, behaviour `1`. +- **Credential:** `xi-api-key: `, plus headers `content-type: application/json` and `accept: audio/mpeg`. +- **Limits:** deadline 120 s; response cap 64 MiB. +- **Sound.** + - Refused before sending, in this order: a blank prompt (`a sound call needs its prompt as text`), or one over 450 characters (`sound effect prompt must be at most 450 characters`); a `duration` outside 0.5 to 30 (`duration must be between 0.5 and 30`); a `prompt_influence` outside 0 to 1 (`prompt_influence must be between 0 and 1`). + - The body is `{text, model_id: , loop, duration_seconds?, prompt_influence?}`, with `loop` false when absent. +- **Speech.** + - Refused before sending, in this order: blank text (`a speech call needs its text`); a blank voice (`a speech call needs its provider voice`); a voice that does not match `^[A-Za-z0-9_-]{1,128}$` (`the provider voice is not a voice id`); text over 5000 characters (`speech text must be at most 5000 characters`); a `stability` outside 0 to 1 (`stability must be between 0 and 1`); a `language_code` that is blank but not `""` (`language_code must not be blank`). + - The body is `{text, model_id, voice_settings?: {stability}, language_code?}`. `language_code` `null` or `""` is not sent. `max_chars` is for pricing only and is never sent. +- **Answer:** the response body is the audio, as file `audio`. Its kind is the normalized `content-type`, or `audio/mpeg` when that header is absent. A `content-type` outside the audio family is kept as its lowercased base type, or as `application/octet-stream` when it is not a media type, so the check refuses it. `data: null`. Cost: `None`, since ElevenLabs reports characters, not dollars. +- **Check, in order:** + - not empty: `