Repository navigation
docs(readme): back every claimed module with evidence, one grading rule #182
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,143 @@ | ||
| # The LLM interface, explained | ||
|
|
||
| > **Status:** Partial · 2026-11-04 | ||
| > **Evidence:** read-only reads of osionos at groot's `develop` pin `1c1d5d84` (the AI code is | ||
| > identical to osionos `develop` `d374c21f`, per `git diff --stat` on `services/`, `docs/ai` and | ||
| > `src/features/{ai,voice,recs}`). Each cited `file:line` was opened at that pin. #275 was read | ||
| > at its merge commit `1e757c3a`. No container was run and no question was asked for this page | ||
| > **To check:** the nginx buffering and timeouts (no test covers them); the provider-error | ||
| > patterns against live providers (`upstreamError.ts:14-19`); the groot pin bump that brings in | ||
| > #275 | ||
| > **Sources:** groot #155 · osionos #270, #275 · osionos `docs/adr/0007-upstream-error-codes.md`, | ||
| > `0008-quota-aware-failover.md` · [ai-stall-diagnosis-2026-10-29](ai-stall-diagnosis-2026-10-29.md) | ||
| > · [runbooks/osionos-ai](../runbooks/osionos-ai.md) | ||
|
|
||
| The README claims the **Complete LLM system interface** major (row 15) for text, streaming, | ||
| error handling and rate limiting. This page follows one question from the keyboard to the model | ||
| and back, so a reviewer can open each step and check that claim. Image generation exists, but it | ||
| is off by default and is not part of the claim. | ||
|
|
||
| Paths are in osionos (`apps/osionos/app`) unless prefixed `G` (groot). Paths under | ||
| `services/osionos-ai/` drop that prefix after the first hop that enters the service (hop 5). | ||
|
|
||
| --- | ||
|
|
||
| ## One question, start to finish | ||
|
|
||
| ```mermaid | ||
| sequenceDiagram | ||
| autonumber | ||
| actor U as User | ||
| participant C as Chat UI<br/>ChatComposer, useChatStore | ||
| participant P as Inference port<br/>aiInferencePort, runStream | ||
| participant N as nginx :4000<br/>/api/ai/ | ||
| participant S as osionos-ai :4300<br/>routes/runs.ts | ||
| participant R as Runner<br/>runner.ts + Redis | ||
| participant L as Provider<br/>gateway/resolve.ts | ||
| U->>C: Enter | ||
| C->>P: runTurn(prompt) | ||
| P->>N: POST /api/ai/runs (runId) | ||
| N->>S: proxy, buffering off | ||
| S->>R: start the run | ||
| R->>R: checkRunRateLimit (20/min per user) | ||
| alt over the limit | ||
| R-->>S: refused 429 rate_limited | ||
| S-->>P: 429 + Retry-After | ||
| P-->>C: countdown, then retry offered | ||
| else allowed | ||
| S-->>P: 200 text/event-stream | ||
| R->>L: stream call (failover chain) | ||
| loop each token | ||
| L-->>R: part | ||
| R-->>S: frame | ||
| S-->>P: id / event / data | ||
| P-->>C: append text | ||
| end | ||
| Note over S,P: ": ping" every 15 s keeps the line open | ||
| opt connection drops | ||
| P->>N: GET events, Last-Event-ID | ||
| N->>S: resume from the last id | ||
| end | ||
| S-->>P: done, or error with a stable code | ||
| P-->>C: final message or error notice | ||
| end | ||
| ``` | ||
|
|
||
| ## The twelve hops | ||
|
|
||
| | # | Hop | file:line | | ||
| |---|---|---| | ||
| | 1 | Send (Enter or the button) | `src/widgets/chat-shell/ui/ChatComposer.tsx:204`, `:305`; slash expansion `src/widgets/chat-shell/model/useSlashPalette.ts:159-169`; `ChatShell.tsx:169-179` | | ||
| | 2 | Chat store `send` and `runTurn` | `src/features/chat/model/useChatStore.ts:373`, `:204`, `:409`; error patch `:222-232`; stop `:434` | | ||
| | 3 | Inference port | `src/features/ai/model/aiInferencePort.ts:111-157` (runId `:113`, abort → cancel `:121`) | | ||
| | 3b | Stream client and resume | `src/features/ai/model/runStream.ts:77-135`: 3 resumes after 500, 1000 and 2000 ms (`:34`, `:124`); dedup + `Last-Event-ID` (`:89-95`); 410 → `run_expired` (`:115`); a 4xx is never resumed (`:109`) | | ||
| | 3c | Transport | `src/features/ai/api/aiClient.ts:118` (`/api/ai`), `:188-207`, `:330` start, `:350` resume, `:358` cancel | | ||
| | 4 | nginx | `G infrastructure/tls/nginx.conf:178-200`: upstream `osionos-ai:4300` `:179`, `proxy_buffering off` `:195`, `proxy_read_timeout 300s` `:197`. Public limit: `G infrastructure/tls/public.d/ai-public.conf:4` + `public.d/http-limits.conf:19` (2 r/s, burst 10) | | ||
| | 5 | Route table | `services/osionos-ai/src/routes/table.ts:148` (POST /runs), `:151` (events), `:157` (cancel), `:141` (/images) | | ||
| | 6 | `handleRuns` | `routes/runs.ts:178-293`: refusal + `Retry-After` `:276-280`; SSE opens `:283`; ping every 15 s `:118`; resume endpoint `:295-324` | | ||
| | 7 | Runner and per-user limit | `runs/runner.ts:205`; `checkRunRateLimit` `:260-264`; `orderByRest` `:388`; `reserveTarget` `:397` | | ||
| | 8 | Failover loop | `runs/upstream.ts:241`, `:248` (`maxRetries` 0 when a chain exists), `:274-289` move on, catch `:362-390`; `chain.ts:26`, `cooldown.ts:99,172`, `moveOn.ts:32-66`, `attempt.ts:49` (3 attempts) | | ||
| | 9 | Stream pump | `runs/streamPump.ts:232-252`; timeouts 30 s idle and 120 s total (`:25`, `:27`) | | ||
| | 10 | Provider | `gateway/resolve.ts:124-140` (anthropic, google, openai-compatible), `:176`; `gateway/platform.ts:28-32`; `connections/providers.ts:13,16` | | ||
| | 11 | SSE framing | `http.ts:108-129` (`x-accel-buffering: no`, id/event/data, `: ping`) | | ||
| | 12 | Frame dispatch → UI error | `src/features/chat/model/runEventDispatch.ts:204-214`; `ChatThread.tsx:278-285`; `src/features/chat/model/runErrorNotice.ts:42-59`; `src/features/chat/model/runErrors.ts:79-124` | | ||
|
|
||
| **Why the hops are shaped this way.** The browser never talks to a provider (hops 4–10 run on | ||
| the server), so a provider key never reaches the client. The answer streams over SSE instead | ||
| of one long response, and nginx must not buffer it (hop 4, hop 11). A dropped connection resumes | ||
| from the last event id instead of starting the question again (hop 3b). And every failure ends | ||
| as a stable code the UI can name (hop 12). | ||
|
|
||
| --- | ||
|
|
||
| ## Subject bullet → code → tests → CI | ||
|
|
||
| The CI jobs are in osionos `.github/workflows/ci.yml`. "AI service" is the job | ||
| `AI service (tsc + suite)` (`:314`), which runs `services/osionos-ai/scripts/ci.sh`. "Canvas" is | ||
| the `Canvas unit tests` step of `quality (root)` (`:285`). | ||
|
|
||
| | Subject bullet | Main files | Tests that prove it | CI job | What is not proven | | ||
| |---|---|---|---|---| | ||
| | Generate text | hops 1–12 | `tests/canvas/ai-chat-store.test.ts`, `ai-run-dispatch.test.ts`; `services/osionos-ai/tests/runs.test.ts`, `complete.test.ts` | Canvas; AI service | — | | ||
| | Streaming | `runStream.ts`, `aiInferencePort.ts`, `routes/runs.ts:283-324`, `http.ts:108-129` | `tests/canvas/ai-run-stream.test.ts`; `runs.test.ts`; `failover.test.ts` | Canvas; AI service | nginx buffering and timeouts; the ping and the idle/total timeouts on the happy path | | ||
| | Error handling | `runs/upstreamError.ts`, `upstream.ts:362-390`, `failover.ts`, `routes/complete.ts`; client `runErrors.ts:79-124` | `tests/upstreamError.test.ts`, `upstreamErrorRuns.test.ts`, `upstreamErrorLeak.test.ts`; `tests/canvas/ai-run-dispatch.test.ts` | AI service; Canvas | the UI shows one sentence for all the new codes (`runErrors.ts:72`, `:87-90`); the patterns were not checked against live providers | | ||
| | Rate limiting | `src/rateLimit.ts:62-84` (fixed 60 s window, Redis `INCR` + `EXPIRE NX`), `OSIONOS_AI_RUNS_PER_MINUTE` default 20 (`config.ts:48`), used at `runner.ts:260`; nginx `public_ai`; UI countdown `runErrorNotice.ts` | `tests/runRateLimit.test.ts`, `approvalRateLimit.test.ts`; `tests/canvas/ai-run-stream.test.ts` | AI service; Canvas | no direct unit test of `rateLimit.ts`; a fixed window lets up to twice the limit through at a boundary (its own caveat, `rateLimit.ts:6-9`) | | ||
| | Generate images (**not claimed**) | `routes/images.ts:141-282`, `images/engine.ts`, `promptGate.ts`, `outputCheck.ts`; client `imageRun.ts`, `AiImageJobView.tsx` | `tests/images.test.ts`, `imagesOff`, `imagesOutputCheck`, `imagesPromptGate`; `tests/canvas/ai-image-run.test.ts` | AI service; Canvas | off by default (404 unless `imagesEnabled`, `images.ts:144`); CI uses a mock engine only | | ||
|
|
||
| **No groot workflow builds `--profile ai`.** The tests above run in osionos CI against the | ||
| service alone. Nothing in groot CI starts the composed stack with the AI profile. | ||
|
|
||
| --- | ||
|
|
||
| ## What #270 and #275 added | ||
|
|
||
| **osionos #270: stable codes for provider failures** (merged 2026-11-03, on develop as | ||
| `553f2afc`, **in** groot's pin). Before it, every provider failure reached the browser as | ||
| `upstream_error`, so a 429, a daily cap and an oversized prompt looked the same. | ||
|
|
||
| - `runs/upstreamError.ts` maps a failure to `rate_limited`, `quota_exhausted`, | ||
| `provider_unavailable`, `context_too_long` or `upstream_error`. The error frame carries the | ||
| code, and `ai_runs.error` records `code[:status[:type]]`. | ||
| - Provider bodies and prompts are kept out of the service logs (`upstreamErrorLeak.test.ts`). | ||
| - The client reads the new codes (`runErrors.ts:88-90`), but still shows them as one sentence. | ||
| - Decision record: `docs/adr/0007-upstream-error-codes.md`. | ||
|
|
||
| **osionos #275: failover that knows the free quota is shared** (merged 2026-11-04 by LESdylan, | ||
| on develop as `1e757c3a`, **not yet in groot's pin `1c1d5d84`**). Every public user spends one | ||
| OpenRouter key on `:free` routes, and the SDK retried each step twice, so one question could | ||
| spend 15 calls. | ||
|
|
||
| - The SDK never retries (`maxRetries: 0`). The chain is the retry, and a run with one route | ||
| retries once on a 5xx or a network failure. | ||
| - `runs/quota.ts`: one Redis key rests every free route for everyone, for up to 300 s. While it | ||
| rests, a run whose routes are all free is refused with 429 and `Retry-After`. | ||
| - A run makes at most 7 provider calls (14 in research mode). | ||
|
Contributor
There was a problem hiding this comment. Qualify the provider-call bound with background summaries. The
 |
||
| - Tests: `tests/quota.test.ts`, `tests/quotaFailover.test.ts`. Decision record: | ||
| `docs/adr/0008-quota-aware-failover.md`. | ||
| - It does not touch `rateLimit.ts`, the per-user limit. | ||
|
|
||
| --- | ||
|
|
||
| See also: [README Modules](../../README.md#modules) · [02-modules](02-modules.md) · | ||
| [ai-stall-diagnosis-2026-10-29](ai-stall-diagnosis-2026-10-29.md) · | ||
| [runbooks/osionos-ai](../runbooks/osionos-ai.md) | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Refresh the pinned source line references.
Several references do not identify the cited code in the stated pin. For example,
runStream.tsdefinesstreamRunat line 71,aiInferencePort.tscreatesrunIdat line 104, andaiClient.tsdefinesstartRun,resumeRun, andcancelRunat lines 300, 319, and 326. The page points to different lines. (raw.githubusercontent.com)The route and CI references are also stale. SSE opens at
routes/runs.ts:276-278, the AI job is atci.yml:289-305, and Canvas tests are atci.yml:263-265. (raw.githubusercontent.com)Regenerate all
file:linereferences from the exact pinned tree before using this page as module evidence.Also applies to: 95-97