diff --git a/CLAUDE.md b/CLAUDE.md index 6cd09e7..11e05c4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -13,7 +13,9 @@ Socrates is freeCodeCamp's hint API. Takes a camper's code, challenge descriptio ### `/hint` request flow -`apiKeyAuthHook` -> `rateLimiterHook` (scoped to `/hint` via an encapsulated plugin context) -> `sanitizeRequest` -> `buildPrompt` -> `generateFromGroq` -> `sanitizeHintOutput` -> response. +`rateLimiterHook` (plugin `preHandler`, scoped to `/hint` via an encapsulated plugin) -> `apiKeyAuthHook` (route `preHandler`) -> `sanitizeRequest` -> `buildPrompt` -> `generateFromGroq` -> `sanitizeHintOutput` -> response. + +Non-obvious: Fastify runs a plugin-level `preHandler` before a route's own `preHandler`, so the rate limiter fires **before** API-key auth — unauthenticated requests still consume the bucket. (`rateLimiterHook` = `instance.addHook` in `src/index.ts`; `apiKeyAuthHook` = route `preHandler:[…]` in `src/routes/hint.ts`.) ## Observability @@ -34,23 +36,14 @@ Socrates is freeCodeCamp's hint API. Takes a camper's code, challenge descriptio ### Build + deploy (CI) -- One workflow, `deploy.yaml` (`CD - Deploy - Socrates`), mirrors the main repo's `deploy-api.yml`: `workflow_dispatch` with NO environment input. **The branch you dispatch from is the environment** — `setup-jobs` reads `github.ref_name`: - - `prod-current` → `site_tld=org`, `tgt_env_short=prd` - - anything else (`prod-staging`, feature branches) → `site_tld=dev`, `tgt_env_short=stg` -- DX: ship staging = run `deploy.yaml` from `prod-staging`; ship prod = fast-forward `prod-staging` → `prod-current`, run from `prod-current`. +Operator walkthrough — scripts, release steps, source maps, required secrets — lives in [docs/README.md](./docs/README.md). Non-obvious invariants only here: + +- One workflow, `deploy.yaml` (`CD - Deploy - Socrates`): `workflow_dispatch`, NO environment input. **The branch you dispatch from is the environment** (`setup-jobs` reads `github.ref_name`): `prod-current` → `site_tld=org` + `tgt_env_short=prd`; anything else (`prod-staging`, feature branches) → `site_tld=dev` + `tgt_env_short=stg`. DX: ship staging = dispatch from `prod-staging`; ship prod = fast-forward `prod-staging` → `prod-current`, dispatch from `prod-current`. - Two deliberate vocabularies — DO NOT conflate: - - **`tgt_env_short` = `stg` / `prd`** → swarm stack name (`-socrates`), Gantry service filter, GitHub deployment environment, Tailscale CI hostname. + - **`tgt_env_short` = `stg` / `prd`** → swarm stack name (`-socrates`), Gantry service filter (`name=_svc-socrates`), GitHub deployment environment, Tailscale CI hostname. - **`site_tld` = `dev` / `org`** → DOCR image namespace AND the **Sentry environment** (= app's `SENTRY_ENVIRONMENT` = `DEPLOYMENT_ENV` in the swarm stack). **Sentry environments are `dev`/`org`, never `stg`/`prd`.** - `NODE_ENV` is `production` on BOTH stg and prd. Never the stg/prd discriminator — `site_tld` / `tgt_env_short` are. -- Deploy mechanism: Tailscale → Gantry webhook (`/hooks/run-gantry`, filter `name=_svc-socrates`); the swarm pulls the freshly built `:` + `:latest` image. - -### Sentry release + source maps (CI) - -- `getsentry/action-release` (SHA-pinned) runs in the build job before docker buildx: creates the release, injects debug IDs, uploads maps from `./dist`, associates commits, finalizes. `pnpm run build` runs on the runner only to emit maps; the Docker image rebuilds independently. -- The production image strips `*.map` from `dist/` in the Dockerfile build stage (V5). Source maps live only in Sentry, never in the running container. -- The deploy job runs `sentry-cli deploys new -e -r ` after the Gantry webhook (the only remaining `sentry-cli` use, installed ad-hoc). -- Release steps are gated on `SENTRY_AUTH_TOKEN` — fork PRs and tokenless dispatches stay green. -- Required secrets: `SENTRY_AUTH_TOKEN` (`project:releases` + `project:write` scopes), `SENTRY_ORG`, `SENTRY_PROJECT`. +- Prod image strips `*.map` from `dist/` in the Docker build stage (V5) — source maps live only in Sentry, never in the running container. Release plumbing is gated on `SENTRY_AUTH_TOKEN`, so fork PRs and tokenless dispatches stay green. ## Gotchas @@ -58,3 +51,4 @@ Socrates is freeCodeCamp's hint API. Takes a camper's code, challenge descriptio - **API key auth skipped outside production/staging.** `apiKeyAuthHook` short-circuits for any other `NODE_ENV`. - **Per-challenge model override.** `groqClient.ts` reads `GROQ_MODEL_` env vars via dynamic `process.env` lookup (not in `env.ts`), falling back to `GROQ_MODEL`. - **Groq has an in-memory circuit breaker + fallback hint.** After `MODEL_CB_FAILURES` failures the breaker opens for `MODEL_CB_COOLDOWN_MS`; `/hint` returns a canned fallback with `model_used: "fallback"`. Intentional — don't "fix" it by throwing. +- **Transient Groq failures MUST NOT escape as unhandled errors** (root cause of SOCRATES-API-3/-4). `makeGroqApiCall` throws `ModelUnavailableError` after exhausting retries on a _retryable_ error (timeout / 5xx / 429 / network); `/hint`'s catch maps `ModelUnavailableError` **and** retryable `GroqApiError` → the graceful fallback. Only _non-retryable_ Groq errors (auth, 4xx) surface to the error handler → Sentry `handled:no` (a real bug you want to see). Do NOT revert the exhausted-retry path to `throw finalError` — that reintroduces the unhandled-500 class. The exhausted-retry summary logs at `warn` (stdout only), never `error` (which ships to Sentry Logs). diff --git a/README.md b/README.md index c67eaff..6ab32e3 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ freeCodeCamp's hint API for coding challenges. When a camper is stuck, Socrates takes their code, the challenge description, and failing tests, then returns a hint that points them in the right direction without giving the answer away. -Built with Fastify and TypeScript. Uses Groq for inference (gpt-oss-20b by default). Supports HTML, CSS, JavaScript, and Python challenges, each with its own system prompt. +Built with Fastify and TypeScript. Uses Groq for inference (`openai/gpt-oss-20b` by default). Supports HTML, CSS, JavaScript, and Python challenges, each with its own system prompt. ## How it works @@ -38,13 +38,13 @@ Response: ```json { "hint": "What value does your function currently return when no explicit return statement is present?", - "model_used": "gpt-oss-20b" + "model_used": "openai/gpt-oss-20b" } ``` ### `GET /health` -Returns service status and uptime. Pass `?extended=true` to also check Redis and Groq connectivity. +Returns service status and uptime. Set `ENABLE_EXTENDED_HEALTH=true` to also check Redis and Groq connectivity. ### `GET /api-docs` @@ -52,13 +52,7 @@ Swagger UI. Only available in development (`NODE_ENV != production`). ### `GET /debug/sentry` -Sentry pipeline smoke test. Deliberately logs an error and throws a 500 so the captured exception and error log can be verified on the Sentry dashboard. Requires an `X-API-Key` header outside of development/testing. Events are tagged `smoke_test=true` so alerts can exclude them. - -```bash -curl -H "X-API-Key: $API_KEY" https:///debug/sentry -``` - -A `500` response is expected — that is the test. Confirm the event in Sentry → Issues and the log in the Logs dataset. +Sentry pipeline smoke test — deliberately logs an error and throws a `500` (the `500` is the expected result). Events are tagged `smoke_test=true` so alerts can exclude them. Gated behind `DEBUG_SOCRATES=true`; when off, all `/debug/*` routes 404. Also needs an `X-API-Key` header outside of development/testing. Full walkthrough in the [operator guide](./docs/README.md#smoke-test). ## Running locally diff --git a/src/lib/__tests__/groqClient.fallback.test.ts b/src/lib/__tests__/groqClient.fallback.test.ts new file mode 100644 index 0000000..098b1ed --- /dev/null +++ b/src/lib/__tests__/groqClient.fallback.test.ts @@ -0,0 +1,95 @@ +import axios, { AxiosError } from 'axios'; +import { beforeEach, describe, expect, it, vi } from 'vitest'; +import { GroqApiError } from '../../errors/groqApiError'; +import { ModelUnavailableError } from '../../errors/modelUnavailableError'; + +vi.mock('../../config/env', () => ({ + GROQ_API_KEY: 'test-key', + GROQ_MODEL: 'openai/gpt-oss-20b', + NODE_ENV: 'test', + LOG_LEVEL: 'silent', + BUILD_VERSION: 'test-build', + GROQ_TIMEOUT_MS: () => 30000, + GROQ_BACKOFF_BASE_MS: () => 1, + GROQ_MAX_RETRIES: () => 2, + GROQ_MAX_TOKENS: () => 1024, + GROQ_MAX_TOKENS_RETRY: () => 2048, + GROQ_EMPTY_RESPONSE_RETRIES: () => 1, + MODEL_CB_FAILURES: 100, + MODEL_CB_COOLDOWN_MS: 30000, +})); + +vi.mock('axios', async (importActual) => { + const actual = await importActual(); + return { + ...actual, + default: { ...actual.default, post: vi.fn() }, + }; +}); + +vi.mock('@sentry/node', () => ({ + startSpan: (_opts: unknown, cb: (span: unknown) => unknown) => + cb({ setAttribute: () => {}, setAttributes: () => {} }), +})); + +const silentLogger = { + info: vi.fn(), + warn: vi.fn(), + error: vi.fn(), + debug: vi.fn(), + fatal: vi.fn(), + trace: vi.fn(), +}; + +function timeoutError() { + return new AxiosError('timeout of 30000ms exceeded', 'ECONNABORTED'); +} + +function httpError(status: number) { + const config = { url: 'https://groq', method: 'post', headers: {} }; + return new AxiosError( + `Request failed with status code ${status}`, + 'ERR_BAD_RESPONSE', + config, + null, + { status, statusText: '', headers: {}, config, data: {} }, + ); +} + +async function call() { + const { generateFromGroq } = await import('../groqClient'); + return generateFromGroq({ + systemPrompt: 's', + userPrompt: 'u', + challengeType: 'javascript', + logger: silentLogger, + }); +} + +describe('generateFromGroq transient-failure handling', () => { + beforeEach(() => { + vi.mocked(axios.post).mockReset(); + for (const fn of Object.values(silentLogger)) fn.mockReset(); + }); + + it('throws ModelUnavailableError when all retries fail on a timeout', async () => { + vi.mocked(axios.post).mockRejectedValue(timeoutError()); + await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError); + }); + + it('throws ModelUnavailableError when all retries fail on a 503', async () => { + vi.mocked(axios.post).mockRejectedValue(httpError(503)); + await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError); + }); + + it('does not log at error level for a transient exhausted-retry failure', async () => { + vi.mocked(axios.post).mockRejectedValue(timeoutError()); + await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError); + expect(silentLogger.error).not.toHaveBeenCalled(); + }); + + it('still surfaces a non-retryable 401 as GroqApiError (genuine bug, Sentry-visible)', async () => { + vi.mocked(axios.post).mockRejectedValue(httpError(401)); + await expect(call()).rejects.toBeInstanceOf(GroqApiError); + }); +}); diff --git a/src/lib/groqClient.ts b/src/lib/groqClient.ts index 5ace6f2..a56b7e1 100644 --- a/src/lib/groqClient.ts +++ b/src/lib/groqClient.ts @@ -201,7 +201,7 @@ async function makeGroqApiCall( // All retries failed - handle circuit breaker const finalError = lastError || new Error('Groq generate failed'); handleAllRetriesFailed(finalError, cb, logger); - throw finalError; + throw new ModelUnavailableError('Groq unavailable after exhausting retries'); } export async function generateFromGroq(options: GroqRequestOptions): Promise { @@ -317,7 +317,7 @@ function handleAllRetriesFailed( cb: { failures: number; openedUntil: number }, logger: Logger, ): void { - logger.error({ err }, 'all groq retry attempts failed'); + logger.warn({ err }, 'all groq retry attempts failed'); // Increment failure count and possibly open circuit breaker cb.failures = (cb.failures || 0) + 1; diff --git a/src/routes/__tests__/hint.test.ts b/src/routes/__tests__/hint.test.ts index 624a7d3..9a880cb 100644 --- a/src/routes/__tests__/hint.test.ts +++ b/src/routes/__tests__/hint.test.ts @@ -41,6 +41,7 @@ vi.mock('../../lib/rateLimiter', () => ({ import Fastify, { type FastifyInstance } from 'fastify'; import { sharedSchemas } from '../../config/swagger'; +import { GroqApiError } from '../../errors/groqApiError'; import { ModelUnavailableError } from '../../errors/modelUnavailableError'; import { generateFromGroq } from '../../lib/groqClient'; import { errorHandler } from '../../middleware/errorHandler'; @@ -155,4 +156,35 @@ describe('POST /hint', () => { expect(body.hint).toContain('temporarily unavailable'); expect(body.model_used).toBe('fallback'); }); + + it('returns 200 with fallback hint when a retryable GroqApiError escapes', async () => { + vi.mocked(generateFromGroq).mockRejectedValueOnce( + new GroqApiError('Groq API error (503): unavailable', 503, true), + ); + + const response = await app.inject({ + method: 'POST', + url: '/hint', + payload: validBody, + }); + + expect(response.statusCode).toBe(200); + + const body = response.json(); + expect(body.model_used).toBe('fallback'); + }); + + it('surfaces a non-retryable GroqApiError as an error response (Sentry-visible)', async () => { + vi.mocked(generateFromGroq).mockRejectedValueOnce( + new GroqApiError('Groq API error (401): unauthorized', 401, false), + ); + + const response = await app.inject({ + method: 'POST', + url: '/hint', + payload: validBody, + }); + + expect(response.statusCode).toBe(401); + }); }); diff --git a/src/routes/hint.ts b/src/routes/hint.ts index 35c24e9..d11176b 100644 --- a/src/routes/hint.ts +++ b/src/routes/hint.ts @@ -1,4 +1,5 @@ import type { FastifyInstance, FastifyRequest } from 'fastify'; +import { GroqApiError } from '../errors/groqApiError'; import { InputValidationError } from '../errors/inputValidationError'; import { ModelUnavailableError } from '../errors/modelUnavailableError'; import { generateFromGroq } from '../lib/groqClient'; @@ -70,7 +71,10 @@ async function hintRoutes(fastify: FastifyInstance) { return reply.send({ hint: sanitizedHint, model_used: result.model_used }); } catch (err: unknown) { if (err instanceof InputValidationError) throw err; - if (err instanceof ModelUnavailableError) { + if ( + err instanceof ModelUnavailableError || + (err instanceof GroqApiError && err.isRetryable) + ) { // Provide a graceful fallback hint rather than failing hard const fallbackHint = 'The hint service is temporarily unavailable. Try validating syntax, checking nesting, and reading the failing test message.';