fix: serve fallback on transient Groq failures

CLAUDE.md

@@ -13,7 +13,9 @@ Socrates is freeCodeCamp's hint API. Takes a camper's code, challenge descriptio
 
 ### `/hint` request flow
 
-`apiKeyAuthHook` -> `rateLimiterHook` (scoped to `/hint` via an encapsulated plugin context) -> `sanitizeRequest` -> `buildPrompt` -> `generateFromGroq` -> `sanitizeHintOutput` -> response.
+`rateLimiterHook` (plugin `preHandler`, scoped to `/hint` via an encapsulated plugin) -> `apiKeyAuthHook` (route `preHandler`) -> `sanitizeRequest` -> `buildPrompt` -> `generateFromGroq` -> `sanitizeHintOutput` -> response.
+
+Non-obvious: Fastify runs a plugin-level `preHandler` before a route's own `preHandler`, so the rate limiter fires **before** API-key auth — unauthenticated requests still consume the bucket. (`rateLimiterHook` = `instance.addHook` in `src/index.ts`; `apiKeyAuthHook` = route `preHandler:[…]` in `src/routes/hint.ts`.)
 
 ## Observability
 
@@ -34,27 +36,19 @@ Socrates is freeCodeCamp's hint API. Takes a camper's code, challenge descriptio
 
 ### Build + deploy (CI)
 
-- One workflow, `deploy.yaml` (`CD - Deploy - Socrates`), mirrors the main repo's `deploy-api.yml`: `workflow_dispatch` with NO environment input. **The branch you dispatch from is the environment** — `setup-jobs` reads `github.ref_name`:
-  - `prod-current` → `site_tld=org`, `tgt_env_short=prd`
-  - anything else (`prod-staging`, feature branches) → `site_tld=dev`, `tgt_env_short=stg`
-- DX: ship staging = run `deploy.yaml` from `prod-staging`; ship prod = fast-forward `prod-staging` → `prod-current`, run from `prod-current`.
+Operator walkthrough — scripts, release steps, source maps, required secrets — lives in [docs/README.md](./docs/README.md). Non-obvious invariants only here:
+
+- One workflow, `deploy.yaml` (`CD - Deploy - Socrates`): `workflow_dispatch`, NO environment input. **The branch you dispatch from is the environment** (`setup-jobs` reads `github.ref_name`): `prod-current` → `site_tld=org` + `tgt_env_short=prd`; anything else (`prod-staging`, feature branches) → `site_tld=dev` + `tgt_env_short=stg`. DX: ship staging = dispatch from `prod-staging`; ship prod = fast-forward `prod-staging` → `prod-current`, dispatch from `prod-current`.
 - Two deliberate vocabularies — DO NOT conflate:
-  - **`tgt_env_short` = `stg` / `prd`** → swarm stack name (`<short>-socrates`), Gantry service filter, GitHub deployment environment, Tailscale CI hostname.
+  - **`tgt_env_short` = `stg` / `prd`** → swarm stack name (`<short>-socrates`), Gantry service filter (`name=<stack>_svc-socrates`), GitHub deployment environment, Tailscale CI hostname.
   - **`site_tld` = `dev` / `org`** → DOCR image namespace AND the **Sentry environment** (= app's `SENTRY_ENVIRONMENT` = `DEPLOYMENT_ENV` in the swarm stack). **Sentry environments are `dev`/`org`, never `stg`/`prd`.**
 - `NODE_ENV` is `production` on BOTH stg and prd. Never the stg/prd discriminator — `site_tld` / `tgt_env_short` are.
-- Deploy mechanism: Tailscale → Gantry webhook (`/hooks/run-gantry`, filter `name=<stack>_svc-socrates`); the swarm pulls the freshly built `:<tagname>` + `:latest` image.
-
-### Sentry release + source maps (CI)
-
-- `getsentry/action-release` (SHA-pinned) runs in the build job before docker buildx: creates the release, injects debug IDs, uploads maps from `./dist`, associates commits, finalizes. `pnpm run build` runs on the runner only to emit maps; the Docker image rebuilds independently.
-- The production image strips `*.map` from `dist/` in the Dockerfile build stage (V5). Source maps live only in Sentry, never in the running container.
-- The deploy job runs `sentry-cli deploys new -e <dev|org> -r <tagname>` after the Gantry webhook (the only remaining `sentry-cli` use, installed ad-hoc).
-- Release steps are gated on `SENTRY_AUTH_TOKEN` — fork PRs and tokenless dispatches stay green.
-- Required secrets: `SENTRY_AUTH_TOKEN` (`project:releases` + `project:write` scopes), `SENTRY_ORG`, `SENTRY_PROJECT`.
+- Prod image strips `*.map` from `dist/` in the Docker build stage (V5) — source maps live only in Sentry, never in the running container. Release plumbing is gated on `SENTRY_AUTH_TOKEN`, so fork PRs and tokenless dispatches stay green.
 
 ## Gotchas
 
 - **`pnpm run build` must copy the Lua script** (`cp -r src/lib/lua dist/lib/lua`). `src/lib/rateLimiter.ts` reads `token_bucket.lua` from disk at startup; dropping the copy silently breaks rate limiting in production.
 - **API key auth skipped outside production/staging.** `apiKeyAuthHook` short-circuits for any other `NODE_ENV`.
 - **Per-challenge model override.** `groqClient.ts` reads `GROQ_MODEL_<TYPE>` env vars via dynamic `process.env` lookup (not in `env.ts`), falling back to `GROQ_MODEL`.
 - **Groq has an in-memory circuit breaker + fallback hint.** After `MODEL_CB_FAILURES` failures the breaker opens for `MODEL_CB_COOLDOWN_MS`; `/hint` returns a canned fallback with `model_used: "fallback"`. Intentional — don't "fix" it by throwing.
+- **Transient Groq failures MUST NOT escape as unhandled errors** (root cause of SOCRATES-API-3/-4). `makeGroqApiCall` throws `ModelUnavailableError` after exhausting retries on a _retryable_ error (timeout / 5xx / 429 / network); `/hint`'s catch maps `ModelUnavailableError` **and** retryable `GroqApiError` → the graceful fallback. Only _non-retryable_ Groq errors (auth, 4xx) surface to the error handler → Sentry `handled:no` (a real bug you want to see). Do NOT revert the exhausted-retry path to `throw finalError` — that reintroduces the unhandled-500 class. The exhausted-retry summary logs at `warn` (stdout only), never `error` (which ships to Sentry Logs).

README.md

@@ -2,7 +2,7 @@
 
 freeCodeCamp's hint API for coding challenges. When a camper is stuck, Socrates takes their code, the challenge description, and failing tests, then returns a hint that points them in the right direction without giving the answer away.
 
-Built with Fastify and TypeScript. Uses Groq for inference (gpt-oss-20b by default). Supports HTML, CSS, JavaScript, and Python challenges, each with its own system prompt.
+Built with Fastify and TypeScript. Uses Groq for inference (`openai/gpt-oss-20b` by default). Supports HTML, CSS, JavaScript, and Python challenges, each with its own system prompt.
 
 ## How it works
 
@@ -38,27 +38,21 @@ Response:
 ```json
 {
   "hint": "What value does your function currently return when no explicit return statement is present?",
-  "model_used": "gpt-oss-20b"
+  "model_used": "openai/gpt-oss-20b"
 }
 ```
 
 ### `GET /health`
 
-Returns service status and uptime. Pass `?extended=true` to also check Redis and Groq connectivity.
+Returns service status and uptime. Set `ENABLE_EXTENDED_HEALTH=true` to also check Redis and Groq connectivity.
 
 ### `GET /api-docs`
 
 Swagger UI. Only available in development (`NODE_ENV != production`).
 
 ### `GET /debug/sentry`
 
-Sentry pipeline smoke test. Deliberately logs an error and throws a 500 so the captured exception and error log can be verified on the Sentry dashboard. Requires an `X-API-Key` header outside of development/testing. Events are tagged `smoke_test=true` so alerts can exclude them.
-
-```bash
-curl -H "X-API-Key: $API_KEY" https://<host>/debug/sentry
-```
-
-A `500` response is expected — that is the test. Confirm the event in Sentry → Issues and the log in the Logs dataset.
+Sentry pipeline smoke test — deliberately logs an error and throws a `500` (the `500` is the expected result). Events are tagged `smoke_test=true` so alerts can exclude them. Gated behind `DEBUG_SOCRATES=true`; when off, all `/debug/*` routes 404. Also needs an `X-API-Key` header outside of development/testing. Full walkthrough in the [operator guide](./docs/README.md#smoke-test).
 
 ## Running locally
 

src/lib/__tests__/groqClient.fallback.test.ts

@@ -0,0 +1,95 @@
+import axios, { AxiosError } from 'axios';
+import { beforeEach, describe, expect, it, vi } from 'vitest';
+import { GroqApiError } from '../../errors/groqApiError';
+import { ModelUnavailableError } from '../../errors/modelUnavailableError';
+
+vi.mock('../../config/env', () => ({
+  GROQ_API_KEY: 'test-key',
+  GROQ_MODEL: 'openai/gpt-oss-20b',
+  NODE_ENV: 'test',
+  LOG_LEVEL: 'silent',
+  BUILD_VERSION: 'test-build',
+  GROQ_TIMEOUT_MS: () => 30000,
+  GROQ_BACKOFF_BASE_MS: () => 1,
+  GROQ_MAX_RETRIES: () => 2,
+  GROQ_MAX_TOKENS: () => 1024,
+  GROQ_MAX_TOKENS_RETRY: () => 2048,
+  GROQ_EMPTY_RESPONSE_RETRIES: () => 1,
+  MODEL_CB_FAILURES: 100,
+  MODEL_CB_COOLDOWN_MS: 30000,
+}));
+
+vi.mock('axios', async (importActual) => {
+  const actual = await importActual<typeof import('axios')>();
+  return {
+    ...actual,
+    default: { ...actual.default, post: vi.fn() },
+  };
+});
+
+vi.mock('@sentry/node', () => ({
+  startSpan: (_opts: unknown, cb: (span: unknown) => unknown) =>
+    cb({ setAttribute: () => {}, setAttributes: () => {} }),
+}));
+
+const silentLogger = {
+  info: vi.fn(),
+  warn: vi.fn(),
+  error: vi.fn(),
+  debug: vi.fn(),
+  fatal: vi.fn(),
+  trace: vi.fn(),
+};
+
+function timeoutError() {
+  return new AxiosError('timeout of 30000ms exceeded', 'ECONNABORTED');
+}
+
+function httpError(status: number) {
+  const config = { url: 'https://groq', method: 'post', headers: {} };
+  return new AxiosError(
+    `Request failed with status code ${status}`,
+    'ERR_BAD_RESPONSE',
+    config,
+    null,
+    { status, statusText: '', headers: {}, config, data: {} },
+  );
+}
+
+async function call() {
+  const { generateFromGroq } = await import('../groqClient');
+  return generateFromGroq({
+    systemPrompt: 's',
+    userPrompt: 'u',
+    challengeType: 'javascript',
+    logger: silentLogger,
+  });
+}
+
+describe('generateFromGroq transient-failure handling', () => {
+  beforeEach(() => {
+    vi.mocked(axios.post).mockReset();
+    for (const fn of Object.values(silentLogger)) fn.mockReset();
+  });
+
+  it('throws ModelUnavailableError when all retries fail on a timeout', async () => {
+    vi.mocked(axios.post).mockRejectedValue(timeoutError());
+    await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError);
+  });
+
+  it('throws ModelUnavailableError when all retries fail on a 503', async () => {
+    vi.mocked(axios.post).mockRejectedValue(httpError(503));
+    await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError);
+  });
+
+  it('does not log at error level for a transient exhausted-retry failure', async () => {
+    vi.mocked(axios.post).mockRejectedValue(timeoutError());
+    await expect(call()).rejects.toBeInstanceOf(ModelUnavailableError);
+    expect(silentLogger.error).not.toHaveBeenCalled();
+  });
+
+  it('still surfaces a non-retryable 401 as GroqApiError (genuine bug, Sentry-visible)', async () => {
+    vi.mocked(axios.post).mockRejectedValue(httpError(401));
+    await expect(call()).rejects.toBeInstanceOf(GroqApiError);
+  });
+});

src/lib/groqClient.ts

@@ -201,7 +201,7 @@ async function makeGroqApiCall(
   // All retries failed - handle circuit breaker
   const finalError = lastError || new Error('Groq generate failed');
   handleAllRetriesFailed(finalError, cb, logger);
-  throw finalError;
+  throw new ModelUnavailableError('Groq unavailable after exhausting retries');
 }
 
 export async function generateFromGroq(options: GroqRequestOptions): Promise<GroqResponse> {
@@ -317,7 +317,7 @@ function handleAllRetriesFailed(
   cb: { failures: number; openedUntil: number },
   logger: Logger,
 ): void {
-  logger.error({ err }, 'all groq retry attempts failed');
+  logger.warn({ err }, 'all groq retry attempts failed');
 
   // Increment failure count and possibly open circuit breaker
   cb.failures = (cb.failures || 0) + 1;

src/routes/__tests__/hint.test.ts

@@ -41,6 +41,7 @@ vi.mock('../../lib/rateLimiter', () => ({
 
 import Fastify, { type FastifyInstance } from 'fastify';
 import { sharedSchemas } from '../../config/swagger';
+import { GroqApiError } from '../../errors/groqApiError';
 import { ModelUnavailableError } from '../../errors/modelUnavailableError';
 import { generateFromGroq } from '../../lib/groqClient';
 import { errorHandler } from '../../middleware/errorHandler';
@@ -155,4 +156,35 @@ describe('POST /hint', () => {
     expect(body.hint).toContain('temporarily unavailable');
     expect(body.model_used).toBe('fallback');
   });
+
+  it('returns 200 with fallback hint when a retryable GroqApiError escapes', async () => {
+    vi.mocked(generateFromGroq).mockRejectedValueOnce(
+      new GroqApiError('Groq API error (503): unavailable', 503, true),
+    );
+
+    const response = await app.inject({
+      method: 'POST',
+      url: '/hint',
+      payload: validBody,
+    });
+
+    expect(response.statusCode).toBe(200);
+
+    const body = response.json();
+    expect(body.model_used).toBe('fallback');
+  });
+
+  it('surfaces a non-retryable GroqApiError as an error response (Sentry-visible)', async () => {
+    vi.mocked(generateFromGroq).mockRejectedValueOnce(
+      new GroqApiError('Groq API error (401): unauthorized', 401, false),
+    );
+
+    const response = await app.inject({
+      method: 'POST',
+      url: '/hint',
+      payload: validBody,
+    });
+
+    expect(response.statusCode).toBe(401);
+  });
 });

src/routes/hint.ts

@@ -1,4 +1,5 @@
 import type { FastifyInstance, FastifyRequest } from 'fastify';
+import { GroqApiError } from '../errors/groqApiError';
 import { InputValidationError } from '../errors/inputValidationError';
 import { ModelUnavailableError } from '../errors/modelUnavailableError';
 import { generateFromGroq } from '../lib/groqClient';
@@ -70,7 +71,10 @@ async function hintRoutes(fastify: FastifyInstance) {
         return reply.send({ hint: sanitizedHint, model_used: result.model_used });
       } catch (err: unknown) {
         if (err instanceof InputValidationError) throw err;
-        if (err instanceof ModelUnavailableError) {
+        if (
+          err instanceof ModelUnavailableError ||
+          (err instanceof GroqApiError && err.isRetryable)
+        ) {
           // Provide a graceful fallback hint rather than failing hard
           const fallbackHint =
             'The hint service is temporarily unavailable. Try validating syntax, checking nesting, and reading the failing test message.';