Skip to content

A non-thinking ollama model makes every local reviewer AND the claim-comment gate fail, silently shrinking the claimable queue #5960

Description

@atomantic

ProblemrunToolFreeLocalCompletion in server/services/codeReview.js attaches reasoning_effort to the /v1/chat/completions body whenever a reviewer effort is configured. Ollama translates reasoning_effort into its thinking parameter, and a model that does not implement thinking rejects the whole request:

ollama API error 400: {"error":{"message":"\"<model>\" does not support thinking","type":"invalid_request_error",...}}

There is no retry and no capability probe, so a reviewer configured as ollama[<non-thinking model>]~effort=low is 100% unavailable — not degraded, not slower, just dead for every call. The effort suffix is what kills it; the same model answers fine with the field omitted.

Two user-visible consequences, both silent:

  1. Local code review never runs. /do:next and /claim record an inconclusive verdict every time. For an ~opt reviewer that is merely a lost review; for a required one it forces review-blocked, so the PR opens and then sits unmerged waiting on a reviewer that can never answer.
  2. The tool-free public-comment gate fails closed, and skips claimable work. runLocalClaimCommentReview goes through the same helper. The claim flow treats a failed gate as "skip this candidate for this run", so every issue that has any comment becomes unclaimable while the misconfiguration stands. Observed live on 2026-09-03: a --swarm=3 run had to skip Put the ungated client reviewer mirrors behind the same drift test that already guards the effort ladders #5704 and The FableLoom hosted-socket suite has one four-in-one test that asserts nothing about the utterance path, and no test for the host-only playback gate #5726 — both otherwise eligible, both with exactly one comment — and fall through to younger issues. Nothing in the run output said "your reviewer model is misconfigured"; it just looked like a shorter queue.

Evidenceserver/services/codeReview.js:283-289:

  const body = {
    model,
    messages,
    temperature: 0.2,
    stream: false,
    ...(resolvedEffort ? { reasoning_effort: resolvedEffort } : {}),
  }
  const response = await fetchWithTimeout(`${baseUrl}/v1/chat/completions`, { ... })

Reproduce against any non-thinking ollama model:

printf 'diff --git a/x.js b/x.js\n+const a = 1;\n' \
  | jq -Rs --arg backend ollama --arg model '<non-thinking-model>' --arg effort low \
      '{ backend: $backend, model: $model, effort: $effort, diff: . }' \
  | node server/scripts/run-local-code-review.mjs
# -> {"ok":false,...,"error":"ollama API error 400: ... does not support thinking"}

Drop --arg effort low and the same call succeeds, which isolates reasoning_effort as the sole cause.

Plan

  1. In runToolFreeLocalCompletion, on a 400 whose body matches /does not support thinking/i, retry once with reasoning_effort stripped. Return the successful result with effort: null so the caller can see the ladder was not honoured. Decision: retry-on-error rather than a capability probe — ollama exposes no reliable per-model "supports thinking" field, and a probe would add a round trip to every review.
  2. Remember the downgrade per backend:model in a module-level Map for the life of the process, so a multi-round review loop pays the 400 once instead of on every round.
  3. Surface the downgrade instead of hiding it: include a effortUnsupported: true flag on the result, and console.warn in the repo's single-line emoji style (⚠️ ollama model ${model} ignores reasoning_effort — retried without it).
  4. Make the failure legible where it is currently invisible: when the claim-comment gate returns ok:false, server/scripts/run-local-code-review.mjs already prints the error to stdout, but the claim flow only reads the schema fields. Have the bridge also write the error string to stderr so an unattended agent's log names the misconfiguration rather than showing an unexplained skipped candidate.
  5. Settings → Code Reviewers: when a model has been observed to reject thinking, show the effort selector as ignored for that model rather than silently offering a value that breaks every call.

Tests — in server/services/codeReview.test.js, mocking fetch:

  • retries without reasoning_effort when the backend rejects thinking — first call 400s with the real body text, second call succeeds; assert the second request body has no reasoning_effort key and the result is ok:true. Uniquely catches a regression that re-sends the field or gives up on the 400.
  • does not retry a 400 that is unrelated to thinking — a 400 with a different message returns ok:false after exactly one fetch. Uniquely catches an over-broad retry that would mask real request errors.
  • caches the downgrade for the same backend+model — two sequential calls issue three fetches, not four. Uniquely catches the per-round re-probe.
  • One case through runLocalClaimCommentReview proving the comment gate benefits from the same retry — it is the path whose failure silently drops claimable issues, and it does not share the code-review entry point's tests.

Acceptance criteria

  • ollama[<non-thinking model>]~effort=low returns findings instead of ok:false.
  • The claim-comment gate returns a schema-valid verdict for the same model.
  • A 400 that is not about thinking still fails, and is not retried.
  • The effort downgrade is visible in the process log and on the result object — never silent.
  • cd server && npm test -- codeReview passes.

Out of scope — the lmstudio branch (it does not reject the field this way; leave it on the shared path and let the retry cover it if it ever does), and any change to the effort ladders themselves in server/lib/cosValidation.js.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingeffort:mediumEffort: mediummodel:mediumModel size: mediumplanTracked by /do:replanplanner:opus-5Plan authored by the opus-5 model

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions