Skip to content

fix(docs-review): constrain the advisory review with the schema it already defines - #466

Open
Agnik47 wants to merge 1 commit into
agentrhq:mainfrom
Agnik47:fix/docs-review-structured-output
Open

fix(docs-review): constrain the advisory review with the schema it already defines#466
Agnik47 wants to merge 1 commit into
agentrhq:mainfrom
Agnik47:fix/docs-review-structured-output

Conversation

@Agnik47

@Agnik47 Agnik47 commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Symptom

The advisory docs-sync review posts this on pull requests whose documentation impact it never actually judged:

🟠 Maintainer review suggested — low confidence
The automated review could not reach a fully supported conclusion.

  • The automated review returned an invalid structured result.

Seen on #465 (run); the job itself exits 0 and logs Documentation sync review updated for #465., so nothing fails loudly.

Cause

That limitation string is only reachable from validateGeminiReview (src/docs-sync-review.ts:396,408) — the model returned JSON that parsed, but was not a record, or its verdict was not one of no_update_needed / review_suggested / likely_missing.

The model has no way to know that key:

  • fix: parse fenced docs review json #260 removed response_format: { type: 'json_schema', strict: true, schema: REVIEW_JSON_SCHEMA } from the request while working around an HTTP 400. REVIEW_JSON_SCHEMA has been exported and referenced nowhere since.
  • buildReviewPrompts names the three verdict values in prose but never states the object shape or any field name.

So the response shape is a guess, and a wrong guess degrades silently into a low-confidence comment rather than a review.

The likely original 400 is maxItems: 5 on findings: strict structured output rejects array length keywords. The workaround dropped the whole schema rather than that one keyword.

Fix

  • Send the schema again, strict: true, as docs_sync_review.
  • Degrade one capability per rung on HTTP 400: reasoning_effort first, response_format only if the model also rejects that. An unusable parameter no longer costs the schema.
  • Drop maxItems from the schema. validateGeminiReview already caps findings with .slice(0, 5), and the cap is now stated in the field description.
  • State the exact keys in the prompt, so the schema-less rung produces a valid object too. Fenced-JSON parsing from fix: parse fenced docs review json #260 stays for that rung.

No behavior change to routing, verdict normalization, comment rendering, or the deterministic path.

Tests

src/docs-sync-review-cli.test.ts:

  • the request carries json_schema / strict: true / the verdict enum;
  • the wire body contains no maxItems or minItems;
  • a 400 on rung 1 drops reasoning_effort but keeps "json_schema";
  • a second 400 drops response_format, and the third rung still resolves;
  • a model that 400s on every rung stops after 3 calls and throws the bounded error.

src/docs-sync-review.test.ts:

  • the prompt names verdict, summary, findings, suggestedPath, behaviorChange;
  • REVIEW_JSON_SCHEMA requires exactly the keys the validator reads and stays inside the strict subset.

npx vitest run --project unit src/docs-sync-review.test.ts src/docs-sync-review-cli.test.ts — 81 passed. The one failure in that file on my machine, reads only regular documentation files inside the repository root, is a pre-existing Windows EPERM: symlink and fails identically on main with this branch stashed.

npm run typecheck and npm run check:typed-error-lint — clean.

Note on maxItems

OpenAI's current structured-outputs page no longer publishes the unsupported-keyword list, and community reports say array length keywords may since have become supported. The fix does not depend on which is true: the keyword is redundant with the existing code-side cap, and the ladder recovers if a rung is rejected for any other reason.

…ready defines

The advisory docs-sync review posts "🟠 Maintainer review suggested — low
confidence / The automated review returned an invalid structured result" on
pull requests whose documentation impact it never actually judged.

That message comes from `validateGeminiReview` when the model's JSON parses but
`verdict` is missing or not one of the three allowed values. It has no way to
know the key: agentrhq#260 removed `response_format: json_schema` from the request to
work around an HTTP 400, and the prompt names the verdict *values* in prose but
never the object shape. `REVIEW_JSON_SCHEMA` has been exported and unused since.
Every model response since has been a guess at the contract, and a wrong guess
degrades silently to a low-confidence comment instead of a review.

The request now carries the schema again, and the 400 fallback drops one
capability per rung — `reasoning_effort` first, `response_format` only if the
model also rejects that — so an unusable parameter no longer costs the schema.
`maxItems` leaves the schema because strict structured output rejects array
length keywords, which is the likely original 400; `validateGeminiReview`
already caps findings at 5. The prompt states the exact keys so the schema-less
rung produces a valid object too, and fenced-JSON parsing stays for it.
@github-actions

Copy link
Copy Markdown
Contributor

🟢 No documentation gap found — medium confidence

The automated review found no documentation gap in the supplied changes.

This review is advisory and does not block merging.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant