fix(docs-review): constrain the advisory review with the schema it already defines - #466
Open
Agnik47 wants to merge 1 commit into
Open
fix(docs-review): constrain the advisory review with the schema it already defines#466Agnik47 wants to merge 1 commit into
Agnik47 wants to merge 1 commit into
Conversation
…ready defines The advisory docs-sync review posts "🟠 Maintainer review suggested — low confidence / The automated review returned an invalid structured result" on pull requests whose documentation impact it never actually judged. That message comes from `validateGeminiReview` when the model's JSON parses but `verdict` is missing or not one of the three allowed values. It has no way to know the key: agentrhq#260 removed `response_format: json_schema` from the request to work around an HTTP 400, and the prompt names the verdict *values* in prose but never the object shape. `REVIEW_JSON_SCHEMA` has been exported and unused since. Every model response since has been a guess at the contract, and a wrong guess degrades silently to a low-confidence comment instead of a review. The request now carries the schema again, and the 400 fallback drops one capability per rung — `reasoning_effort` first, `response_format` only if the model also rejects that — so an unusable parameter no longer costs the schema. `maxItems` leaves the schema because strict structured output rejects array length keywords, which is the likely original 400; `validateGeminiReview` already caps findings at 5. The prompt states the exact keys so the schema-less rung produces a valid object too, and fenced-JSON parsing stays for it.
Contributor
🟢 No documentation gap found — medium confidenceThe automated review found no documentation gap in the supplied changes. This review is advisory and does not block merging. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Symptom
The advisory docs-sync review posts this on pull requests whose documentation impact it never actually judged:
Seen on #465 (run); the job itself exits 0 and logs
Documentation sync review updated for #465., so nothing fails loudly.Cause
That limitation string is only reachable from
validateGeminiReview(src/docs-sync-review.ts:396,408) — the model returned JSON that parsed, but was not a record, or itsverdictwas not one ofno_update_needed/review_suggested/likely_missing.The model has no way to know that key:
response_format: { type: 'json_schema', strict: true, schema: REVIEW_JSON_SCHEMA }from the request while working around an HTTP 400.REVIEW_JSON_SCHEMAhas been exported and referenced nowhere since.buildReviewPromptsnames the three verdict values in prose but never states the object shape or any field name.So the response shape is a guess, and a wrong guess degrades silently into a low-confidence comment rather than a review.
The likely original 400 is
maxItems: 5onfindings: strict structured output rejects array length keywords. The workaround dropped the whole schema rather than that one keyword.Fix
strict: true, asdocs_sync_review.reasoning_effortfirst,response_formatonly if the model also rejects that. An unusable parameter no longer costs the schema.maxItemsfrom the schema.validateGeminiReviewalready caps findings with.slice(0, 5), and the cap is now stated in the field description.No behavior change to routing, verdict normalization, comment rendering, or the deterministic path.
Tests
src/docs-sync-review-cli.test.ts:json_schema/strict: true/ the verdict enum;maxItemsorminItems;reasoning_effortbut keeps"json_schema";response_format, and the third rung still resolves;src/docs-sync-review.test.ts:verdict,summary,findings,suggestedPath,behaviorChange;REVIEW_JSON_SCHEMArequires exactly the keys the validator reads and stays inside the strict subset.npx vitest run --project unit src/docs-sync-review.test.ts src/docs-sync-review-cli.test.ts— 81 passed. The one failure in that file on my machine,reads only regular documentation files inside the repository root, is a pre-existing WindowsEPERM: symlinkand fails identically onmainwith this branch stashed.npm run typecheckandnpm run check:typed-error-lint— clean.Note on
maxItemsOpenAI's current structured-outputs page no longer publishes the unsupported-keyword list, and community reports say array length keywords may since have become supported. The fix does not depend on which is true: the keyword is redundant with the existing code-side cap, and the ladder recovers if a rung is rejected for any other reason.