Skip to content

review --json emits no result (and no parseError): the built-in reviewer path never produces schema-shaped output, unlike adversarial-review #679

Description

@arcobaleno64

Summary

In 1.0.6, codex-companion.mjs review --json returns a payload with no result key at all — and no rawOutput or parseError either, so a consumer cannot distinguish "structured output was not produced" from "structured output failed to parse". adversarial-review --json on the same repository, same working tree, returns result conforming to the plugin's own schemas/review-output.schema.json.

The two subcommands therefore emit two different --json shapes, and only one of them is machine-readable. Nothing in the payload marks which shape a consumer received.

I hit this building an automated benchmark that scores the plugin's reviewer against planted ground truth: the harness reads payload.result, and every case skipped with no result in payload while the companion itself exited 0 and looked healthy.

Repro

Minimal git repo, one file, one breaking change in the working tree (the change deletes module.exports):

node scripts/codex-companion.mjs review --scope working-tree --json --cwd /path/to/repo

Top-level keys of the emitted JSON:

review, target, threadId, sourceThreadId, codex

codex.stdout holds prose:

The change removes the module's only export, breaking all existing consumers that import the function.
Review comment:

  • [P1] Restore the module export — …

Same repo, same working tree, other subcommand:

node scripts/codex-companion.mjs adversarial-review --scope working-tree --json --cwd /path/to/repo

Top-level keys:

review, target, threadId, context, codex, result, rawOutput, parseError, reasoningSummary

with parseError: null and

{"verdict":"needs-attention","summary":"Do not ship: the change removes the module's public export…","findings":[{"severity":"high","title":"Function is no longer exported", }]}

Why the shapes differ

executeReviewRun branches on the review name (scripts/codex-companion.mjs, ~L357–L455):

  • reviewName === "Review"runAppServerReview(...), i.e. codex's built-in reviewer. No outputSchema is passed. The payload is assembled as { review, target, threadId, sourceThreadId, codex: { status, stderr, stdout: result.reviewText, reasoning } } — there is no result field in that object literal.
  • any other review name (e.g. Adversarial Review) → the prompt-template path, which passes outputSchema: readOutputSchema(REVIEW_SCHEMA) and runs the response through parseStructuredOutput, producing result / rawOutput / parseError.

So the absence is by construction, not a parse failure. parseStructuredOutput itself always returns parsed and parseError (null on failure) — it is simply never called on the native path.

Impact

  • schemas/review-output.schema.json ships with the plugin but the default review never satisfies it.
  • A --json consumer must special-case the review name to know whether findings are readable, and there is no field in the payload that says which path produced it.
  • The failure is silent in the direction that matters: no result, and no parseError explaining its absence.

This looks distinct from #496, which is about the schema path failing to conform on long multi-tool-call turns; here the schema path is not taken at all. #496 also describes review as sharing the runAppServerTurn path, which no longer matches 1.0.6.

Suggestions (any one would resolve it for a consumer)

  1. Map the built-in reviewer's findings into the same review-output.schema.json shape and emit them as result on the review path.
  2. Failing that, emit an explicit result: null plus a parseError-style reason (e.g. "built-in reviewer returns prose; use adversarial-review for structured output") so the absence is stated rather than inferred.
  3. And/or document the two --json shapes, keyed by subcommand, so consumers can branch deliberately.

Environment

  • codex plugin 1.0.6 (installed via the Claude Code marketplace)
  • codex-cli 0.149.0
  • Node v24.14.1
  • Windows 11 (10.0.26200)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions