Skip to content

fix(merge): strict item schema so OpenAI Structured Outputs accept the merge assessment request - #507

Merged
mortondev merged 2 commits into
QuackbackIO:mainfrom
bemany:fix/merge-assessment-structured-output-schema
Sep 7, 2026
Merged

fix(merge): strict item schema so OpenAI Structured Outputs accept the merge assessment request#507
mortondev merged 2 commits into
QuackbackIO:mainfrom
bemany:fix/merge-assessment-structured-output-schema

Conversation

@bemany

@bemany bemany commented Sep 6, 2026

Copy link
Copy Markdown
Contributor

Fixes #505.

What is broken

assessMergeCandidates describes its response items with z.record(z.string(), z.unknown()). zod renders that as:

{"type":"object","propertyNames":{"type":"string"},"additionalProperties":{}}

OpenAI's Structured Outputs reject propertyNames, so the request is refused before the model runs:

400 Invalid schema for response_format 'structured_output':
In context=('properties', 'results', 'type', '0', 'items'), 'propertyNames' is not permitted.

Every merge check therefore fails against api.openai.com. Worse, checkPostForMergeCandidates only stamps mergeCheckedAt on success, so the sweep keeps retrying the same posts on every pass. Duplicate detection is permanently unavailable, and it is not model-specific — the schema is rejected, so switching chat models does not help.

The change

Give the items the shape the system prompt already demands.

I want to be explicit that the loose shape was deliberate, and the comment says why: it let one malformed item be skipped rather than failing the whole batch. That reasoning does not survive structured outputs — the provider guarantees the shape, so there is no malformed item left to skip. Everything else is kept:

  • results still has .catch([]), so a present-but-wrong-shaped top level degrades to "no assessments" instead of throwing.
  • The typeof guards in the filter loop stay untouched. They simply stop being load-bearing.
  • The comment is rewritten to record the new reason rather than deleted.

Testing

Added one regression test to the existing suite. It asserts that the schema handed to chat() emits no propertyNames, taken from the actual mockChat call rather than from an exported internal, so nothing new had to be made public.

  • apps/web merge-assessment suite: 13 passed (12 existing + 1 new)
  • Reverting only the service change makes exactly the new test fail, and the other 12 still pass — so the test targets this bug and nothing else
  • oxlint and prettier --check clean on both files

Verified in production as well: with this change, a deliberately duplicated post is found by embedding similarity and confirmed by the model (confidence 0.95), and the merge suggestion is created. Running against gpt-4.1-mini.

Note

Unrelated to this PR, but worth flagging for anyone testing on OpenAI directly: the max_tokens this service sends is rejected by the gpt-5 family, which wants max_completion_tokens. That is a separate matter and not touched here.

…e request

`assessMergeCandidates` described its response items with
`z.record(z.string(), z.unknown())`. zod renders that with a `propertyNames`
keyword, which OpenAI's Structured Outputs reject:

  400 Invalid schema for response_format 'structured_output':
  In context=('properties', 'results', 'type', '0', 'items'),
  'propertyNames' is not permitted.

The request is refused before the model runs, so every merge check fails and
no merge suggestion is ever created. `checkPostForMergeCandidates` then never
stamps `mergeCheckedAt`, so the sweep retries the same posts on every pass.

The loose item shape was deliberate — it let a single malformed item be
skipped instead of failing the whole batch. Under structured outputs that
tolerance is moot: the provider guarantees the shape. `results` keeps
`.catch([])`, so a present-but-wrong-shaped top level still degrades to
"no assessments" rather than failing the request, and the `typeof` guards in
the filter loop stay as they are.

Adds a regression test that asserts the schema handed to `chat()` emits no
`propertyNames`. It fails on the previous schema and passes on this one.

Fixes QuackbackIO#505

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@CLAassistant

CLAassistant commented Sep 6, 2026

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

@mortondev mortondev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good — this is a real bug, not a speculative schema tweak.

z.record() really does emit propertyNames, OpenAI Structured Outputs reject that keyword, and a 400 means mergeCheckedAt never gets stamped so the sweep retries forever. The strict item schema matches the prompt, the regression test would fail on the old schema, and I applied the two nits (shorter comment, JSDoc spacing) on the fork.

@mortondev
mortondev merged commit de5af85 into QuackbackIO:main Sep 7, 2026
11 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] Merge assessment always fails with 400 against OpenAI: z.record renders propertyNames, which Structured Outputs reject

3 participants