Skip to content

feat(evaluations): normalize text behaviors - #155

Merged
aknight-ld merged 2 commits into
mainfrom
AIC-3642-match-spec-content-hash
Oct 23, 2026
Merged

aknight-ld merged 2 commits into
mainfrom
AIC-3642-match-spec-content-hash

Conversation

@aknight-ld

@aknight-ld aknight-ld commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Normalizing both python and js SDKs to use a neutral stringify for consistency in content hashing, message history, and judge reasoning. This helps us avoid shoving a bunch of python based conventions into js (and other future builds) and keep our standards language agnostic.

BEGIN_COMMIT_OVERRIDE
feat(evaluations): hash eventId and contentHash as RFC 8785 canonical JSON, the same in every SDK
fix(client): show a judge an output of 0, false, {} or []
END_COMMIT_OVERRIDE


Note

Overview
Replaces Python-specific text formatting with language-neutral JSON rules so the JS SDK matches other SDKs on evaluations hashing and judge-facing text.

contentHash / eventId: canonicalJson is now RFC 8785 (JCS)—sorted keys, no whitespace, JSON.stringify number/string spelling (no pythonJsonNumber or ASCII-escaping non-ASCII). contentHashFor hashes that canonical form directly; reference hashes in tests change to align with the shared Python canonical_json tests.

Judges: Removes python-format.ts. buildMessageHistory and parseJudgeResponse use toolResultString (JSON for non-strings). Empty sections use isEmptyPart instead of Python truthiness, so 0 and false outputs are included in message_history. Offline scorer/judge validation errors use describeValue (JSON) instead of pythonRepr, so messages look like got true / got "0.5" rather than Python literals.

Docs and tests are updated (judge-scoring.test.ts added, python-format.test.ts removed).

Reviewed by Cursor Bugbot for commit e4eff49. Bugbot is set up for automated code reviews on this repo. Configure here.

@jeffdupont jeffdupont left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice cleanup: dropping python-format.ts is the right direction. canonicalJson matches py#209's canonical_json byte for byte on all 26 edge cases I tried (floats, integers past 2^53, lone surrogates, UTF-16 key order). The client's 1782 tests and tsc --noEmit pass locally.

The red "Type check" isn't from this PR. main has failed "Check API reports" since #135 merged without updating claude-agents' report. #160 fixes it. Once #160 merges, rebase and this goes green.

One thing before this merges:

feat(evaluations) hides a core judges change

buildMessageHistory is also used by the online judges in judges.ts (lines 191 and 413), so online judges now see an output of 0, false, {} or [] that they used to skip. (parseJudgeResponse is used only by evaluations, so the reasoning change stays experimental.) Under an evaluations scope, release-please lists the whole PR under Experimental, so that core fix never reaches the core changelog. Could you add this to the description?

BEGIN_COMMIT_OVERRIDE
feat(evaluations): hash eventId and contentHash as RFC 8785 canonical JSON, the same in every SDK
fix(client): show a judge an output of 0, false, {} or []
END_COMMIT_OVERRIDE

On the "rubric reads the same in every SDK" goal: JS is already the reference form here, so it needs no change. The gap is on the Python side (compact separators and number spelling), and I've left the details on py#209.

jeffdupont added a commit that referenced this pull request Oct 23, 2026
…160)

#135 added `cwd` and `settingSources` to `createClaudeAgentsHandler`,
`claudeAgents`, `claudeGraph` and `toClaudeAgents`, but did not update
`etc/ai-claude-agents.api.md`. Since it merged (6acd53d), the "Check API
reports" step has failed on `main` and on every open PR (for example
#155).

This commits the report `yarn api:update` generates, nothing else. `yarn
api:check` passes for all 13 packages locally.

The report shows `ClaudeAgentsHostOptions` as an `ae-forgotten-export`:
the type is used in public signatures but not exported. That matches how
the other reports already treat unexported types (18 such warnings on
main), so exporting it is left for a separate decision.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> **Overview**
> Regenerates the `@launchdarkly/ai-claude-agents` API Extractor report
so it matches the public surface after #135. The report now reflects
**`ClaudeAgentsHostOptions`** on `claudeAgents`, `claudeGraph`,
`createClaudeAgentsHandler`, and `toClaudeAgents`, and adds the
**`SettingSource`** import from `@anthropic-ai/claude-agent-sdk`.
> 
> This is a generated-only change (`yarn api:update`) to fix failing
**`yarn api:check`** on `main`; it does not alter runtime code. The
report includes the existing **`ae-forgotten-export`** warning for
`ClaudeAgentsHostOptions` (used in public signatures but not re-exported
from the package entry).
> 
> <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit
369505e. Bugbot is set up for automated
code reviews on this repo. Configure
[here](https://www.cursor.com/dashboard/bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@aknight-ld
aknight-ld force-pushed the AIC-3642-match-spec-content-hash branch from e4eff49 to 9a569d5 Compare October 23, 2026 19:09
@aknight-ld
aknight-ld merged commit c971649 into main Oct 23, 2026
12 checks passed
@aknight-ld
aknight-ld deleted the AIC-3642-match-spec-content-hash branch October 23, 2026 19:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants