Repository navigation
feat(evaluations): normalize text behaviors - #155
Conversation
jeffdupont
left a comment
There was a problem hiding this comment.
Nice cleanup: dropping python-format.ts is the right direction. canonicalJson matches py#209's canonical_json byte for byte on all 26 edge cases I tried (floats, integers past 2^53, lone surrogates, UTF-16 key order). The client's 1782 tests and tsc --noEmit pass locally.
The red "Type check" isn't from this PR. main has failed "Check API reports" since #135 merged without updating claude-agents' report. #160 fixes it. Once #160 merges, rebase and this goes green.
One thing before this merges:
feat(evaluations) hides a core judges change
buildMessageHistory is also used by the online judges in judges.ts (lines 191 and 413), so online judges now see an output of 0, false, {} or [] that they used to skip. (parseJudgeResponse is used only by evaluations, so the reasoning change stays experimental.) Under an evaluations scope, release-please lists the whole PR under Experimental, so that core fix never reaches the core changelog. Could you add this to the description?
BEGIN_COMMIT_OVERRIDE
feat(evaluations): hash eventId and contentHash as RFC 8785 canonical JSON, the same in every SDK
fix(client): show a judge an output of 0, false, {} or []
END_COMMIT_OVERRIDE
On the "rubric reads the same in every SDK" goal: JS is already the reference form here, so it needs no change. The gap is on the Python side (compact separators and number spelling), and I've left the details on py#209.
…160) #135 added `cwd` and `settingSources` to `createClaudeAgentsHandler`, `claudeAgents`, `claudeGraph` and `toClaudeAgents`, but did not update `etc/ai-claude-agents.api.md`. Since it merged (6acd53d), the "Check API reports" step has failed on `main` and on every open PR (for example #155). This commits the report `yarn api:update` generates, nothing else. `yarn api:check` passes for all 13 packages locally. The report shows `ClaudeAgentsHostOptions` as an `ae-forgotten-export`: the type is used in public signatures but not exported. That matches how the other reports already treat unexported types (18 such warnings on main), so exporting it is left for a separate decision. 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- CURSOR_SUMMARY --> --- > [!NOTE] > **Overview** > Regenerates the `@launchdarkly/ai-claude-agents` API Extractor report so it matches the public surface after #135. The report now reflects **`ClaudeAgentsHostOptions`** on `claudeAgents`, `claudeGraph`, `createClaudeAgentsHandler`, and `toClaudeAgents`, and adds the **`SettingSource`** import from `@anthropic-ai/claude-agent-sdk`. > > This is a generated-only change (`yarn api:update`) to fix failing **`yarn api:check`** on `main`; it does not alter runtime code. The report includes the existing **`ae-forgotten-export`** warning for `ClaudeAgentsHostOptions` (used in public signatures but not re-exported from the package entry). > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 369505e. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup> <!-- /CURSOR_SUMMARY --> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
e4eff49 to
9a569d5
Compare
Normalizing both python and js SDKs to use a neutral stringify for consistency in content hashing, message history, and judge reasoning. This helps us avoid shoving a bunch of python based conventions into js (and other future builds) and keep our standards language agnostic.
BEGIN_COMMIT_OVERRIDE
feat(evaluations): hash eventId and contentHash as RFC 8785 canonical JSON, the same in every SDK
fix(client): show a judge an output of 0, false, {} or []
END_COMMIT_OVERRIDE
Note
Overview
Replaces Python-specific text formatting with language-neutral JSON rules so the JS SDK matches other SDKs on evaluations hashing and judge-facing text.
contentHash/eventId:canonicalJsonis now RFC 8785 (JCS)—sorted keys, no whitespace,JSON.stringifynumber/string spelling (nopythonJsonNumberor ASCII-escaping non-ASCII).contentHashForhashes that canonical form directly; reference hashes in tests change to align with the shared Pythoncanonical_jsontests.Judges: Removes
python-format.ts.buildMessageHistoryandparseJudgeResponseusetoolResultString(JSON for non-strings). Empty sections useisEmptyPartinstead of Python truthiness, so0andfalseoutputs are included inmessage_history. Offline scorer/judge validation errors usedescribeValue(JSON) instead ofpythonRepr, so messages look likegot true/got "0.5"rather than Python literals.Docs and tests are updated (
judge-scoring.test.tsadded,python-format.test.tsremoved).Reviewed by Cursor Bugbot for commit e4eff49. Bugbot is set up for automated code reviews on this repo. Configure here.