feat(llma): add typed offline evaluations SDK - #1005
Open
Radu-Raicea wants to merge 2 commits into
Open
Radu-Raicea wants to merge 2 commits into
Radu-Raicea wants to merge 2 commits into
Conversation
Contributor
posthog-python Compliance ReportDate: 2026-10-02T17:09:24.431900+00:00 ✅ All Tests Passed!121/121 tests passed Capture_V1 Tests✅ 95/95 tests passed View Details
Capture_Ai Tests✅ 5/5 tests passed View Details
Feature_Flags Tests✅ 17/17 tests passed View Details
Feature_Flags_Local_Evaluation Tests✅ 4/4 tests passed View Details
|
Contributor
|
[Medium risk] Adds a new offline evaluations SDK module. The PR appears safe to merge; the remaining unchecked scorer-version return type is non-blocking. Reviews (2) · Last reviewed commit: "fix(llma): reject integer scores that lo..." |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Comment on lines
+367
to
+372
| def create_version( | ||
| self, | ||
| scorer_id: str | UUID, | ||
| *, | ||
| config: _ConfigT, | ||
| base_version: int | None = None, |
Contributor
There was a problem hiding this comment.
create_version() infers its return type from the supplied config without checking the target scorer’s kind. Passing a numeric config for a boolean scorer can leave callers with a value typed as Scorer[NumericScorerConfig] even if the operation is rejected or returns a boolean scorer. The async method has the same issue. Tie the config type to the scorer kind instead of relying on a cast.
Prompt To Fix With AI
This is a comment left during a code review.
Path: posthog/ai/evaluations/_scorers.py
Line: 367-372
Comment:
**Version return type is unchecked** `create_version()` infers its return type from the supplied config without checking the target scorer’s kind. Passing a numeric config for a boolean scorer can leave callers with a value typed as `Scorer[NumericScorerConfig]` even if the operation is rejected or returns a boolean scorer. The async method has the same issue. Tie the config type to the scorer kind instead of relying on a cast.
---
For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
💡 Motivation and Context
Add
posthog.ai.evaluationsso Python callers can upload offline evaluation results and manage experiments through typed synchronous and asynchronous clients. Includes acknowledged bulk uploads, stable identities for safe retries and recovery, and explicit completion/failure.Scorer management includes typed boolean, numeric, and categorical configurations with polarity, metadata updates, immutable version creation, and programmatic version-ID discovery. Includes usage examples and a minor changeset.
💚 How did you test it?
📝 Checklist
If releasing new changes
.sampo/changesets/.🤖 Agent context
Autonomy: Human-driven (agent-assisted)
Implemented with Codex and collaborating agents using shell/Python tooling, Git, and the GitHub CLI. Local Codex session.
The API follows the current backend contract. Configuration types expose supported fields; scorer mutations avoid automatic retries, while replay-safe offline operations retain identities and content.