test(evals): move repository metadata out of criteria.json - #92
Merged
Merged
Conversation
Closes #89. tessl eval lint rejects the metadata key and warns on the per-item category field. criteria.json now holds only context, type, and the checklist items Tessl's schema knows; criteria-meta.json next to it carries this repository's metadata object and a categories map from checklist name to category. The validators merge the two, the classifier reads the sidecar, and a criteria.json that still carries metadata or categories fails validation. Pure metadata move: no task, criterion text, or max_score changed, so no hosted rerun is needed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tessl eval lint(CLI 0.102 and later) rejects themetadatakey that everycriteria.jsonhere carries and warns on the per-itemcategoryfield. Publishing still works today, but the format is outside Tessl's schema. Tracked in the linked issue.criteria-meta.jsonwith themetadataobject and acategoriesmap from checklist name to category;criteria.jsonkeeps onlycontext,type, and checklist items withname,description,max_score.validate_eval_criteria.pymerges the two (and fails acriteria.jsonthat still carries metadata or categories),classify_eval_result.pyreads the sidecar,eval_impact.pyfingerprints it, the test fixture writes it, and the agent docs describe the layout.max_scorechanged anywhere (pure metadata move, so no hosted rerun is needed perdocs/agents/evals.md); runtime skill untouched.Change Type
Linked Issue
User-Visible Behavior
None. The published scenarios carry one extra small JSON file.
Validation
Checks most contributors can run:
python3 scripts/validate_eval_criteria.py evals evals-reference evals-regression-> passedpython3 scripts/test_validate_eval_criteria.py-> OKpython3 -m py_compile scripts/*.py-> passedtessl eval lint .,tessl eval lint evals-reference,tessl eval lint evals-regression-> all scenarios valid, no errors, no warningsscripts/pre_submit_gate.sh --plan-only-> runsTessl-authenticated checks:
docs/agents/evals.mdexempts it from hosted reruns.Human Verification
Review Checklist
AI Assistance (if used)