Skip to content

test(evals): move repository metadata out of criteria.json - #92

Merged
martinfrancois merged 1 commit into
mainfrom
test/criteria-sidecar
Sep 21, 2026
Merged

martinfrancois merged 1 commit into
mainfrom
test/criteria-sidecar

Conversation

@martinfrancois

Copy link
Copy Markdown
Owner

Summary

  • Problem: tessl eval lint (CLI 0.102 and later) rejects the metadata key that every criteria.json here carries and warns on the per-item category field. Publishing still works today, but the format is outside Tessl's schema. Tracked in the linked issue.
  • Why it matters: a future CLI could turn the lint into a publish-time rejection, and the repository's own validators depend on that metadata.
  • What changed: each scenario gets a criteria-meta.json with the metadata object and a categories map from checklist name to category; criteria.json keeps only context, type, and checklist items with name, description, max_score. validate_eval_criteria.py merges the two (and fails a criteria.json that still carries metadata or categories), classify_eval_result.py reads the sidecar, eval_impact.py fingerprints it, the test fixture writes it, and the agent docs describe the layout.
  • What did not change: no task text, criterion name, description, or max_score changed anywhere (pure metadata move, so no hosted rerun is needed per docs/agents/evals.md); runtime skill untouched.

Change Type

  • Skill behavior
  • Evals or scoring
  • Documentation
  • CI, release, or dependency automation
  • Repository metadata or contribution process
  • Other maintenance

Linked Issue

User-Visible Behavior

None. The published scenarios carry one extra small JSON file.

Validation

Checks most contributors can run:

  • python3 scripts/validate_eval_criteria.py evals evals-reference evals-regression -> passed
  • python3 scripts/test_validate_eval_criteria.py -> OK
  • python3 -m py_compile scripts/*.py -> passed
  • tessl eval lint ., tessl eval lint evals-reference, tessl eval lint evals-regression -> all scenarios valid, no errors, no warnings
  • scripts/pre_submit_gate.sh --plan-only -> runs

Tessl-authenticated checks:

  • Not needed: pure metadata move; the repository rule in docs/agents/evals.md exempts it from hosted reruns.

Human Verification

Diffed every criteria.json before and after the move: only the metadata object and category fields left; names, descriptions, and max_score are byte-identical.

Review Checklist

  • The change is scoped to the sections, skill files, evals, or workflows described above.
  • Validation that applies to this change is checked above, or any unavailable check is explained.
  • If evals or benchmark claims changed, the eval scenarios remain fair and do not leak answer keys, run IDs, or fixed score claims into runtime references.
  • New or moved eval scenarios follow the classifier recommendation, or the PR explains the maintainer-approved override. (None moved.)
  • Every retained eval scenario has a 100% with-context result, or any below-100 result is documented as blocking follow-up rather than classified/reportable coverage. (Unchanged scoring.)
  • PR title or squash title uses Conventional Commits.
  • Redaction checked: no tokens, private links, private eval artifacts, local host paths, or proprietary Java source.

AI Assistance (if used)

  • AI-assisted PR
  • I confirm I understand and reviewed the change

Closes #89. tessl eval lint rejects the metadata key and warns on the
per-item category field. criteria.json now holds only context, type,
and the checklist items Tessl's schema knows; criteria-meta.json next
to it carries this repository's metadata object and a categories map
from checklist name to category. The validators merge the two, the
classifier reads the sidecar, and a criteria.json that still carries
metadata or categories fails validation. Pure metadata move: no task,
criterion text, or max_score changed, so no hosted rerun is needed.
@martinfrancois
martinfrancois merged commit 2c979b3 into main Sep 21, 2026
7 of 8 checks passed
@martinfrancois
martinfrancois deleted the test/criteria-sidecar branch September 21, 2026 04:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

test(evals): criteria.json metadata fails the current tessl eval lint

1 participant