Repository navigation
feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track - #214
Open
napetrov wants to merge 6 commits into
Open
feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track#214napetrov wants to merge 6 commits into
napetrov wants to merge 6 commits into
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…oo sets on the answers track questions import translates the answer-only formats that tasks inspect detects into questions.v2 records with a format-neutral grading block. eval reference scores answers or arms output against those checks: deterministic checks need no model, judge checks use the checklist judge in both polarities. eval native-import reads Inspect json logs and promptfoo outputs so native verdicts sit next to dokimi's. Inspect @task files are parsed with ast, never imported or run. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi eval native-import`
+ argument `log`
+ option `--answers-out`
+ option `--eval-out`
+ option `--help/-h`
+ added command `dokimi eval reference`
+ option `--answers`
+ option `--concurrency`
+ option `--help/-h`
+ option `--judge-model`
+ option `--judge-provider`
+ option `--no-judge`
+ option `--output/-o`
+ option `--questions`
+ added command `dokimi questions import`
+ argument `path`
+ option `--format`
+ option `--grader`
+ option `--help/-h`
+ option `--output/-o`
+ option `--scorer`
+ option `--task`
+ added command `dokimi tasks inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
+ option `--out`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #214 +/- ##
==========================================
- Coverage 92.01% 91.62% -0.40%
==========================================
Files 223 242 +19
Lines 27768 29677 +1909
Branches 4563 4892 +329
==========================================
+ Hits 25552 27192 +1640
- Misses 1570 1734 +164
- Partials 646 751 +105
🚀 New features to boost your workflow:
|
Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
The skill an evals.json belongs to (skill_name, else the <skill>/evals/ directory) is each question's expected_skill, so the imported set also scores skill selection on a skill-catalog: arm. Without either, the import report notes it. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov
marked this pull request as ready for review
October 25, 2026 00:37
napetrov
requested review from
antonwolfy,
jharlow-intel,
vchamarthi and
xaleryb
as code owners
October 25, 2026 00:37
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #210 (
tasks inspect). Review only the top commit until #210 merges.Summary
This PR runs the answer-only formats that
tasks inspectdetects on the existing questions, answers and judge track. Each format gets an import adapter into one canonical record, so there is no parallel runner.What does not carry over is listed in
docs/answer-formats.md. The main points:multiple_choice/system_messageare not reproduced.fact, promptfoollm-rubric) all become the yes/no checklist judge.similar,costandlatencybecomeunsupported, which is reported and not counted.Evidence
Haiku 4.5 on Bedrock, at most 20 samples per set. Native tools were pinned: inspect-ai 0.3.283 and promptfoo 0.125.0. OpenAI evals graders were reproduced from upstream 8eac7a7d.
The deterministic graders agree on 100% of the same completions. Every disagreement is a judge verdict. A split verdict (0.5) fails here, which is stricter than Inspect
P.Tests:
tasks inspect born_first.jsonlgaveconsumers: [none-yet].[questions import].tests/test_answer_imports.py(20 tests) andtests/test_reference_checks.py(28 tests). Both are offline and use a fake judge.make ci-localpasses 28 checks and fails 1: pip-audit flags filelock and setuptools in uv.lock, which this PR does not change.Merge Danger
Door: two-way. The new commands and modules are additive. Two schemas are widened: questions.v2 gains
gradingandprompt_style, and task_inspect.v1 accepts thequestions importconsumer.Blast Radius: small. The answerer changes only for questions with
prompt_style: verbatim.answers generate --doc-source noneis a new opt-in flag.benchmarks/spec.v1.yamlmax_docsgoes from 31 to 32 for the new doc.Decisions for reviewers:
expected_skill?--doc-source noneacceptable as a flag?🤖 Generated with Claude Code