Repository navigation
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi tasks inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
+ option `--out`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #210 +/- ##
==========================================
+ Coverage 92.01% 92.07% +0.05%
==========================================
Files 223 232 +9
Lines 27768 28658 +890
Branches 4563 4686 +123
==========================================
+ Hits 25552 26388 +836
- Misses 1570 1600 +30
- Partials 646 670 +24
🚀 New features to boost your workflow:
|
napetrov
marked this pull request as ready for review
October 24, 2026 15:57
napetrov
requested review from
antonwolfy,
jharlow-intel,
vchamarthi and
xaleryb
as code owners
October 24, 2026 15:57
This was referenced Oct 24, 2026
`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 24, 2026
…azily A task.yaml task loads as a TaskSpec and runs under docker-oracle and docker-claude as terminal-bench's harness runs it (solution.sh at /oracle, run-tests.sh + tests/ at /tests, image user, TEST_DIR and compose env, pytest-summary verdict). Tasks one docker run cannot reproduce are refused with the reason. Harbor-backed harnesses and benchflow run a content-keyed Harbor export whose test.sh applies the same pytest-summary rule. Stacked on intel#210. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
New
dokimi tasks inspect <path> [--json]. It works out which eval-task format a file, task directory or tree is in, then reports the kind, the item count, which parts are present or missing (instruction / environment / oracle / verifier / metadata), lint problems, and which dokimi backends can consume it today. This PR only detects formats. It doesn't run or convert any new format, andtasks list/tasks run/catalogbehave as before.harbor_task_problemis None), docker-*sandbox=)expected_skill)load_questionsaccepts it)I took field names from upstream source, not from memory: SWE-bench
swebench/types.py, Inspectdataset/_dataset.py(Sample/FieldSpec), terminal-benchoriginal-tasks/*(241 task.yaml keys tallied), skill-creatorreferences/schemas.mdandSKILL.md, skill-reachmodels.py/queries.py(including the legacyid->query_idmigration and the out_of_scope rule), the promptfoo examples (149 configs tallied), and openai/evalsdocs/build-eval.md.Evidence
tasks inspectdoesn't exist, and nothing reads the trigger sets that dokimi writes.After: a run over real upstream samples:
tests/test_task_format_detect.py: 37 tests. They cover every format, ambiguity, unknown, malformed JSON/JSONL/YAML/TOML/YAML-alias inputs (each becomes a problem, not a crash), an Inspect file that would write a marker if imported (the marker is never written), a mixed tree, and a regression check thatdiscover_tasksis unchanged.make ci-local: tests (7840 passed), mypy, ruff, growth, import contract, coverage, mkdocs and cli-surface all pass. pip-audit fails on the setuptools GHSA-xmj3-9gf7-h52w advisory; that failure is on main too and isn't caused by this PR.Merge Danger
Door: two-way. It adds a new command and a new package and changes no existing behaviour or persisted format. The
--jsonoutput isn't a versioned artifact yet.Blast Radius: additive
🤖 Generated with Claude Code