Repository navigation
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #212 +/- ##
==========================================
+ Coverage 92.01% 92.10% +0.08%
==========================================
Files 223 236 +13
Lines 27768 29147 +1379
Branches 4563 4759 +196
==========================================
+ Hits 25552 26845 +1293
- Misses 1570 1618 +48
- Partials 646 684 +38
🚀 New features to boost your workflow:
|
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi tasks inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
+ option `--out`
~ `dokimi tasks run` parameters changed:
+ added option `--network`Updated on every push; the |
Contributor
Author
|
docker-oracle on 25 sampled upstream tasks: 20/25 pass. Failures: 2 OOM under the 2g default memory (lm-eval-harness, predicate-pushdown-bench), 2 upstream rot (build-pmars pinned apt version gone, cron-broken-network example.com content), 1 task-side (build-tcc-qemu). docker-claude pilot with haiku: 3/3 pass, $0.006. |
…efault build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…azily A task.yaml task loads as a TaskSpec and runs under docker-oracle and docker-claude as terminal-bench's harness runs it (solution.sh at /oracle, run-tests.sh + tests/ at /tests, image user, TEST_DIR and compose env, pytest-summary verdict). Tasks one docker run cannot reproduce are refused with the reason. Harbor-backed harnesses and benchflow run a content-keyed Harbor export whose test.sh applies the same pytest-summary rule. Stacked on intel#210. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…om compose terminal-bench sets no memory limit; dokimi's 2 GB default OOM-killed add-benchmark-lm-eval-harness and predicate-pushdown-bench. A v1 task now reads compose deploy.resources.limits.memory when set, else 8192 MB (DOKIMI_TBENCH_V1_MEMORY_MB overrides), and the Harbor export writes it to task.toml. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
napetrov
force-pushed
the
feat/run-tbench-v1
branch
from
October 24, 2026 19:48
1260499 to
183a6f3
Compare
… solve default Rebased onto intel#218. The corpus manifest's terminal-bench hello-world now names its consumers (docker-*, harbor, benchflow), not none-yet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #218 (
feat/task-network-policy, itself on #210); review/merge after both. (AGENTS.md says PRs are not stacked; stacked here on request.)Summary
TaskSpec stays canonical; Harbor is only an export target, written lazily.
Evidence
tasks run --harnesses docker-oracleon a terminal-benchoriginal-tasks/dir: no task.toml/task.md, not loadable;tasks inspectconsumersnone-yet.After:
tests/test_tbench_v1.py(37 unit + 1 docker test gated byDOKIMI_DOCKER_TESTS=1): load/defaults, refusals, pytest-summary verdict incl. exit-code mismatch both ways, awk == Python verdict parity, export layout, cache keying, harbor harness runs the export. Full suite 7877 passed;make ci-local --offline0 failed.predicate-pushdown-benchpasses;add-benchmark-lm-eval-harnessruns its solution to the end, then fails at test collection (transformershas noAutoModelForVision2Seq; unpinned dependency, upstream rot).Scoring: pytest summary vs exit code
docker-oracle with the real
solution.shand with a no-op one (true), 60 sampled tasks (the earlier 25 + 35 random, seed 212), 8 CPUs / 8 GB per container: 118 runs verified (qemu-startup: image build fails, apt). The summary verdict andexit == 0agree on all 118. A static scan of all 241run-tests.shfinds the scripts where the exit code is not pytest's:pytest ...thenif [ $? -eq 0 ]; then echo <summary>; fi: anifwith a false test and noelseexits 0run_tests_in_same_shell, refused by docker-*)pytest ... | tee, thenecho "Pytest end."$?andexit $exit_codeharness.py_is_resolved(L536-542, all parsed tests PASSED; no summary means unresolved), andPytestParsermaps SKIPPED/XFAIL to passed and XPASS/ERROR to failed; the exit code is never read (d28711d).mappers/terminal_bench.pyREWARD_LOGGING_SUFFIX(L37-44,_EXIT_CODE=$?-> reward 1/0), added torun-tests.shbycopy_test_script_with_reward_logging(L357-361, L480) (harbor 6099b8c). It does not checkrun_tests_in_same_shellorparser_name.Merge Danger
Door: two-way
New format path; harbor/task-md code paths only touched by refactors (build context, step objects) covered by existing tests.
Blast Radius: docker-harnesses
Regressions would show as changed
docker build/cpargv for existing Harbor/task-md tasks.🤖 Generated with Claude Code