Repository navigation
Conversation
A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…efault build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
github-actions
Bot
added
the
cli-interface-change
CLI interface change detectedThis PR changes dokimi's user-facing CLI surface (commands and/or options): + added command `dokimi tasks inspect`
+ argument `path`
+ option `--help/-h`
+ option `--json`
+ option `--out`
~ `dokimi tasks run` parameters changed:
+ added option `--network`Updated on every push; the |
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #218 +/- ##
==========================================
+ Coverage 92.01% 92.09% +0.07%
==========================================
Files 223 233 +10
Lines 27768 28837 +1069
Branches 4563 4714 +151
==========================================
+ Hits 25552 26557 +1005
- Misses 1570 1606 +36
- Partials 646 674 +28
🚀 New features to boost your workflow:
|
napetrov
added a commit
to napetrov/dokimi
that referenced
this pull request
Oct 24, 2026
… solve default Rebased onto intel#218. The corpus manifest's terminal-bench hello-world now names its consumers (docker-*, harbor, benchflow), not none-yet. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #210 (
feat/task-format-detect); review/merge after it. #213 (SWE-bench) is rebased on this branch.Summary
Every
docker-*cell gets a network per phase, recorded with where it came from:Skills that need the network declare it in
SKILL.mdasmetadata.network: "allowlist:pypi.org,..."(reason incompatibility, the Agent Skills spec field). It is recorded asmetrics.network.skill.missing, not granted. Widening only the skill arm would put its baseline on another network.Evidence
Unit tests:
tests/test_network_policy.py(22) cover resolution order, Harbor fields, bad--networkspecs, and docker argv order (disconnect < solve.sh < connect < test.sh). They also check that proxy env reaches only the public step, that the verifier gets its own sidecar, and that an errored cell keepsnetwork.json. Full suite: 7954 passed.make ci-local: all green except pip-audit (filelock / setuptools CVEs in the lock, also on main).Real runs with docker-oracle on this host (full tables in
scratch/netpolicy/RESULTS.md):verify=publicverify=allowlist:pypi.orgAllowlist on real tasks:
reward-kit-exampleandhello-alpinepass withverify=allowlist:<7-8 HTTPS hosts>and fail withpypi.orgalone. Each time,verify_egress.deniednamed the missing hosts.Merge Danger
Door: two-way
Behaviour change: a task with
[environment] network_mode = "public"now also gives the oracle the network, and a verifier that nothing grants a network runs fully offline (--network none, where before it was an internal network). Undeclared Harbor tasks that install their test deps at verify time score 0 until run with--network verify=public. That was already true before; Harbor's own default is public.Blast Radius: docker-*
Only
docker-*harnesses;--networkwith any other harness is refused. Plain HTTP (apt over :80) through an allowlist still fails behind an upstream proxy (an existing egress_proxy rule).🤖 Generated with Claude Code