feat(tasks): run SWE-bench rows with docker-* harnesses by napetrov · Pull Request #213 · intel/dokimi

napetrov · 2026-10-24T17:12:18Z

Summary

Stacked on #210 (review only the top commit). Runs SWE-bench rows through dokimi's own TaskSpec and the local docker-* harnesses; Harbor stays an export target only.

rows.jsonl ──stage()──> TaskSpec(format="swe-bench")
                         ├ environment/Dockerfile  FROM swebench/sweb.eval.x86_64.<id>  (/testbed -> /app)
                         ├ solution/               gold patch, upstream apply chain
                         └ tests/                  upstream eval script + stdlib grade.py -> reward.txt
dokimi tasks run --swe-bench rows.jsonl --tasks <id> --harnesses docker-oracle,docker-claude

dokimi/harnesses/swebench.py: the loader. It uses prebuilt instance images, or a row's own image_name/docker_image, so SWE-smith/rebench rows carry their own image.
dokimi/harnesses/swebench_grade.py: a stdlib port of the SWE-bench log parsers and the RESOLVED_FULL rule (MIT attribution).
swebench_specs.json: generated from swebench==4.1.0 by scripts/gen_swebench_specs.py. dokimi does not depend on swebench.
tasks inspect now names docker-* as the consumer when every repo in the file has a spec.

Evidence

Oracle (docker-oracle; gold should score 1, an empty patch 0):

Instance	Gold	Empty
django-11099 / 11133	1 / 1	0 / 0
pytest-5227 / 7373	1 / 1	0 / 0
sympy-20590	1	0
requests-863	0 (live-httpbin P2P tests: dataset drift)	0
requests-2317	hung on httpbin, killed	-

docker-claude pilot (sonnet, 3 instances × 1 trial): 3/3 resolved in 84 s for $0.08.
Local checks: ruff and mypy clean, 7951 tests passed. A docker-gated test runs when DOKIMI_SWEBENCH_DOCKER=1.

Merge Danger

Low. The change is additive: a new --swe-bench flag and new modules, plus small edits to tasks.py, datasets.py, the docs and package-data.
The tasks run with network_mode=public, and the harness forwards the host proxy into the verifier, so tests that use the network can change outcome.
The harness container name dokimi-run-<task>-<pid> collides when one task runs twice in parallel in one process.
Must merge after feat(tasks): detect eval-task formats with tasks inspect #210.

🤖 Generated with Claude Code

@task

A probe registry (dokimi.task_formats) names the format of a file, a task directory or a tree, and reports kind, item count, present/missing parts, lint problems and which dokimi backends consume it today. Probes: harbor and task-md (wrapping harnesses.tasks), terminal-bench v1, SWE-bench, Inspect AI samples and @task files (text scan, never imported), OpenAI evals, skill-creator evals.json and trigger sets, skill-reach QuerySet, promptfoo and dokimi questions. Unknown paths get a best-effort structure report; malformed files are lint problems. Detection only: nothing new runs. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

`tasks inspect --json` now prints a stamped task_inspect.v1 artifact and `--out FILE` writes it through save_artifact, as `repo budget` does. Probes added or widened from runs over upstream checkouts: - a dataset file a config names is read as that config says: Inspect FieldSpec remaps, OpenAI evals registry specs (openai-evals-spec, with model-graded samples needing no ideal), promptfoo `tests: file://` (promptfoo-tests); - git-LFS pointer stubs are git-lfs-pointer, with the expected format when a config names them; - CSV datasets, promptfooconfig.<variant>.yaml and redteam configs, ExploreBench task YAML (dokimi-explore-task, localization); - a task.md with no BenchFlow front matter and no task part directories is no longer a task-md task. YAML aliases are reported as refused, not as invalid YAML. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

tests/fixtures/task_formats/ holds small upstream files of every format the probes know (intel/skills, skillsbench, terminal-bench, Inspect AI, openai/evals, promptfoo; licenses in LICENSES/) and generated ones where no public file exists (SWE-bench row, skill-creator evals.json, skill-reach QuerySet, and the output of dokimi's trigger-set writers). MANIFEST.yaml records each file's source, commit and license and what inspect must report; test_task_format_corpus.py inspects the corpus as one mixed tree, validates every --json report as task_inspect.v1, and reads back what to_skill_creator, to_reach and to_questions write. The fixture tree is kept out of pytest collection, ruff and the license header check: its test_*.py are the verifiers of copied tasks. Fixes found on the way: a task.md scan leaked the loader's UserWarning from the fingerprint, and a path that is neither file nor directory (/dev/null) crashed the tree scan. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

github-actions · 2026-10-24T17:19:37Z

CLI interface change detected

This PR changes dokimi's user-facing CLI surface (commands and/or options):

+ added command `dokimi tasks inspect`
    + argument `path`
    + option `--help/-h`
    + option `--json`
    + option `--out`
~ `dokimi tasks run` parameters changed:
    + added option `--network`
    + added option `--swe-bench`

Updated on every push; the cli-interface-change label is on while the diff is non-empty.

codecov-commenter · 2026-10-24T17:26:08Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.55429% with 119 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.99%. Comparing base (09be35b) to head (313ed32).

Files with missing lines	Patch %	Lines
dokimi/dokimi/harnesses/swebench_grade.py	70.06%	35 Missing and 12 partials ⚠️
dokimi/dokimi/task_formats/source.py	84.11%	12 Missing and 5 partials ⚠️
dokimi/dokimi/task_formats/detect.py	87.96%	8 Missing and 5 partials ⚠️
dokimi/dokimi/harnesses/network_policy.py	94.07%	5 Missing and 4 partials ⚠️
dokimi/dokimi/task_formats/dirs.py	91.08%	4 Missing and 5 partials ⚠️
dokimi/dokimi/task_formats/datasets.py	96.87%	4 Missing and 4 partials ⚠️
dokimi/dokimi/harnesses/swebench.py	93.75%	3 Missing and 4 partials ⚠️
dokimi/dokimi/commands/tasks.py	90.74%	2 Missing and 3 partials ⚠️
dokimi/dokimi/task_formats/configs.py	97.71%	1 Missing and 3 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #213      +/-   ##
==========================================
- Coverage   92.01%   91.99%   -0.03%     
==========================================
  Files         223      235      +12     
  Lines       27768    29130    +1362     
  Branches     4563     4781     +218     
==========================================
+ Hits        25552    26798    +1246     
- Misses       1570     1642      +72     
- Partials      646      690      +44

Files with missing lines	Coverage Δ
dokimi/dokimi/__init__.py	`100.00% <ø> (ø)`
dokimi/dokimi/artifacts.py	`95.12% <ø> (ø)`
dokimi/dokimi/harnesses/container_agent.py	`96.00% <100.00%> (+0.10%)`	⬆️
dokimi/dokimi/harnesses/docker_solver.py	`97.75% <100.00%> (+0.04%)`	⬆️
dokimi/dokimi/harnesses/runner.py	`99.17% <100.00%> (+0.04%)`	⬆️
dokimi/dokimi/harnesses/tasks.py	`95.69% <100.00%> (-0.32%)`	⬇️
dokimi/dokimi/task_formats/__init__.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/task_formats/links.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/task_formats/model.py	`100.00% <100.00%> (ø)`
dokimi/dokimi/task_formats/pointers.py	`100.00% <100.00%> (ø)`
... and 9 more

... and 1 file with indirect coverage changes

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…efault build, solve and verify each get none, allowlist (egress sidecar) or public. Defaults: build public, solve and verify none. Sources, later wins: the defaults, task.toml ([environment]/[agent]/[verifier] network_mode and allowed_hosts, Harbor's allow_internet, environment.egress), then dokimi tasks run --network PHASE=MODE[:hosts]. The plan and each phase's source land in metrics.network; a skill's SKILL.md metadata.network is recorded, not granted. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…swe-bench) Stage a SWE-bench row as a TaskSpec (format swe-bench) on its prebuilt instance image, with the upstream eval script and a stdlib grader that follows SWE-bench's parsers and resolution rule. The repo/version test table is generated from swebench==4.1.0; dokimi does not import it. Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…r check skips .json) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

…-bad rows requests-863/-2317 fetch httpbin.org live; the other six are the Harbor SWE-bench adapter's oracle failures (harbor 6099b8c README, Known Issues). Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov · 2026-10-24T18:40:01Z

Rebased on #218 (network policy). Changes in 313ed32:

Staged rows no longer set network_mode = "public", so solve and verify run offline. On the 7-row pilot the gold patch gives the same verdict offline and with --network solve=public --network verify=public (5 pass; requests-863/2317 fail in both).
--all skips swebench.KNOWN_BAD: requests-863/2317, whose tests fetch httpbin.org live, and the 6 oracle failures the Harbor SWE-bench adapter lists (harbor 6099b8c, adapters/swebench/README.md, Known Issues). Naming a row in --tasks still runs it.

🤖 Generated with Claude Code

napetrov added 3 commits October 23, 2026 22:18

github-actions Bot added the cli-interface-change Changes dokimi's user-facing CLI surface label Oct 24, 2026

napetrov force-pushed the feat/run-swebench branch 2 times, most recently from 6675b0f to 313ed32 Compare October 24, 2026 18:31

napetrov added 4 commits October 24, 2026 11:31

feat(harnesses): write network.json in every docker-* cell dir, error…

bab9fa9

…ed cells too Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

test(tasks): keep the SWE-bench fixture as a JSON list (license-heade…

0398866

…r check skips .json) Signed-off-by: Nikolay Petrov <nikolay.a.petrov@intel.com>

napetrov mentioned this pull request Oct 24, 2026

feat(harnesses): per-phase network policy for docker-* runs, off by default #218

Draft

Navigation Menu

Repository navigation

feat(tasks): run SWE-bench rows with docker-* harnesses - #213

feat(tasks): run SWE-bench rows with docker-* harnesses#213
napetrov wants to merge 8 commits into
intel:mainfrom
napetrov:feat/run-swebench

napetrov commented Oct 24, 2026

Uh oh!

github-actions Bot commented Oct 24, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 24, 2026 •

edited

Loading

Uh oh!

napetrov commented Oct 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

napetrov commented Oct 24, 2026

Summary

Evidence

Merge Danger

Uh oh!

github-actions Bot commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

CLI interface change detected

Uh oh!

codecov-commenter commented Oct 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

napetrov commented Oct 24, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

github-actions Bot commented Oct 24, 2026 •

edited

Loading

codecov-commenter commented Oct 24, 2026 •

edited

Loading