feat(tasks): run SWE-bench rows with docker-* harnesses

.github/.licenserc.yaml

@@ -59,6 +59,9 @@ header:
     # CHANGELOG.md, which carries the header itself.
     - 'dokimi/changelog.d/[0-9]*.md'
     - 'dokimi/changelog.d/fragment_template.md.j2'
+    # Third-party eval files copied unchanged, under their own licenses
+    # (dokimi/tests/fixtures/task_formats/MANIFEST.yaml, LICENSES/).
+    - 'dokimi/tests/fixtures/task_formats/**'
 
   comment: never
 

dokimi/changelog.d/20261023_120000_napetrov_task-format-detect.md

@@ -0,0 +1,16 @@
+### Added
+
+- **`tasks inspect <path> [--json] [--out FILE]`** — detects the eval-task
+  format of a file, a task directory or a whole tree and reports its
+  structure: kind (executable, answer-only, trigger, localization), item
+  count, present and missing parts, lint problems and which dokimi backends
+  consume it today. Recognises Harbor, BenchFlow `task.md`, terminal-bench v1,
+  SWE-bench instances, Inspect AI samples (JSONL and CSV) and `@task` files
+  (scanned as text, never imported), OpenAI evals JSONL and registry specs,
+  skill-creator `evals.json` and trigger sets, skill-reach query sets,
+  promptfoo configs and test files, dokimi question sets and ExploreBench
+  tasks; anything else is reported as `unknown` with its files. A dataset
+  file a config names (an Inspect `FieldSpec` remap, an OpenAI evals
+  model-graded spec, promptfoo `tests: file://`) is read as that config says,
+  and a git-LFS pointer stub is reported as one. `--json` and `--out` write
+  the `task_inspect.v1` artifact. Detection only: no new format runs yet.

dokimi/changelog.d/20261024_100000_napetrov_run-swebench.md

@@ -0,0 +1,11 @@
+### Added
+
+- **`tasks run --swe-bench <rows.jsonl>`** runs SWE-bench rows (Lite,
+  Verified) with the `docker-*` harnesses: each row runs in its prebuilt
+  `swebench/sweb.eval.*` image, `docker-oracle` applies the gold patch, and the
+  verifier runs SWE-bench's eval script and grades the log with the repo's
+  SWE-bench parser (reward 1 only when every `FAIL_TO_PASS` and `PASS_TO_PASS`
+  test passes). `tasks inspect` now names `docker-*` as the consumer of a
+  SWE-bench file whose repos all have a test spec.
+  The rows run offline by default; `--all` skips the 8 rows in
+  `swebench.KNOWN_BAD` whose gold patch fails in the prebuilt image.

dokimi/changelog.d/20261024_120000_napetrov_task-network-policy.md

@@ -0,0 +1,15 @@
+### Added
+
+- **Per-phase network policy for `docker-*` runs** — build, solve and verify
+  each get `none`, `allowlist` (egress sidecar) or `public`. Defaults: build
+  public, solve and verify none. Set from task.toml (`[environment]`,
+  `[agent]`, `[verifier]` `network_mode` / `allowed_hosts`, Harbor's
+  `allow_internet`), or `dokimi tasks run --network PHASE=MODE[:hosts]`.
+  The effective plan and each phase's source land in `metrics.network`; a
+  skill's `metadata.network` is recorded as `metrics.network.skill`, not
+  granted.
+
+### Changed
+
+- **A task's `network_mode = "public"` now also reaches the oracle**, and the
+  verifier no longer gets the network unless something grants it.

dokimi/docs/architecture.md

@@ -70,7 +70,7 @@ points up:
 | Layer | Modules |
 | --- | --- |
 | front end | `cli`, `commands`, `__main__` (the argparse front end; no layer below imports it) |
-| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, and the checkers `config_check`, `schema_check`, `schema_compat` |
+| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, `task_formats`, and the checkers `config_check`, `schema_check`, `schema_compat` |
 | evaluation | `eval`, `personas`, `questions`, `treatments`, `plugins` |
 | components | `ingest`, `metrics`, `gate`, `mcp`, `skills`, `agent_profiles` |
 | foundation | `__init__` (the `dokimi` package itself), `_version`, `artifacts`, `records`, `llm`, `process`, `caching`, `config`, `registry`, `intents`, `defaults`, `console`, `utils`, `metrics/usage.py` |

dokimi/docs/coding-harnesses.md

@@ -78,6 +78,88 @@ refused (a few hundred bytes of nested aliases expand without bound), and
 frontmatter nested too deeply to parse is a parse error. YAML and TOML dates
 and times become ISO strings, and YAML sets sorted lists.
 
+### SWE-bench rows: `tasks run --swe-bench`
+
+`python cli.py tasks run --swe-bench <rows.jsonl> --tasks <instance_id>[,...]
+--harnesses docker-oracle,docker-claude` runs SWE-bench rows (Lite, Verified,
+or any file with the same fields) with the `docker-*` harnesses; `--all`
+runs every row. Each row is staged under `<output-dir>/swe-bench-tasks/<id>/`
+as a task the harness already runs, by `dokimi/harnesses/swebench.py`:
+
+- environment: the prebuilt SWE-bench instance image
+  (`swebench/sweb.eval.x86_64.<id with __ as _1776_>:latest`, or the row's own
+  `image_name`/`docker_image`), with the checkout moved to `/app` and `/testbed`
+  left as a link to it. Nothing is built but that one layer; the first run pulls
+  the image (about 4 GB, 1-2.5 GB of it unique per instance).
+- oracle (`docker-oracle`): the row's gold `patch`, applied as SWE-bench applies
+  a prediction (`git apply`, then `patch --fuzz=5`).
+- verifier: SWE-bench's eval script (reinstall the repo, reset and apply
+  `test_patch`, run the repo's test command on the files it touches), then
+  `swebench_grade.py`, which parses the log with the repo's SWE-bench parser:
+  reward 1 only when every `FAIL_TO_PASS` and `PASS_TO_PASS` test passes.
+
+The per-repo test commands and parsers are a table,
+`dokimi/harnesses/swebench_specs.json`, generated from the `swebench`
+package by `scripts/gen_swebench_specs.py`; dokimi does not depend on
+`swebench`. Rows from a repo outside the table (SWE-bench Multilingual,
+SWE-smith, SWE-Gym repos) are refused with the repo's name. The task sets no
+network, so the solve and the verifier run offline (see
+[Network policy](#network-policy)); the image already has the repo's
+dependencies, and `--network verify=public` gives the verifier upstream's
+network.
+
+`--all` skips the rows in `swebench.KNOWN_BAD`, whose gold patch fails in the
+prebuilt image: `psf__requests-863` and `-2317` (their tests fetch
+httpbin.org), and the six Harbor's SWE-bench adapter lists as failing under its
+oracle (harbor 6099b8c, `adapters/swebench/README.md`, Known Issues). Naming
+one in `--tasks` still runs it.
+
+### Other eval formats: `tasks inspect`
+
+`python cli.py tasks inspect <path> [--json] [--out FILE]` names the eval-task format of a
+file, a task directory, or every item under a tree, and reports its structure:
+kind, item count, which parts are present or missing (instruction,
+environment, oracle, verifier, metadata), lint problems, and what in dokimi
+can consume it today. It only reads: nothing is run or converted, a Python
+file is scanned as text and never imported, and a file that does not parse is
+a problem on its report, not an error. A path no format claims is reported as
+`unknown` with its files (or keys) and likely prompt and verifier files; when
+two formats claim a path about equally, the report lists the other as
+`ambiguous` with its evidence. `--json` prints and `--out` writes the report
+as a `task_inspect.v1` artifact (`schemas/task_inspect.v1.json`).
+
+A dataset file a config in the same tree names is read as that config says:
+an Inspect `@task` file's `json_dataset("x.jsonl", FieldSpec(input="question",
+target="answer"))` makes `x.jsonl` Inspect samples under those keys, an OpenAI
+evals registry spec makes its `samples_jsonl` OpenAI evals (with no `ideal`
+needed when the spec's class is model-graded), and promptfoo `tests:
+file://tests.csv` makes that file promptfoo tests. The report says which file
+named it. A file that is a git-LFS pointer stub (a clone without `git lfs
+pull`) is reported as `git-lfs-pointer`, with the format a config expects it
+to be when one names it. `tests/fixtures/task_formats/` holds a small corpus of
+real files in each format, with their sources in its `MANIFEST.yaml`.
+
+| Format | Recognised by | Kind | Consumed by |
+|---|---|---|---|
+| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` |
+| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` |
+| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | none yet |
+| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | `docker-*` via `tasks run --swe-bench` (Python SWE-bench repos) |
+| `inspect-samples` | JSON/JSONL/CSV rows with `input` and `target` (or the keys a `FieldSpec` names) | answer-only | none yet |
+| `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | none yet |
+| `openai-evals` | JSONL rows with `input` (chat messages) and `ideal` | answer-only | none yet |
+| `openai-evals-spec` | registry YAML with `class: evals.elsuite.*` run specs | answer-only | none yet |
+| `skill-creator-evals` | `evals.json` with `skill_name` and `evals[].prompt` | answer-only | none yet |
+| `skill-creator-trigger` | rows with `query` and `should_trigger` | trigger | none yet |
+| `skill-reach` | `QuerySet` (`queries[]` with `query_id`/`id`, `text`, `expected_skill`) | trigger | none yet |
+| `promptfoo` | `promptfooconfig*.yaml`/`.json`, or a config with `prompts`, `providers`/`targets` and `tests`/`scenarios`/`redteam` | answer-only | none yet |
+| `promptfoo-tests` | CSV with `__expected` columns, or rows of `{vars, assert}` | answer-only | none yet |
+| `dokimi-questions` | questions.v1/v2 JSON | answer-only (trigger with `expected_skill`) | questions track |
+| `dokimi-explore-task` | YAML with `id`, `query` and `references[].path` | localization | explore track |
+| `git-lfs-pointer` | a git-LFS spec v1 pointer stub | unknown | none (fetch it first) |
+
+`tasks list` and `tasks run` still read only the first two.
+
 The `docker-*` harnesses mount the oracle at `/solution` and `/oracle` and the
 verifier at `/tests` and `/verifier`, because a script written for one layout
 refers to its siblings by that layout's path. The directory's contents are
@@ -261,11 +343,11 @@ The result is in `metrics.isolation_probe` and `isolation_probe.json`:
       `docker-probe` reads `violated` on a passing
       cell, because the probe requests denied hosts (`example.com`) by design;
       judge a probe by `isolation_probe.json`, not by `egress.status`.
-- **`network_mode = "public"`.** It applies to the verifier only. The
-  container joins the default bridge after the agent exits. Every
-  uxlfoundation/skills task declares it, and none of their solutions or tests
-  uses the network, so a task whose solve needs a host must list it under
-  `environment.egress`.
+- **`network_mode = "public"`.** An `[environment]`-level `public` applies
+  to the oracle and the verifier, not to an agent: every uxlfoundation/skills
+  task declares it, and none of their solutions or tests uses the network. A
+  task whose agent needs a host lists it under `environment.egress`. See
+  [Network policy](#network-policy) for the other sources.
 - **Environment.** HOME is its own tmpfs, `/run/abhome`; the image's `/tmp`
   is left alone. The environment is:
     - the provider-credential allowlist;
@@ -474,6 +556,42 @@ A `command:` harness (including a `--command-template` that wraps `harbor`)
 keeps its own rule: a `reward.txt` / `reward.json` under the run dir, else a
 `reward: x` on its output, else exit code 0 is a pass.
 
+### Network policy
+
+Every `docker-*` run has a network per phase, recorded in
+`metrics.network` with the source of each phase's setting, so two runs can
+be checked for the same network before their verdicts are compared:
+
+| Phase | Default | What runs |
+|---|---|---|
+| `build` | `public` (host network, host proxy as build args) | `docker build` |
+| `solve` | `none` (an in-container agent still reaches its LLM through the sidecar) | the oracle, or the agent |
+| `verify` | `none` | `tests/test.sh` |
+
+The modes are Harbor's: `none` (Harbor's `no-network`), `public` (the
+default bridge plus the host's proxy variables), and `allowlist` (the listed
+`host[:port]` entries only, through an egress sidecar; the verifier's
+decisions land in `metrics.network.verify_egress`). Later sources win:
+
+1. The defaults above.
+2. The task: `[environment]` `network_mode` and `allowed_hosts` (or Harbor's
+   deprecated `allow_internet`), then `[agent]` and `[verifier]` overrides,
+   and `environment.egress` for an agent's solve. An unknown mode refuses
+   the cell.
+3. The operator: `dokimi tasks run --network PHASE=MODE[:host,...]`,
+   repeatable, with `all` for every phase. For example
+   `--network verify=none` grades a public task offline, and
+   `--network solve=allowlist:pypi.org,files.pythonhosted.org` lets every
+   arm install packages.
+
+A skill that needs the network at solve time declares it in its `SKILL.md`
+frontmatter as `metadata.network` (`allowlist:host,...` or `public`), with
+the reason for people in `compatibility`, the Agent Skills spec's field for
+environment needs. The run does not widen the skill arm for it: its baseline
+would then run on another network. `metrics.network.skill.missing` lists the
+declared hosts this run did not grant; pass them with `--network solve=...`
+so every arm gets them.
+
 ## BenchFlow Cells
 
 Skills are not bind-mounted: BenchFlow copies the `--skills-dir` root into the

dokimi/dokimi/__init__.py

@@ -22,7 +22,8 @@
   [`dashboard`][dokimi.dashboard] aggregates results across libraries and
   [`orchestrator`][dokimi.orchestrator] runs the one-command pipeline.
 - Executable tasks: [`harnesses`][dokimi.harnesses] runs coding agents on task
-  directories.
+  directories; [`task_formats`][dokimi.task_formats] names the format of other
+  eval-task files and directories (detection only).
 - [`context`][dokimi.context] reads what a run's context window held.
 - Content novelty: [`novelty`][dokimi.novelty]. Repositories as subjects:
   [`repos`][dokimi.repos]. Subject scorecards, one per run:

dokimi/dokimi/artifacts.py

@@ -10,8 +10,9 @@
 model), ``guidance_reach`` (L1 reach probes), ``repo_readiness`` (L0 claim
 checks), ``tool_preference`` (which of two equal tools a model picks, and
 what moves it), ``repo_environment`` (static inspection of what a
-repository offers an agent) and ``repo_budget`` (what a repository puts
-into an agent's context). This module stamps each with a
+repository offers an agent), ``repo_budget`` (what a repository puts
+into an agent's context) and ``task_inspect`` (which eval-task format each
+path under a tree is). This module stamps each with a
 ``schema_version`` and validates it against the JSON Schema in
 ``dokimi/schemas/`` on save and load, so fixtures and baselines stay
 forward-compatible and self-describing.
@@ -52,6 +53,7 @@
     "tool_preference": "tool_preference.v1",
     "repo_environment": "repo_environment.v1",
     "repo_budget": "repo_budget.v1",
+    "task_inspect": "task_inspect.v1",
 }
 
 # kind -> superseded versions still accepted on *load*, newest first. An artifact written under an