feat(questions): run Inspect, OpenAI evals, skill-creator and promptfoo sets on the answers track

.github/.licenserc.yaml

@@ -59,6 +59,9 @@ header:
     # CHANGELOG.md, which carries the header itself.
     - 'dokimi/changelog.d/[0-9]*.md'
     - 'dokimi/changelog.d/fragment_template.md.j2'
+    # Third-party eval files copied unchanged, under their own licenses
+    # (dokimi/tests/fixtures/task_formats/MANIFEST.yaml, LICENSES/).
+    - 'dokimi/tests/fixtures/task_formats/**'
 
   comment: never
 

dokimi/benchmarks/spec.v1.yaml

@@ -76,11 +76,11 @@ future:
 # on the include/exclude globs, not a target: min_docs catches a glob that
 # matched almost nothing, max_docs catches one that escaped into the tree (8
 # markdown files live outside docs/). Measured selection at the time of
-# writing: 31 files. Adding a doc past the ceiling is meant to be a deliberate
+# writing: 32 files. Adding a doc past the ceiling is meant to be a deliberate
 # edit here, so raise max_docs in the same commit that adds the doc.
 golden_manifest:
   min_docs: 10
-  max_docs: 31
+  max_docs: 32
   include:
     - docs/**/*.md
   exclude:

dokimi/changelog.d/20261023_120000_napetrov_task-format-detect.md

@@ -0,0 +1,16 @@
+### Added
+
+- **`tasks inspect <path> [--json] [--out FILE]`** — detects the eval-task
+  format of a file, a task directory or a whole tree and reports its
+  structure: kind (executable, answer-only, trigger, localization), item
+  count, present and missing parts, lint problems and which dokimi backends
+  consume it today. Recognises Harbor, BenchFlow `task.md`, terminal-bench v1,
+  SWE-bench instances, Inspect AI samples (JSONL and CSV) and `@task` files
+  (scanned as text, never imported), OpenAI evals JSONL and registry specs,
+  skill-creator `evals.json` and trigger sets, skill-reach query sets,
+  promptfoo configs and test files, dokimi question sets and ExploreBench
+  tasks; anything else is reported as `unknown` with its files. A dataset
+  file a config names (an Inspect `FieldSpec` remap, an OpenAI evals
+  model-graded spec, promptfoo `tests: file://`) is read as that config says,
+  and a git-LFS pointer stub is reported as one. `--json` and `--out` write
+  the `task_inspect.v1` artifact. Detection only: no new format runs yet.

dokimi/changelog.d/20261024_120000_napetrov_answer-formats.md

@@ -0,0 +1,27 @@
+### Added
+
+- **`questions import <path>`**: imports Inspect AI samples and `@task` files
+  (read with `ast`, never imported), OpenAI evals JSONL (graded per its
+  registry YAML), skill-creator `evals.json` and promptfoo configs as
+  questions.v2 sets. Each question has a `grading` block of format-neutral
+  checks (`match`, `exact`, `fuzzy`, `regex`, `choice`, `json`, `judge`,
+  `unsupported`) and `prompt_style: verbatim`. The import report lists what
+  did not carry over.
+- **`eval reference`**: scores `answers generate` or `arms run` output against
+  those checks. Deterministic checks need no model. `judge` checks go to the
+  checklist judge in both polarities. It writes eval.v1 or an arms artifact.
+- **`eval native-import <log>`**: reads an `inspect eval --log-format json`
+  log or a `promptfoo eval -o` output as answers.v1 plus eval.v1, so native
+  verdicts can be compared with dokimi's on the same completions.
+- skill-creator imports set `expected_skill` to the skill the `evals.json`
+  belongs to, so the same set also feeds the `skill-catalog:` trigger eval.
+- `answers generate --doc-source none` answers the baseline only.
+- `tasks inspect` lists `questions import` as a consumer of the answer-only formats.
+
+### Changed
+
+- The answerer sends a question with `prompt_style: verbatim` as it is,
+  without the expert-answer wrapper.
+- task_inspect.v1 accepts the `questions import` consumer.
+- questions.v2 declares `grading` and `prompt_style` (a widening; existing
+  files are unchanged).

dokimi/docs/README.md

@@ -255,6 +255,8 @@ code.
 - [coding-harnesses.md](coding-harnesses.md) — run executable coding tasks
   through Codex, Claude Code, or terminal-bench/Harbor harnesses and compare
   verifier pass rate against a baseline.
+- [answer-formats.md](answer-formats.md) — run Inspect AI, OpenAI evals,
+  skill-creator and promptfoo answer-only sets on the answers track
 - [context-window.md](context-window.md) — read a run's context window from
   its Claude Code trace with `cli.py context trace`: start-up load, growth,
   compactions, tool-result weight. No LLM call.

dokimi/docs/answer-formats.md

@@ -0,0 +1,107 @@
+---
+depends_on:
+  - dokimi/dokimi/task_formats/imports/**
+  - dokimi/dokimi/eval/reference_checks.py
+  - dokimi/dokimi/eval/reference_run.py
+  - dokimi/dokimi/commands/answer_formats.py
+---
+<!--
+Copyright (C) 2026 Intel Corporation
+SPDX-License-Identifier: Apache-2.0
+-->
+
+# Answer-only eval formats
+
+Inspect AI, OpenAI evals, skill-creator `evals.json` and promptfoo sets run on
+dokimi's answer track. There is no separate runner. `questions import`
+translates a set into a questions.v2 file in which every question carries a
+`grading` block of format-neutral checks. The usual commands then answer it
+(`answers generate`, `arms run`), and `eval reference` scores the answers
+against those checks.
+
+```bash
+python cli.py questions import evals/born_first.jsonl --grader registry/evals/born-first.yaml -o q.json
+python cli.py answers generate --product born-first --questions q.json --doc-source none \
+    --model claude-haiku-5-5 --provider amazon-bedrock --output answers.json
+python cli.py eval reference --answers answers.json --questions q.json -o eval.json
+```
+
+`tasks inspect <path>` names the format, and `questions import` uses the same
+detection. Use `--format` to override it.
+
+## The canonical record
+
+| Key | From |
+|---|---|
+| `question` | The prompt as the native tool sends it. Chat inputs are flattened to one prompt, with `[role]` headings when there is more than one turn. MCQ rows are rendered with Inspect's single-answer template. |
+| `prompt_style: verbatim` | The answerer sends `question` as it is, without its expert-answer wrapper, so that `ANSWER: X` and exact-match graders see a reply to the original prompt. A context arm puts the docs first and the prompt after them. |
+| `expected_topics` | The reference answers, assertion values or expectations. `eval checklist` and the judge read them. |
+| `grading.checks` | One check per native grader or assertion (below). Optional fields: `threshold`, `native_scorer`, `source_format`. |
+| `expected_skill` | skill-creator only: the skill the `evals.json` belongs to (`skill_name`, else the `<skill>/evals/` directory), so the set also scores [skill selection](evaluating-treatments.md#skill-selection-does-the-agent-load-the-right-skill) on a `skill-catalog:` arm. |
+| `metadata` | Sample metadata, promptfoo vars and prompt label, and the skill-creator `files`. |
+
+Ids match the native tool's, so a native log joins on the same ids. An Inspect
+sample keeps its `id`; samples without one are numbered from 1. A promptfoo
+question is `<testIdx>` when the config has one prompt and
+`<testIdx>.p<promptIdx>` when it has several. Other formats use the row's
+`id` when it has one and its position otherwise.
+
+## Checks
+
+| Check | Semantics | Native graders mapped to it |
+|---|---|---|
+| `match` | The value at `location` (`begin`, `end`, `any`, `exact`), with optional `ignore_case`, `ignore_punctuation`, `strip`, `numeric` (Inspect `match_str`) | Inspect `match`, `includes`; OpenAI `Match` (case-sensitive `startswith`), `Includes`; promptfoo `contains*`, `icontains*`, `equals`, `starts-with` |
+| `exact` | SQuAD-normalised equality | Inspect `exact` |
+| `fuzzy` | Normalised containment, in either direction | OpenAI `FuzzyMatch` |
+| `regex` | `re.search` | promptfoo `regex`, Inspect `pattern` (approximated) |
+| `choice` | The letters of the last `ANSWER: ...` line, parsed the way Inspect parses them | Inspect `choice` with `multiple_choice()` |
+| `json` | Parses as JSON. With `values`, it must equal one of them. With `contains`, it is the first JSON value anywhere in the answer. | OpenAI `JsonMatch`; promptfoo `is-json`, `contains-json`, `equals` with an object value |
+| `judge` | One required point (`criterion`, `reference` or both), asked of the [checklist](evaluating-treatments.md) judge in both polarities. 1 when the two agree that it is covered, 0 when they agree that it is not, 0.5 when they split. | Inspect `model_graded_qa`/`model_graded_fact`; OpenAI `ModelBasedClassify` specs; promptfoo `llm-rubric`, `g-eval`, `factuality`, `model-graded-closedqa`; every skill-creator expectation |
+| `unsupported` | Listed with the native name and never counted | promptfoo `javascript`, `python`, `similar`, `cost`, `latency` and other asserts; unknown Inspect scorers |
+
+Every check accepts `mode` (`any`/`all` over its `values`), `negate` and
+`weight`. A question's score is the weighted mean of the checks that were
+counted. It passes when it reaches `threshold`; without a threshold, it passes
+when every counted check passes. A failed answer call leaves its checks
+unscored rather than failed. `--no-judge` leaves the `judge` checks unscored
+and makes no model calls; an arms artifact written that way has `judge_model`
+and `judge_provider` set to `reference`.
+
+`eval reference` writes eval.v1 for an `answers generate` file and an arms
+artifact for an `arms run` file. Each block carries `scorer: reference`, and
+the run's pass rates and mean scores are under `run_metadata.summary` (or
+`reference_scoring`).
+
+## What does not carry over
+
+| Format | Lossy |
+|---|---|
+| All | Only one message is sent: system prompts and few-shot turns are folded into the prompt. Tools, files and sandboxes are not provided. |
+| Inspect `@task` | The file is parsed with `ast` and is never imported or run. It reads `Sample(...)` literals, `json_dataset("<file inside the task dir>", FieldSpec(...))`, the `multiple_choice()` and `system_message("...")` solvers and the literal arguments of the first scorer. A computed dataset, `hf_dataset` or `csv_dataset` is refused. Other solvers (`chain_of_thought`, ...) are noted and not reproduced. |
+| Inspect model-graded | The C/P/I grade becomes the checklist's covered/split/absent verdict. Inspect's grader template is not used. |
+| OpenAI evals | The registry YAML (`--grader`) names the class. With no YAML, `match` is the default. Custom `evals.elsuite` classes are refused. `ModelBasedClassify` specs (`fact`, `closedqa`, ...) and their choice scores become one judge check against `ideal`. |
+| skill-creator | The grader marks expectations against the run's transcript and output files. Here they are judged on the answer text. `files` are recorded but not attached. |
+| promptfoo | `providers` are replaced by the answer track's model. Only `{{ var }}` substitution is done (filters, tags and `file://` vars are left as written). Prompt files must be text/JSON/YAML inside the config directory (`.js`/`.py` prompts are not run). Assert `transform`, `scenarios` and the schema of `is-json` are not applied. |
+
+## Delegating to the native tool
+
+When a set cannot be translated (a computed dataset, a custom scorer, a
+tool-using solver), run it with its own CLI. Then read the log with
+`eval native-import`:
+
+```bash
+inspect eval task.py --model bedrock/... --log-format json --log-dir logs
+python cli.py eval native-import logs/<run>.json --answers-out native_answers.json --eval-out native_eval.json
+promptfoo eval -c promptfooconfig.yaml -o out.json
+python cli.py eval native-import out.json --answers-out native_answers.json --eval-out native_eval.json
+```
+
+The completions become an answers.v1 file. They are stored as `without_docs`,
+because the native run had no dokimi doc context. The tool's own verdicts
+become an eval.v1 file with `scorer: native:<tool>`. Running `eval reference`
+on that answers file re-grades the same completions with dokimi's checks,
+which shows where the two graders disagree. An Inspect sample that errored
+keeps its error on the answer, so `eval reference` leaves it unscored, as
+Inspect does. A promptfoo output must come from
+a single provider. Supported log formats are Inspect `json` logs (inspect-ai
+0.3.283) and promptfoo `-o *.json` (0.125.0).

dokimi/docs/architecture.md

@@ -70,7 +70,7 @@ points up:
 | Layer | Modules |
 | --- | --- |
 | front end | `cli`, `commands`, `__main__` (the argparse front end; no layer below imports it) |
-| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, and the checkers `config_check`, `schema_check`, `schema_compat` |
+| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, `task_formats`, and the checkers `config_check`, `schema_check`, `schema_compat` |
 | evaluation | `eval`, `personas`, `questions`, `treatments`, `plugins` |
 | components | `ingest`, `metrics`, `gate`, `mcp`, `skills`, `agent_profiles` |
 | foundation | `__init__` (the `dokimi` package itself), `_version`, `artifacts`, `records`, `llm`, `process`, `caching`, `config`, `registry`, `intents`, `defaults`, `console`, `utils`, `metrics/usage.py` |

dokimi/docs/coding-harnesses.md

@@ -78,6 +78,52 @@ refused (a few hundred bytes of nested aliases expand without bound), and
 frontmatter nested too deeply to parse is a parse error. YAML and TOML dates
 and times become ISO strings, and YAML sets sorted lists.
 
+### Other eval formats: `tasks inspect`
+
+`python cli.py tasks inspect <path> [--json] [--out FILE]` names the eval-task format of a
+file, a task directory, or every item under a tree, and reports its structure:
+kind, item count, which parts are present or missing (instruction,
+environment, oracle, verifier, metadata), lint problems, and what in dokimi
+can consume it today. It only reads: nothing is run or converted, a Python
+file is scanned as text and never imported, and a file that does not parse is
+a problem on its report, not an error. A path no format claims is reported as
+`unknown` with its files (or keys) and likely prompt and verifier files; when
+two formats claim a path about equally, the report lists the other as
+`ambiguous` with its evidence. `--json` prints and `--out` writes the report
+as a `task_inspect.v1` artifact (`schemas/task_inspect.v1.json`).
+
+A dataset file a config in the same tree names is read as that config says:
+an Inspect `@task` file's `json_dataset("x.jsonl", FieldSpec(input="question",
+target="answer"))` makes `x.jsonl` Inspect samples under those keys, an OpenAI
+evals registry spec makes its `samples_jsonl` OpenAI evals (with no `ideal`
+needed when the spec's class is model-graded), and promptfoo `tests:
+file://tests.csv` makes that file promptfoo tests. The report says which file
+named it. A file that is a git-LFS pointer stub (a clone without `git lfs
+pull`) is reported as `git-lfs-pointer`, with the format a config expects it
+to be when one names it. `tests/fixtures/task_formats/` holds a small corpus of
+real files in each format, with their sources in its `MANIFEST.yaml`.
+
+| Format | Recognised by | Kind | Consumed by |
+|---|---|---|---|
+| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` |
+| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` |
+| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | none yet |
+| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | none yet |
+| `inspect-samples` | JSON/JSONL/CSV rows with `input` and `target` (or the keys a `FieldSpec` names) | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
+| `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | `questions import` ([answer-only formats](answer-formats.md)) |
+| `openai-evals` | JSONL rows with `input` (chat messages) and `ideal` | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
+| `openai-evals-spec` | registry YAML with `class: evals.elsuite.*` run specs | answer-only | none yet |
+| `skill-creator-evals` | `evals.json` with `skill_name` and `evals[].prompt` | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
+| `skill-creator-trigger` | rows with `query` and `should_trigger` | trigger | none yet |
+| `skill-reach` | `QuerySet` (`queries[]` with `query_id`/`id`, `text`, `expected_skill`) | trigger | none yet |
+| `promptfoo` | `promptfooconfig*.yaml`/`.json`, or a config with `prompts`, `providers`/`targets` and `tests`/`scenarios`/`redteam` | answer-only | `questions import` ([answer-only formats](answer-formats.md)) |
+| `promptfoo-tests` | CSV with `__expected` columns, or rows of `{vars, assert}` | answer-only | none yet |
+| `dokimi-questions` | questions.v1/v2 JSON | answer-only (trigger with `expected_skill`) | questions track |
+| `dokimi-explore-task` | YAML with `id`, `query` and `references[].path` | localization | explore track |
+| `git-lfs-pointer` | a git-LFS spec v1 pointer stub | unknown | none (fetch it first) |
+
+`tasks list` and `tasks run` still read only the first two.
+
 The `docker-*` harnesses mount the oracle at `/solution` and `/oracle` and the
 verifier at `/tests` and `/verifier`, because a script written for one layout
 refers to its siblings by that layout's path. The directory's contents are

dokimi/dokimi/__init__.py

@@ -22,7 +22,8 @@
   [`dashboard`][dokimi.dashboard] aggregates results across libraries and
   [`orchestrator`][dokimi.orchestrator] runs the one-command pipeline.
 - Executable tasks: [`harnesses`][dokimi.harnesses] runs coding agents on task
-  directories.
+  directories; [`task_formats`][dokimi.task_formats] names the format of other
+  eval-task files and directories (detection only).
 - [`context`][dokimi.context] reads what a run's context window held.
 - Content novelty: [`novelty`][dokimi.novelty]. Repositories as subjects:
   [`repos`][dokimi.repos]. Subject scorecards, one per run:

dokimi/dokimi/artifacts.py

@@ -10,8 +10,9 @@
 model), ``guidance_reach`` (L1 reach probes), ``repo_readiness`` (L0 claim
 checks), ``tool_preference`` (which of two equal tools a model picks, and
 what moves it), ``repo_environment`` (static inspection of what a
-repository offers an agent) and ``repo_budget`` (what a repository puts
-into an agent's context). This module stamps each with a
+repository offers an agent), ``repo_budget`` (what a repository puts
+into an agent's context) and ``task_inspect`` (which eval-task format each
+path under a tree is). This module stamps each with a
 ``schema_version`` and validates it against the JSON Schema in
 ``dokimi/schemas/`` on save and load, so fixtures and baselines stay
 forward-compatible and self-describing.
@@ -52,6 +53,7 @@
     "tool_preference": "tool_preference.v1",
     "repo_environment": "repo_environment.v1",
     "repo_budget": "repo_budget.v1",
+    "task_inspect": "task_inspect.v1",
 }
 
 # kind -> superseded versions still accepted on *load*, newest first. An artifact written under an