feat(catalog): read trigger sets back and list skill package formats

dokimi/changelog.d/20261023_120000_napetrov_task-format-detect.md

@@ -0,0 +1,11 @@
+### Added
+
+- **`tasks inspect <path> [--json]`** — detects the eval-task format of a file,
+  a task directory or a whole tree and reports its structure: kind
+  (executable, answer-only, trigger), item count, present and missing parts,
+  lint problems and which dokimi backends consume it today. Recognises Harbor,
+  BenchFlow `task.md`, terminal-bench v1, SWE-bench instances, Inspect AI
+  samples and `@task` files (scanned as text, never imported), OpenAI evals
+  JSONL, skill-creator `evals.json` and trigger sets, skill-reach query sets,
+  promptfoo configs and dokimi question sets; anything else is reported as
+  `unknown` with its files. Detection only: no new format runs yet.

dokimi/changelog.d/20261024_120000_napetrov_trigger_formats.md

@@ -0,0 +1,5 @@
+### Added
+
+- **Read trigger sets back** — `dokimi catalog trigger-set --trigger-set PATH[=TARGET]` reads a google/skill-reach QuerySet, an anthropics skill-creator / agentskills.io `[{query, should_trigger}]` set (TARGET names its skill), or a questions set with `expected_skill`, relints it against `--catalog`, and writes any of the three formats, so an imported set runs on the `skill-catalog:` arm. `arms run` given a skill-reach or skill-creator file names the conversion command; `tasks inspect` lists `catalog trigger-set` as the consumer of both formats.
+- **Skill package formats** — `dokimi catalog formats --root DIR [--json]` lists Agent Skills `SKILL.md` packages (with the host read from the path, the agentskills.io name/description limits linted, and Codex `agents/openai.yaml` noted), Cursor `.mdc` rules, Copilot `*.instructions.md`, Gemini CLI `gemini-extension.json` and Claude Code `.claude-plugin/plugin.json` bundles, with how each is selected (description, globs, always, install).
+- **Skill selection precision** — `skill_selection_summary` and the arms report's *Skill selection* table carry `precision`: hits over the questions where the arm loaded any skill, so misroutes and false triggers count against it.

dokimi/docs/architecture.md

@@ -70,7 +70,7 @@ points up:
 | Layer | Modules |
 | --- | --- |
 | front end | `cli`, `commands`, `__main__` (the argparse front end; no layer below imports it) |
-| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, and the checkers `config_check`, `schema_check`, `schema_compat` |
+| tracks | `runner`, `report`, `subjects`, `novelty`, `harnesses`, `context`, `repos`, `explore`, `tool_preference`, `external`, `dashboard`, `orchestrator`, `datasets_export`, `coverage`, `task_formats`, and the checkers `config_check`, `schema_check`, `schema_compat` |
 | evaluation | `eval`, `personas`, `questions`, `treatments`, `plugins` |
 | components | `ingest`, `metrics`, `gate`, `mcp`, `skills`, `agent_profiles` |
 | foundation | `__init__` (the `dokimi` package itself), `_version`, `artifacts`, `records`, `llm`, `process`, `caching`, `config`, `registry`, `intents`, `defaults`, `console`, `utils`, `metrics/usage.py` |

dokimi/docs/coding-harnesses.md

@@ -78,6 +78,35 @@ refused (a few hundred bytes of nested aliases expand without bound), and
 frontmatter nested too deeply to parse is a parse error. YAML and TOML dates
 and times become ISO strings, and YAML sets sorted lists.
 
+### Other eval formats: `tasks inspect`
+
+`python cli.py tasks inspect <path> [--json]` names the eval-task format of a
+file, a task directory, or every item under a tree, and reports its structure:
+kind, item count, which parts are present or missing (instruction,
+environment, oracle, verifier, metadata), lint problems, and what in dokimi
+can consume it today. It only reads: nothing is run or converted, a Python
+file is scanned as text and never imported, and a file that does not parse is
+a problem on its report, not an error. A path no format claims is reported as
+`unknown` with its files (or keys) and likely prompt and verifier files; when
+two formats claim a path about equally, the report lists the other as
+`ambiguous` with its evidence.
+
+| Format | Recognised by | Kind | Consumed by |
+|---|---|---|---|
+| `harbor` | directory with `task.toml` | executable | Harbor-backed harnesses (unless Harbor would refuse it), `docker-*` |
+| `task-md` | directory with a BenchFlow `task.md` | executable | `benchflow*`, `docker-*` |
+| `terminal-bench-v1` | directory with `task.yaml` + `run-tests.sh` / `Dockerfile` / `solution.sh` | executable | none yet |
+| `swe-bench` | JSON/JSONL rows with `instance_id`, `repo`, `base_commit`, `problem_statement` (`FAIL_TO_PASS`) | executable | none yet |
+| `inspect-samples` | rows with `input` and `target` | answer-only | none yet |
+| `inspect-task` | `.py` importing `inspect_ai` with a `@task` function | answer-only (executable with a `sandbox=`) | none yet |
+| `openai-evals` | JSONL rows with `input` (chat messages) and `ideal` | answer-only | none yet |
+| `skill-creator-evals` | `evals.json` with `skill_name` and `evals[].prompt` | answer-only | none yet |
+| `skill-creator-trigger` | rows with `query` and `should_trigger` | trigger | none yet |
+| `skill-reach` | `QuerySet` (`queries[]` with `query_id`/`id`, `text`, `expected_skill`) | trigger | none yet |
+| `dokimi-questions` | questions.v1/v2 JSON | answer-only (trigger with `expected_skill`) | questions track |
+
+`tasks list` and `tasks run` still read only the first two.
+
 The `docker-*` harnesses mount the oracle at `/solution` and `/oracle` and the
 verifier at `/tests` and `/verifier`, because a script written for one layout
 refers to its siblings by that layout's path. The directory's contents are

dokimi/dokimi/__init__.py

@@ -22,7 +22,8 @@
   [`dashboard`][dokimi.dashboard] aggregates results across libraries and
   [`orchestrator`][dokimi.orchestrator] runs the one-command pipeline.
 - Executable tasks: [`harnesses`][dokimi.harnesses] runs coding agents on task
-  directories.
+  directories; [`task_formats`][dokimi.task_formats] names the format of other
+  eval-task files and directories (detection only).
 - [`context`][dokimi.context] reads what a run's context window held.
 - Content novelty: [`novelty`][dokimi.novelty]. Repositories as subjects:
   [`repos`][dokimi.repos]. Subject scorecards, one per run:

dokimi/dokimi/commands/arms.py

@@ -460,6 +460,7 @@ def _load_questions(path: str) -> list[dict]:
     """
     from dokimi.questions.artifact import load_questions
 
+    _refuse_external_trigger_set(path)
     questions = load_questions(path).questions
     ids = [question_id(q) for q in questions]
     for i, qid in enumerate(ids):
@@ -471,6 +472,30 @@ def _load_questions(path: str) -> list[dict]:
     return questions
 
 
+def _refuse_external_trigger_set(path: str) -> None:
+    """Point a skill-reach or skill-creator trigger set at the command that reads it.
+
+    Those sets are read through ``catalog trigger-set --trigger-set``, which also
+    lints them against the catalog the arm will offer; a questions loader that
+    guessed at them would drop that lint and, for skill-creator, the target skill.
+    """
+    import json
+
+    from dokimi.external.trigger_set import FORMAT_QUESTIONS, detect_trigger_format
+
+    try:
+        data = json.loads(Path(path).read_text(encoding="utf-8"))
+    except (OSError, ValueError):
+        return  # the questions loader reports it
+    fmt = detect_trigger_format(data)
+    if fmt and fmt != FORMAT_QUESTIONS:
+        target = "=<skill>" if fmt == "skill-creator" else ""
+        raise ValueError(
+            f"{path} is a {fmt} trigger set, not a questions set; convert it with "
+            f"`dokimi catalog trigger-set --catalog <skills> --trigger-set {path}{target} "
+            f"--format questions --out q.json`, then run the skill-catalog: arm on q.json")
+
+
 def _refuse_duplicate_ids(ids: list, path: str) -> None:
     """Raise on a repeated id."""
     counts = Counter(qid for qid in ids if qid is not None)

dokimi/dokimi/commands/catalog.py

@@ -141,7 +141,6 @@ def cmd_catalog_trigger_set(args: argparse.Namespace) -> None:
 
     from dokimi.external import trigger_set as ts
     from dokimi.skills import discover_skills
-    from dokimi.utils import read_json_input
 
     catalog_root = Path(args.catalog)
     if not catalog_root.is_dir():
@@ -154,16 +153,11 @@ def cmd_catalog_trigger_set(args: argparse.Namespace) -> None:
         if args.target not in {s.name for s in catalog}:
             raise SystemExit(f"--target {args.target!r} is not a skill in --catalog "
                              f"(have: {', '.join(sorted(s.name for s in catalog))})")
-    sources = []
-    for spec in args.questions:
-        path, _, rule = spec.rpartition("=")  # labels never contain '=', paths may
-        if not path or not rule:
-            raise SystemExit(f"--questions needs PATH=LABEL (a skill name, 'none' or '@field'): {spec}")
-        data = read_json_input(path, flag="--questions")
-        questions = data.get("questions", data) if isinstance(data, dict) else data
-        if not isinstance(questions, list) or not all(isinstance(q, dict) for q in questions):
-            raise SystemExit(f"{path}: expected a list of question objects")
-        sources.append((questions, rule))
+    sources = _question_sources(args.questions or [])
+    sources += _trigger_set_sources(args.trigger_set or [], catalog)
+    if not sources:
+        raise SystemExit("give --questions PATH=LABEL or --trigger-set PATH[=TARGET]")
+    specs = list(args.questions or []) + list(args.trigger_set or [])
 
     rewrite = None
     if args.rewrite:
@@ -181,7 +175,7 @@ def rewrite(text: str) -> str:
     # An @field rule that resolves for no question in its source is a typo in the
     # path, not a set of negatives; refuse it rather than write "nothing loads".
     start = 0
-    for (qs, rule), spec in zip(sources, args.questions, strict=True):
+    for (qs, rule), spec in zip(sources, specs, strict=True):
         chunk = queries[start:start + len(qs)]
         start += len(qs)
         if rule.startswith("@") and chunk and not any(q.labelled for q in chunk):
@@ -217,6 +211,86 @@ def rewrite(text: str) -> str:
     print(f"wrote {args.out} ({args.format}, {n_out} queries, catalog of {len(catalog)})")
 
 
+def cmd_catalog_formats(args: argparse.Namespace) -> None:
+    import json
+
+    from dokimi.skills import formats
+
+    root = Path(args.root)
+    if not root.is_dir():
+        raise SystemExit(f"--root is not a directory: {root}")
+    packages = formats.detect_skill_packages(root)
+    summary = formats.summarize(packages)
+    if args.json:
+        print(json.dumps({"root": str(root), "summary": summary,
+                          "packages": [p.to_dict(root) for p in packages]}, indent=2))
+        return
+    for p in packages:
+        problems = "; ".join(p.problems) or "ok"
+        print(f"{p.format}\t{p.name or '-'}\t{p.selected_by}\t{p.host or '-'}\t"
+              f"{p.path.relative_to(root).as_posix()}\t{problems}")
+    print()
+    for fmt, row in summary.items():
+        print(f"# {fmt}: {row['n']} ({row['loadable']} loadable as skills, "
+              f"{row['with_problems']} with problems); selected by {row['selected_by']}")
+
+
+def _question_sources(specs: list) -> list:
+    """Each ``--questions PATH=LABEL`` as a (questions, rule) source."""
+    from dokimi.utils import read_json_input
+
+    sources = []
+    for spec in specs:
+        path, _, rule = spec.rpartition("=")  # labels never contain '=', paths may
+        if not path or not rule:
+            raise SystemExit(f"--questions needs PATH=LABEL (a skill name, 'none' or '@field'): {spec}")
+        data = read_json_input(path, flag="--questions")
+        questions = data.get("questions", data) if isinstance(data, dict) else data
+        if not isinstance(questions, list) or not all(isinstance(q, dict) for q in questions):
+            raise SystemExit(f"{path}: expected a list of question objects")
+        sources.append((questions, rule))
+    return sources
+
+
+def _trigger_set_sources(specs: list, catalog) -> list:
+    """Each ``--trigger-set PATH[=TARGET]`` read back as a (questions, rule) source.
+
+    The set is read into the canonical trigger model and handed on as questions
+    labelled by their own ``expected_skill``, so it is linted (and rewritten) like
+    any other source. TARGET names the skill a skill-creator set was written for.
+    """
+    from dokimi.external import trigger_set as ts
+    from dokimi.utils import read_json_input
+
+    names = {s.name for s in catalog}
+    out = []
+    for spec in specs:
+        path, target = spec, None
+        if not Path(spec).exists() and "=" in spec:
+            path, _, target = spec.rpartition("=")
+        data = read_json_input(path, flag="--trigger-set")
+        try:
+            fmt, queries = ts.read_trigger_set(data, target=target)
+        except ValueError as exc:
+            raise SystemExit(f"{path}: {exc}") from exc
+        if fmt == ts.FORMAT_SKILL_CREATOR:
+            if target not in names:
+                raise SystemExit(f"{spec}: skill-creator target {target!r} is not a skill in "
+                                 f"--catalog (have: {', '.join(sorted(names))})")
+            if len(names) > 1:
+                print(f"note: {path}: skill-creator negatives are read as 'no skill loads'; in a "
+                      f"catalog of {len(names)} a rival load counts as a false trigger", file=sys.stderr)
+        elif target:
+            raise SystemExit(f"{spec}: =TARGET applies to skill-creator sets only, this is {fmt}")
+        print(f"read {path} as {fmt} ({len(queries)} queries)", file=sys.stderr)
+        rows = ts.to_questions(queries)["questions"]
+        for row, q in zip(rows, queries, strict=True):
+            if q.reach:
+                row[ts.REACH_FIELDS] = q.reach
+        out.append((rows, "@expected_skill"))
+    return out
+
+
 def register(sub, positive_int) -> None:
     """Add the `catalog` subcommand group."""
 
@@ -253,16 +327,31 @@ def register(sub, positive_int) -> None:
     )
     scan_p.set_defaults(func=cmd_catalog_scan)
 
+    fm_p = catalog_sub.add_parser(
+        "formats",
+        help="List the skill/rule packages under --root by ecosystem (Agent Skills, Cursor "
+             "rules, Copilot instructions, Gemini extensions, Claude plugins)",
+    )
+    fm_p.add_argument("--root", required=True, help="Local checkout to scan")
+    fm_p.add_argument("--json", action="store_true", help="Print JSON instead of a table")
+    fm_p.set_defaults(func=cmd_catalog_formats)
+
     ts_p = catalog_sub.add_parser(
         "trigger-set",
         help="Build a skill-selection query set (labels + leakage/framing lint) from question sets",
     )
     ts_p.add_argument("--catalog", required=True, help="Directory of the skills the agent will see")
     ts_p.add_argument(
-        "--questions", action="append", required=True, metavar="PATH=LABEL",
+        "--questions", action="append", metavar="PATH=LABEL",
         help="Question file and its label: a skill name, 'none' (no skill should load), or "
              "'@field.path' read per question (e.g. @metadata.skill_under_test). Repeatable.",
     )
+    ts_p.add_argument(
+        "--trigger-set", action="append", dest="trigger_set", metavar="PATH[=TARGET]",
+        help="An existing trigger set to read back: a skill-reach QuerySet, a skill-creator / "
+             "agentskills.io [{query, should_trigger}] set (=TARGET names its skill), or questions "
+             "with expected_skill. Repeatable; combines with --questions.",
+    )
     ts_p.add_argument("--format", choices=["reach", "skill-creator", "questions"], default="reach",
                       help="reach: google/skill-reach QuerySet; skill-creator: one target's "
                            "eval set; questions: a questions artifact for the skill-catalog: arm")

dokimi/dokimi/commands/tasks.py

@@ -47,6 +47,43 @@ def cmd_tasks_list(args: argparse.Namespace) -> None:
         print(f"{task.name}\t{difficulty}\t{','.join(tags)}")
 
 
+def _inspect_line(report: dict) -> str:
+    n = report["items"]
+    items = "" if n is None else f"\t{n} item{'' if n == 1 else 's'}"
+    return f"{report['path']}\t{report['format']}\t{report['kind']}{items}\t-> {', '.join(report['consumers'])}"
+
+
+def _print_inspect(reports: list[dict]) -> None:
+    for r in reports:
+        print(_inspect_line(r))
+        if r["present"] or r["missing"]:
+            print(f"  parts: {', '.join(r['present']) or '-'}; missing: {', '.join(r['missing']) or '-'}")
+        for alt in r["alternatives"]:
+            print(f"  ambiguous: also {alt['format']} ({alt['confidence']:.2f}: {', '.join(alt['evidence'])})")
+        for problem in r["problems"]:
+            print(f"  problem: {problem}")
+    by_format: dict[str, int] = {}
+    for r in reports:
+        by_format[r["format"]] = by_format.get(r["format"], 0) + 1
+    print(f"{len(reports)} item{'' if len(reports) == 1 else 's'}: " + ", ".join(f"{f} {n}" for f, n in sorted(by_format.items())))
+
+
+def cmd_tasks_inspect(args: argparse.Namespace) -> None:
+    """Detect the eval-task format of a path (or of everything under it) and report its structure."""
+    import json
+
+    from dokimi.task_formats import inspect_path
+
+    try:
+        reports = [r.as_dict() for r in inspect_path(Path(args.path))]
+    except FileNotFoundError as exc:
+        raise SystemExit(f"error: {exc}") from None
+    if args.json:
+        print(json.dumps({"root": str(args.path), "items": reports}, indent=2, default=str))
+    else:
+        _print_inspect(reports)
+
+
 def cmd_tasks_harnesses(args: argparse.Namespace) -> None:
     from dokimi.harnesses import known_harnesses
 
@@ -185,6 +222,14 @@ def register(sub, positive_int) -> None:
     list_p.add_argument("--tasks-root", default=None, help=_TASKS_ROOT_HELP)
     list_p.set_defaults(func=cmd_tasks_list)
 
+    inspect_p = tasks_sub.add_parser(
+        "inspect",
+        help="Detect the eval-task format of a file or directory tree and report its structure (no run)",
+    )
+    inspect_p.add_argument("path", help="A task directory, a dataset file, or a tree holding either")
+    inspect_p.add_argument("--json", action="store_true", help="Print the reports as JSON")
+    inspect_p.set_defaults(func=cmd_tasks_inspect)
+
     harnesses_p = tasks_sub.add_parser("harnesses", help="List known coding harness aliases")
     harnesses_p.set_defaults(func=cmd_tasks_harnesses)
 

dokimi/dokimi/eval/arm_summary.py

@@ -130,7 +130,7 @@ def summarize_skill_selection(records: Iterable[RecordLike]) -> Dict[str, Any]:
     or ``None`` — no skill should load. Arms that offered no skill are skipped.
 
     The skills loaded in the first turn that loaded any are one choice. ``recall``
-    is strict: a hit needs every one of them acceptable. ``recall_any`` (one
+    and ``precision`` (hits over questions where any skill loaded) are strict: a hit needs every one of them acceptable. ``recall_any`` (one
     acceptable skill among them), ``multi_load_rate`` and ``mean_first_turn_loads``
     show how far the strict figure depends on multi-loading.
     """
@@ -192,6 +192,9 @@ def rate(a: int, b: int) -> Any:
             "n_failed": s["failed"],
             "selection_accuracy": rate(s["correct"], s["n"]),
             "recall": rate(s["hit"], s["pos"]),
+            # Of the questions where the arm loaded a skill, the share where every
+            # loaded skill was acceptable (misroutes and false triggers count against).
+            "precision": rate(s["hit"], s["loaded_n"]),
             # Lenient: the right skill was among those loaded in the first turn.
             "recall_any": rate(s["hit_any"], s["pos"]),
             "multi_load_rate": rate(s["multi"], s["loaded_n"]),