feat: benchmark manifest schema and selection mechanism

[randomparity]

What this does

Adds the machine-readable task-manifest schema and the deterministic selection,
validation, and materialization mechanisms that encode the v1 benchmark protocol
(docs/benchmarks/adept-workflow-v1.md).

Components

• benchmarks/manifest.py — pinned protocol constants (dataset/evaluator revisions, supported licenses, required fields) and canonical-JSON SHA-256 digest computation.
• benchmarks/validate_manifest.py — pure-Python validator with 16 constraint checks (protocol pins, group/task counts, repository disjointness, SHA format, license allowlist, GitHub URL format, field presence, digest match). CLI exits 0/1/2.
• benchmarks/select_tasks.py — deterministic selection: enumerate SWE-bench_Verified test split by ascending instance_id, filter by license/author eligibility, enumerate 3-task combinations per repo in lexical order, test candidate common revisions, select first two repository-disjoint qualifying groups. validate_fn seam for testing.
• benchmarks/validate_task.py — per-task validation: clone, apply gold patch via stdin, check Docker availability, run pre-patch failure and post-patch pass checks. Repo format validation prevents command injection.
• benchmarks/materialize_issues.py — create benchmark-owned GitHub issues feat: import the skills tree, without the companion or prose assertions #1/docs: design first-party replacements for the superpowers-derived skills #2/docs: plan the companion strip and migration realignment #3 per group with exact labels. Idempotency via existing-issue detection. Dry-run mode. Topology digest.
• benchmarks/fetch_dataset.py — normalize SWE-bench_Verified rows to canonical JSONL. Lazy-imports huggingface_hub and pyarrow.

Toolchain

• ruff.toml — Python 3.12 target, line length 100, lint rules E/F/W/I/UP/B.
• Justfile — py-test (unittest discovery) and py-lint (ruff check + format) recipes wired into commit-check and verify.
• CI workflow — ruff added to Homebrew install line.
• .gitignore — __pycache__/ and *.pyc (Python bytecode contains host paths).

Design records

• ADR 0019 — Python for benchmark infrastructure, bash for gate scripts.
• Spec — manifest schema, selection algorithm, validation, materialization, security boundaries.
• Plan — 8-task TDD implementation plan.

Testing

68 unittest tests across 7 test modules. No network, no Docker, no GitHub API — all external interactions are stubbed via callable parameters. just verify green (all shell gates, Python tests, ruff lint, format, records, public-safety, plugin-check, actions-check).

Review

ADR, spec, and plan adversarially reviewed via $gauntlet (ADR approved; spec 5-iteration review; plan 2-iteration review). Branch diff reviewed via $trial-loop (approved, 3 iterations, 5 findings fixed). Security scan via $detect-evil (clean — repo name validation, no shell=True, pinned revisions).

Closes #119

[randomparity]

Review — issue #119

• verdict: approve
• findings: 0 (final pass)
• iterations: 3 (branch trial-loop)
• security: detect-evil clean (no findings)
• deferrals: none

Review arms run:

• ADR 0019: $gauntlet file-list review — approved (2 iterations, 1 finding fixed)
• Spec: $gauntlet file-list review — 5 iterations, 8→8→4→3→6 findings, all resolved
• Plan: $gauntlet file-list review — 2 iterations, 10→7 findings, all resolved
• Branch diff: $trial-loop --base main — approved (3 iterations, 5 findings fixed, 3 rejected)
• Security: $detect-evil --base main — clean

Suppressed: 0. Deferrals: 0.