[flyinprogrammer]

Closes #301

Summary

A second, independent query set: each query is a commit subject from the pinned repo's own history, and its targets are the 1–3 non-test source files that commit changed.

 benchmarks/
 ├── annotations/              # 1,251 generated queries (unchanged)
+├── annotations_commits/      # 2,416 commit-subject queries, same JSON format
+├── mine_commit_queries.py    # history-only clones → annotations_commits/
 ├── data.py                   # load_tasks(annotations_dir=…), opt-in --query-set
 ├── run_benchmark.py          # --query-set {annotations,commits}
 └── baselines/ablations.py    # --query-set {annotations,commits}

mine_repo(repo)
  history-only clone (--filter=blob:none), log from the pinned revision (--no-merges --no-renames)
  for each commit, newest first
    query   = subject without PR refs / conventional prefixes; 4-20 words; not a chore
    targets = 1-3 non-test source files changed, present at the pinned revision, inside benchmark_root
    skip commits touching more than 6 files
  keep up to 40, deduplicated

Query set	Queries	Query words found in target	Every word in target
annotations	1,251	74%	31%
commits	2,416	68%	13%

Evidence

• Defaults unchanged: run_benchmark with no flags gives NDCG@10 0.852 on main and on this branch. The other baselines don't take --query-set, so nothing changes for them.
• New set: run_benchmark --query-set commits gives NDCG@10 0.735, written to results/semble-hybrid-commits-<sha>.json.
• Reproducible: re-running mine_commit_queries writes byte-identical files (2,416 queries).
• Checks: make test 391 passed; make lint and make typecheck clean; ruff check and ruff format --check clean on benchmarks/.

Merge Danger

Door: two-way

Benchmark tooling only; nothing under src/ changes.

Blast Radius: benchmarks

The data adds 740 KB in 63 JSON files. Mining needs network access to clone history, about 600 MB of history-only clones in ~/.cache/semble-bench-history.