Goal
Increase benchmark difficulty and size so the selection signal is harder to game and more useful for research.
Scope
Expand beyond the current PostgreSQL authentication slice by adding more chunks, more queries, and harder distractors.
Suggested additions
- add one or more adjacent PostgreSQL sections
- add 10 to 20 more queries
- increase multi-hop and troubleshooting coverage
- add more version-sensitive stale distractors
- enlarge candidate pools where useful
Acceptance criteria
corpus_chunks_v1.jsonl and queries_v1.jsonl are expanded materially
- candidate pools and context sets are regenerated consistently
- the benchmark still validates cleanly
- analysis output demonstrates whether the larger slice changes the relative strategy rankings
Goal
Increase benchmark difficulty and size so the selection signal is harder to game and more useful for research.
Scope
Expand beyond the current PostgreSQL authentication slice by adding more chunks, more queries, and harder distractors.
Suggested additions
Acceptance criteria
corpus_chunks_v1.jsonlandqueries_v1.jsonlare expanded materially