Skip to content

Expand PostgreSQL benchmark beyond current 10-query slice #4

Description

@pro-utkarshM

Goal

Increase benchmark difficulty and size so the selection signal is harder to game and more useful for research.

Scope

Expand beyond the current PostgreSQL authentication slice by adding more chunks, more queries, and harder distractors.

Suggested additions

  • add one or more adjacent PostgreSQL sections
  • add 10 to 20 more queries
  • increase multi-hop and troubleshooting coverage
  • add more version-sensitive stale distractors
  • enlarge candidate pools where useful

Acceptance criteria

  • corpus_chunks_v1.jsonl and queries_v1.jsonl are expanded materially
  • candidate pools and context sets are regenerated consistently
  • the benchmark still validates cleanly
  • analysis output demonstrates whether the larger slice changes the relative strategy rankings

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions