Skip to content

feat: implement HeadQA - #1248

Merged
chakravarthik27 merged 2 commits into
release/2.8.0from
feature/implement-the-headqa-dataset
Jul 30, 2026
Merged

feat: implement HeadQA#1248
chakravarthik27 merged 2 commits into
release/2.8.0from
feature/implement-the-headqa-dataset

Conversation

@chakravarthik27

Copy link
Copy Markdown
Collaborator

Harness setup:

harness = Harness(
    task="question-answering",
    model={
        "model": "gpt-4.1-mini",
        "hub": "openai"
    },
    data={
        'data_source': 'HeadQA',
    },
    config={
        "tests": {
            "defaults": {
                "min_pass_rate": 0.8
            },
            "robustness": {
                "add_ocr_typo": {"min_pass_rate": 0.8},
                "uppercase": {"min_pass_rate": 0.8},
                "add_speech_to_text_typo": {"min_pass_rate": 0.8}
            }
        }
    }
)

@chakravarthik27 chakravarthik27 self-assigned this Jul 30, 2026
@chakravarthik27 chakravarthik27 changed the title feat: implement HeadQA dataset loading and update ensure_download_and… feat: implement HeadQA Jul 30, 2026

@blidiselalin blidiselalin left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good.

@chakravarthik27
chakravarthik27 merged commit 418a8da into release/2.8.0 Jul 30, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants