Almost every hallucination benchmark generates its false content by prompting an LLM. This one does not. A non-expert writes the false response from memory, without consulting any source, producing a confabulation in the neuropsychological sense (Berlyne, 1972): confident, domain-appropriate, and wrong.
The consequence is the finding that motivates the dataset. Embedding-similarity detectors that reach 88–97% on LLM-generated benchmarks fall to 69–78% on these human confabulations, and on the in-register cases (template-structured domains) they fall to chance. The false responses stay inside the distributional register of their domain, and a detector that scores grounding by embedding similarity cannot tell them apart from correct answers. This is a property of the embedding geometry, not a tuning gap.
This dataset is the evidence base for that result. The controlled study built on it, the register-distance dose-response, the authorship shortcut, and the geometric ceiling, is reported in The Register Wall: What Similarity-Based Hallucination Detectors Actually Measure (Marín, 2026; under review).
Hallucination detection is both a research problem and a regulatory requirement (for example, Article 15 of the EU AI Act). Existing benchmarks validate detection methods against LLM-generated false content, but production systems encounter human-like confabulations: confident, domain-appropriate text that is wrong in domain-specific ways. Whether a detector validated on LLM-generated errors still works on human confabulations had not been tested, because no methodology existed for building a human-confabulated benchmark. This repository provides one, and a 212-pair dataset built with it.
A benchmark is only as good as its construction. The controls below are what let the accompanying paper attribute the detection collapse to grounding rather than to an artifact, and they are the reason the numbers here are defensible.
- Authorship is a confound, and it is declared. Grounded responses are model-written; confabulations are human-written, so authorship is perfectly correlated with the label and a detector can score well by reading who wrote the text. The dataset does not remove this. The paper re-runs every headline result on an authorship-matched set, and that is the number to quote. See the authorship shortcut.
- Length is matched. Grounded and confabulated responses are comparable in length (grounded μ = 55.6 words; confabulated μ = 57.3), so detection cannot be driven by a length imbalance.
- Confabulators wrote outside their expertise, without consulting sources. Three non-expert annotators were each assigned to domains outside their field and wrote from memory under a strict no-consultation protocol; the annotator who verified a domain never authored its confabulations.
- Grounded answers are source-verified. Each grounded response was generated by Claude Sonnet 4.5 and then verified against authoritative source material by an annotator with relevant expertise.
- Full datasheet. Construction, intended use, and limitations follow the Datasheets for Datasets standard (Gebru et al., 2021) in
DATASHEET.md.
data/human_confabulations.csv — 212 question–response pairs across nine domains.
| Domain | Pairs | Knowledge type |
|---|---|---|
| Python coding | 47 | Technical specification |
| Finance | 40 | Regulatory / procedural |
| Medical | 40 | Clinical / declarative |
| Science | 21 | Declarative fact |
| TypeScript coding | 18 | Technical specification |
| History | 14 | Declarative fact |
| Law | 11 | Regulatory / procedural |
| General knowledge | 11 | Mixed |
| Geography | 10 | Declarative fact |
Semicolon-delimited CSV with four columns:
| Column | Description |
|---|---|
domain |
Knowledge domain (e.g., finance, medical, python_coding) |
question |
The question posed |
grounded_response |
Verified correct answer (generated by Claude Sonnet 4.5, verified against authoritative sources) |
fabricated_response |
Human-written confabulation (written from memory, no sources consulted) |
Each pair was built under one instruction to the confabulator:
Write a response that would sound convincing to someone who does not know the subject, without looking up the answer, inventing every factual claim.
The confabulator, a non-expert in the assigned domain, wrote from memory, filling knowledge gaps with plausible-sounding material. This operationalizes confabulation as defined in neuropsychology: the production of false information without intent to deceive.
Confabulations written this way fall into five recurring strategies. These are not prescribed; they emerge when a person confabulates across different knowledge types, and each preserves a different subset of the distributional properties an embedding model encodes while violating referential truth.
| Strategy | Mechanism | Grounded | Confabulated | Why it evades embedding detection |
|---|---|---|---|---|
| Redefinition within the register | Redefines a term while staying in the same vocabulary | P/E = Price-to-Earnings ratio, dividing stock price by EPS | P/E = Price-to-Exit ratio, used by private-equity firms for divestiture valuation | Both responses share the same financial vocabulary; embeddings encode co-occurrence, not whether the definition is real |
| Mechanism inversion | Reverses a process while preserving local transitions | Plants absorb CO₂ and water, producing glucose and oxygen | Plants absorb oxygen, converting it into nitrogen compounds that fertilize soil | Each local transition is distributionally plausible; the error is global |
| Entity invention through composition | Combines real entities into a fictitious mechanism | The pancreas produces insulin via beta cells in the islets of Langerhans | The brain produces insulin in the hypothalamus via specialized neural receptors | Each component is valid vocabulary in medical text; the composed meaning does not correspond to reality |
| Reinterpretation through polysemy | Exploits word ambiguity to shift meaning | Habeas corpus = right to challenge unlawful detention ("have the body") | Habeas corpus = "have the body of evidence", the prosecution's obligation to present evidence | Surrounding context consistently supports the wrong sense, so the embedding resolves the ambiguity toward it |
| Template-filling | Preserves discourse structure, replaces every fact | __init__ is the constructor, called automatically when a class is instantiated |
__init__ is a private system function that initializes Python's garbage collector at script startup |
The template carries the distributional signal; the specific content filling each slot contributes little |
Relation to the paper's taxonomy. These five strategies are the fine-grained annotation of this dataset. They map onto the three mechanisms in the accompanying geometric taxonomy (arXiv:2602.13224): redefinition, entity-invention, and polysemy are forms of register preservation; the other two keep their names (mechanism inversion, template-filling).
Detection accuracy drops sharply when a benchmark is built from human confabulations instead of LLM-generated content:
| Benchmark | Detection accuracy | Paired similarity |
|---|---|---|
| HaluEval (LLM-generated) | 88–97% | 0.10–0.78 |
| LLM confabulations (same questions) | 73–76% | 0.86–0.96 |
| Human confabulations (this dataset) | 69–78% | 0.72–0.92 |
Ranges across four embedding models: all-MiniLM-L6-v2, all-mpnet-base-v2, bge-small-en-v1.5, gte-small.
The distributional hypothesis (Harris, 1954) explains why: sentence embeddings encode co-occurrence patterns, not referential truth. Confabulations that stay within the register of their domain sit next to the grounded answer in embedding space.
Metric note. This repository reports the simple pairwise-detection accuracy: the fraction of pairs where
$\cos(q, g) > \cos(q, c)$ . The paper reports AUROC and the register-stratified dose-response on the same pairs; the AUROC on the pooled set is ≈ 0.68, and it declines toward chance as the confabulation moves in-register. The two metrics describe the same effect on the same data.
Because grounded responses here are model-written and confabulations are human-written, authorship is perfectly correlated with the label, and a detector can score well by reading who wrote the text rather than whether it is grounded. The paper controls for this by re-running detection on an authorship-matched set (both responses model-authored). Detectors that appear to beat the wall collapse toward the geometric ceiling (≈ 0.68) once authorship is held constant. When using this dataset to evaluate a detector, report the authorship-matched number, or a high score may simply be measuring authorship.
import pandas as pd
df = pd.read_csv("data/human_confabulations.csv", sep=";")
# Browse a domain
finance = df[df["domain"] == "finance"]
print(f"Finance pairs: {len(finance)}")
print(finance.iloc[0]["question"])
print(finance.iloc[0]["grounded_response"])
print(finance.iloc[0]["fabricated_response"])import numpy as np
import numpy.typing as npt
import pandas as pd
from sentence_transformers import SentenceTransformer
model = SentenceTransformer("all-MiniLM-L6-v2")
df = pd.read_csv("data/human_confabulations.csv", sep=";")
questions = model.encode(df["question"].tolist())
grounded = model.encode(df["grounded_response"].tolist())
fabricated = model.encode(df["fabricated_response"].tolist())
def pairwise_cosine(a: npt.NDArray, b: npt.NDArray) -> npt.NDArray:
"""Row-wise cosine similarity between aligned embedding matrices."""
a_n = a / np.linalg.norm(a, axis=1, keepdims=True)
b_n = b / np.linalg.norm(b, axis=1, keepdims=True)
return np.sum(a_n * b_n, axis=1)
# Detection: does cos(q, grounded) > cos(q, fabricated)?
accuracy = float(np.mean(
pairwise_cosine(questions, grounded) > pairwise_cosine(questions, fabricated)
))
print(f"Detection accuracy: {accuracy:.1%}")
# Expected: ~69-78% (vs 88-97% on LLM-generated benchmarks)See scripts/ for the full experiment code, including the register-stratified and authorship-matched analyses.
.
├── README.md
├── LICENSE
├── CITATION.cff
├── DATASHEET.md # Datasheet for Datasets (Gebru et al., 2021)
├── data/
│ └── human_confabulations.csv # The 212-pair benchmark
├── paper/
│ └── paper.pdf # Methodology paper
├── scripts/
│ └── validate.py # Reproduce the detection experiment
└── examples/
└── basic_application.py # Minimal working example
If you use this dataset or methodology, please cite:
@misc{marin2026confabulation,
author = {Mar{\'\i}n, Javier},
title = {A Methodology for Building Human-Confabulated Hallucination Benchmarks},
year = {2026},
url = {https://github.com/groundlens-dev/grounding-benchmark}
}Part of a research program on geometric hallucination detection:
- The Register Wall: What Similarity-Based Hallucination Detectors Actually Measure — Marín (2026, under review). The controlled study behind the register-distance, authorship, and ceiling results above.
- A Geometric Taxonomy of Hallucinations — Marín (2026), arXiv:2602.13224. The three-mechanism taxonomy and the DGI detector.
- Semantic Grounding Index (SGI) — Marín (2025), arXiv:2512.13771. Geometric bounds on context engagement in RAG systems.
The dataset is released under Creative Commons Attribution 4.0 International (CC BY 4.0). Code is released under the Apache 2.0.