[aitianshi]

Closes #131

What

GSM8K gains a second dataset, GSM8K-Platinum: the test split with its errors cleaned (Vendrow et al., 2025; 110 problems removed, 10 answers corrected), beside the original split, so the same model can be run on each and the difference the benchmark's own errors make can be read.

• Pin. The file is main/test-00000-of-00001.parquet of madrylab/gsm8k-platinum at revision e762492, checked by its sha256, downloaded by Nullius on the first run as any url dataset is.
• Unit ids. A Platinum problem keeps the number of the test file's problem it is. All 1,209 Platinum questions are byte-identical to a question of the original test.jsonl, each once and in the same order, so the driver finds the number by the question. Platinum's ids are therefore a subset of the original's, with gaps where a problem was removed; unit 0289 is the same problem in both. A Platinum question the test file lacks is an error, never a new number.
• Check. Unchanged: the upstream's own extract_answer and is_correct, applied to Platinum's answer field, which has the same #### <number> format.
• Driver. The only code of ours is loading glue: read the parquet file, name each row by its number in the test file. Its environment gains pyarrow==26.0.1, because Platinum is published as parquet only.
• Entry. task_count 1319 so the headline is the set of problems, not the sum of two overlapping datasets; the authors' citation beside GSM8K's, as they ask; one Good to know line on the shared numbering.

Touches

catalogue · the standard (GSM8K's definition and driver) · docs and site (the generated GSM8K page) · tests

Verified

• python3 app/tests/parallel.py: 1988 tests in 116 modules, OK. Route suite (app/ui_tests): 422 tests in 40 modules, OK. catalogue_docs.py --write is clean.
• nullius standard prove gsm8k --dataset platinum, all 1,209 problems, in a temporary profile: units stable and unique, no leak 1209/1209, nop 0/1209, oracle 1209/1209. That record is the proven.platinum block of this diff.
• prove gsm8k --dataset test against the changed driver, all 1,319: same ids, nop 0/1319, oracle 1319/1319. Its recorded proof is not touched.
• Both datasets loaded through the driver: 1,319 and 1,209 ids; every Platinum id is an original id (110 absent); the prompt under the same id is identical for all 1,209.
• The Original setup run end to end on problem 0289 (an answer Platinum revised from 3 to 4) against a local stand-in model that always answers #### 4: reward 0.0 on test, 1.0 on platinum. A --units file naming a problem Platinum removed is refused on Platinum by name.
• Browser, 1280 wide, light theme: the builder's Dataset step for GSM8K shows two cards, GSM8K test (selected) and GSM8K-Platinum, each with its four checks; the benchmark page lists both datasets, both proofs and both citations.
• The new driver test (test_a_platinum_problem_keeps_its_number_in_the_test_file) passes where GSM8K's upstream and driver environment are installed; it is skipped elsewhere, like the driver tests beside it.
• A real run of the Original setup on each whole dataset, meta/muse-spark-1.3-contributor through OpenRouter, provider default temperature: 1292/1319 (98.0%) on test, 1204/1209 (99.6%) on platinum. Of the 27 misses on test, 18 are problems Platinum removed, 7 are answers Platinum revised to what the model gave, and 2 passed on the Platinum run with an unchanged reference (the model answered differently). Of the 5 misses on platinum, 3 are revised answers where the model gave the original one, and 2 passed on the test run with an unchanged reference. The two jobs compared by unit id with no mapping.
• Not verified: a run on a Settings host or in the container backend, the dark theme and narrow widths. No before and after images are attached.

Risk

GSM8K's driver environment changes (pyarrow added), so the first run after this rebuilds it. If the pinned Hugging Face file became unavailable, only the Platinum dataset would fail to start, at the download, with the original split unaffected. test_agents pinned GSM8K as the one-dataset example of the proof form; it now checks the selector on GSM8K and the fixed value on HumanEval. Undone by a revert; no stored data changes shape.

[github-actions]

🟡 PR judge · 7 of 7 yes · #132 ↔ #131

Merge risk	🔴 MAJOR	the judge's reading of the diff
Priority	🟡 MEDIUM	the deepest path is app/nullius/catalogue/benchmarks/gsm8k/benchmark.json

Question	Verdict
Changes do what the issue asks · 8 of 8 criteria	✅ yes
Issue and pull request describe the same change	✅ yes
Description matches the diff, no more, no less	✅ yes
Diff stays in scope	✅ yes
Tests cover the change, or Verified says how	✅ yes
Docs follow the change	✅ yes
Risk named in the description	✅ yes
Before and after shown	–

Acceptance criteria, 8 of 8 met

#	Criterion	Verdict
1	The GSM8K entry lists two datasets, the original test split and GSM8K-Platinum, and the builder offers both for GSM8K.	✅ met
2	The Platinum file is pinned: a fixed revision of madrylab/gsm8k-platinum in its address and the file's sha256.	✅ met
3	A Platinum unit carries the id of the same problem in the original test split (its line number there), and the choice i…	✅ met
4	The score of both datasets is the upstream's own extract_answer and is_correct, with no scoring code of Nullius's.	✅ met
5	The original test dataset's units, ids and proofs are unchanged.	✅ met
6	The Platinum dataset's controls are proven (units, leak, nop, oracle) or the entry says plainly that they are not.	✅ met
7	The Supported pages and the docs that describe GSM8K name the new dataset (catalogue_docs.py --write is clean).	✅ met
8	The backend and route suites pass.	✅ met

Checks the script runs itself, 4 of 4 pass

No machine attribution trailer in the commits or the description	✓
New code under app/, docs/, .github/ or .claude/	✓
No Flask import under the backend package	✓
No key-shaped string in the diff	✓