Repository navigation
harness: the suite runner identifies "the time" positionally, so a fixture whose first labelled line is a ratio scores as parity #9374
Description
Activity
Important correction after reading the fixture: this is not a fixture bug, and that makes the runner's part worse rather than better.
bench_typed_array_untyped_access.tsemits the ratio first on purpose, and says so:The untyped/typed ratio is the #5525 metric: it normalizes out machine speed (Node ~= 1.0 — no gap; Perry > 1 until static typed-array-kind inference lands). Emit it first so a single-line metric consumer (the suite runner reads only the first
label:numberline) tracks the gap, not absolute time.So the fixture and the runner agree perfectly, and the quantity they agree on is sound for its stated purpose: perry-vs-perry regression tracking of the #5525 gap. What is invalid is the step neither of them takes responsibility for — feeding that number into a perry-vs-node comparison, where a ratio of two perry timings against a ratio of two node timings says nothing at all.
That changes the fix. Do NOT rename the fixture's first line: it is load-bearing for the regression metric it was designed for, and renaming it would silently break that. Instead:
- The runner must distinguish "this row's metric is a time" from "this row's metric is a ratio." A parity sweep may only score rows of the first kind; a ratio row is
NOT-COMPARABLEin a cross-runtime table and should be reported as such, with its absolute lines (ta_untyped_access/ta_typed_access) scored instead when a cross-runtime number is wanted. - Reject non-integer values in a millisecond column. A peer's suggestion and the cheapest possible guard:
0.9396551724137931is not a plausible elapsed-millisecond count, and that one check would have caught this on day one across every fixture at once. - Fail loudly on no time-labelled line rather than falling back to position.
An audit of all 30 fixtures' first lines found exactly two non-time rows: this one and
01_startup(which printsstartedand must be timed externally — measured properly it is perry 7.59 ms median vs node 90.72 ms, 12x faster, not the "0 ms" a truncatingtime -preported).Also worth recording from the same audit, the mirror-image hazard:
02_loop_overheadand08_string_concatnow print perry=0 ms against node's 63 ms and 8 ms. Genuine wins, but those fixtures have hit their own resolution floor — a future regression of up to 1 ms is invisible, and further improvement is unmeasurable. Same family as #9338 arriving from the winning side; they need a larger N to remain useful as guards.- The runner must distinguish "this row's metric is a time" from "this row's metric is a ratio." A parity sweep may only score rows of the first kind; a ratio row is
- added a commit that references this issue
on Sep 1, 2026
Filed after two independent sweeps (mine and a peer session's) both scored
bench_typed_array_untyped_accessas parity when it is a 7.7x loss.What happened
Every runner in use extracts the benchmark's time as the first
label:numberline:For that fixture the first labelled line is
ta_untyped_typed_ratio— a perry-to-perry ratio (0.945 for perry, 0.883 for node), deliberately so, because the fixture's own header names the untyped/typed ratio as the #5525 metric it exists to protect. Comparing perry's ratio to node's ratio is meaningless across runtimes, and it happens to land near 1.0, so the row scored 1.00-1.07x.The absolute numbers, same box:
ta_untyped_accessta_typed_accessWhy this is worse than the zero-millisecond rows (#9338)
A
0msrow is obviously wrong and invites a second look. This one prints a plausible number in the right range, so nothing prompts one. Two sessions independently mis-scored it for days. A harness that cannot distinguish "parity" from "I read the wrong number" is worse than no harness, because it manufactures confidence.Proposed fix
<name>:<ms>where<name>matches the benchmark, or a dedicatedelapsed_ms:line), nothead -1.UNSCOREABLE, not a number. Silence is the bug; an error is a finding.bench_typed_array_untyped_access.ts(e.g.ratio_untyped_over_typed:) so it cannot be mistaken for a duration even by a positional reader, and keep the absoluteta_untyped_access/ta_typed_accesslines as the scored ones.Related: #9338 (fixtures whose work is dead-code-eliminated, scoring 0 ms). Same family — the runner trusting a number it has not established is a time.
The 7.7x row itself is being taken separately as a typed-array kind-dispatch issue (my lane, adjacent to #9342/#9360); this issue is only about the measurement apparatus that hid it.