Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions eval/PREREG-desk-11.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# Round 11: round 10 again, on a Solid with every eval bug fixed and the current guide

Pre-registration, committed before any round 11 agent runs. Status: FROZEN (Ryan, 2026-11-10: "freeze").

## Why

Round 10 (`results/desk-2026-11-02`) confirmed H10a: small .rx desks had 6.7 defects, small React desks 14.7. It
rejected H10c: frontier React desks had 3.0. Most of the .rx frontier defects traced to Solid bugs, and those have
all been fixed since (#4627 #4629 #4631 #4638 #4661 #4666 #4689). Regrading round 10's own apps on the fixed Solid
moved frontier .rx from 7.3 to 4.7 raw defects. Since round 10 the guide has gained `guide/reference.md`, the rules on
one page (PR #29), and the instant model (PR #34). A 3-desk check with `reference.md` (`results/cost-2026-11-03`, not
counted) gave frontier .rx 2.0 defects. Round 11 asks whether that holds with fresh agents, all four arms, both tiers,
and the same design as round 10, so the two rounds compare directly.

Round 10's plan named maintenance as round 11. It moves to round 12 and runs on round 11's apps, so the apps it extends
were written with the current guide on a fixed Solid.

## Task and arms (as round 10 unless stated)

- Task: `tasks/desk9` with `brief-plain.md`. **One change from round 10**: the detail's `retry` button is now always
present while an issue is selected, not only after a failure. In round 10, 25 of 36 desks across all arms failed
"a detail that loads while a rename is out keeps the new title" because the brief left this unclear. Graded by
`hidden.mjs` (77) and `hidden-overlap.mjs` (38), unchanged.
- Arms: `rx` with `--inline-docs` (now `language.md`, `async.md` and `reference.md`), `foldkit` with `--inline-docs`,
`react`, `vanilla`. `rxc` is built from the commit that holds this file once it is frozen.
- Solid: `solid-next` at `next` 6751472f (2026-11-10; includes the fixes for #4689, #4724, #4737, #4738, #4739, #4748
and #4751, none yet in an rc), built from source. The same build is used for every .rx run and grade. rc.22 isn't out; if it ships before the round starts and
holds every fix, it replaces this pin.
- Harness: subagents, as in round 10. Numbers are comparable only within one harness: on the Foldkit side check
(2026-11-08), the CLI harness alone moved Foldkit from 4.7 to 2.7 defects.

## Runs, grading

As round 10: 3 frontier and 6 small runs per arm (36), one interleaved batch, at most 15 `./check`/`./test` calls,
graded by `harness/grade.sh`. The definitions of correct and cost are round 10's. **Only defects are counted**: no
legal hold and no failure the brief caused is in any count or table.

- A legal hold is a consistent hold that settles (2026-10-30 ruling). The one hidden test that asserted one (overlap,
"two issues assigned at once": it required the declined row to revert while the other assignment was still out,
which a list held whole until both settle does not do) now checks only what is a defect, that the decline does not
take the other row with it and that both rows are right once everything has answered. Committed with this file.
- A fuzz hole counts unless it is traced to a legal hold or to behaviour Solid's rules make legal; each exclusion is
listed with its trace.
- Round 10's numbers, wherever they are compared with round 11's, are regraded with the same tests, and without "a
detail that loads while a rename is out keeps the new title", which round 10's brief caused (above).

## Hypotheses and tests

- **H11a** (primary, replication of H10a): small .rx runs have fewer defects than small React runs. One-sided
permutation test on mean defects, 100,000 relabellings, seed 11.
- **H11b**: small .rx has fewer defects than small plain, and fewer than small Foldkit. Same test, Holm-corrected with
H11a.
- **H11c**: frontier .rx runs have no more defects than frontier React runs. Difference in means with a 90% bootstrap
interval (10,000 resamples, seed 11); the claim holds if the upper end is under 1 defect. (Round 10's H10c compared
small .rx with frontier React. That comparison is reported again, as H10c was.)
- **Cost per correct app** (descriptive), by arm and tier. In round 10 no app was correct, so this was undefined.
- Reported alongside: the change from round 10 for each arm and tier, defects by kind, turns, cost per run, and lines.

The direction is the claim being tested. A null or reversed result is reported as plainly.

## Gate before freezing

The 12 existing .rx desks (round 10 1-9, cost check 1-3) are regraded on the pinned Solid. A regression against
rc.21 is filed upstream and fixed or recorded before the round starts.

Result (2026-11-08): on 2789ca9c one desk had a new deterministic hole (round 10 rx-8, fuzz seed 3: a close guess
during another action's refresh, then the refresh fails). It came from the lowering, not from the app: a whole guess
(`optimistic rows := rs => rs.map(...)`) was written into the draft row by row, which replaces each matched row. It now
lowers to the setter's return, which Solid diffs shallowly against the view as guesses (its default). With that, the
other 11 desks grade as before the change (52 defects over the 12; round 10 rx-9 fuzz seed 8, a known hole, is
deterministic on both lowerings and was missed once by timing), and seed 3 is gone. The plain Solid case is solidjs/solid#4724.

Result (2026-11-10), repinned to 6751472f: regrading on the intermediate next builds (c9816808, 79b6541d) turned up
five more Solid bugs, each reduced to a plain Solid repro and fixed upstream (#4724 by #4729, #4737/#4738/#4739 by
#4740, #4748 by #4749, #4751 by #4753). On 6751472f the 12 desks total 51 defects, against 52 on 2789ca9c: no desk
gained a hole, and round 10 rx-6 fuzz seed 12 is gone. The gate passes.

## Cost

About $30 at round 10's prices: frontier $2.90 (.rx), $1.35 (React), $1.24 (plain), $3.56 (Foldkit) per run;
small under $0.16.
110 changes: 110 additions & 0 deletions eval/results/desk-2026-11-11/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,110 @@
# Round 11 results: defects and cost by model tier, four approaches (2026-11-11)

**Pre-registration:** `eval/PREREG-desk-11.md`, frozen at b8b9e0b.

**Runs:** 36, which is 4 arms × (3 frontier + 6 small). Frontier runs used the current Opus model, small runs the current Haiku model; the model is read from each transcript.

**Toolchain:** Solid `next` 6751472f, built from source; rxc from b8b9e0b.

**Grading:** `harness/grade.sh` (hidden 77, overlap 38, 20 fuzz seeds). Defects are failed hidden tests, plus failed overlap tests, plus fuzz seeds that found a hole. Only defects are counted. No fuzz hole was traced to a legal hold, so nothing is excluded.

## Defects per app

| arm | frontier mean | small mean | small per desk |
|---|---|---|---|
| .rx (inline docs) | 2.0 | **3.8** | 3, 3, 4, 2, 9, 2 |
| React | 2.0 | 9.7 | 13, 2, 17, 14, 1, 11 |
| plain TypeScript | 3.0 | 14.5 | 12, 16, 11, 9, 20, 19 |
| Foldkit (inline docs) | 2.7 | 13.2 | 17, 13, 13, 12, 10, 14 |

Frontier per desk: .rx 0, 5, 1; React 0, 2, 4; plain 3, 3, 3; Foldkit 4, 2, 2.

## Tests

All tests are on the small tier unless named otherwise; full output is in `stats.txt`.

- **H11a** (.rx < React): permutation p = 0.047, Holm 0.047. **Supported.**
- **H11b** (.rx < plain): p = 0.0021, Holm 0.0041. **Supported.**
- **H11b** (.rx < Foldkit): p = 0.0010, Holm 0.0031. **Supported.**
- **H11c** (frontier .rx no worse than frontier React): difference 0.0, 90% bootstrap interval [−2.3, 2.3]. The upper end is not under 1, so it is **not shown**. Both arms average 2.0, but three runs each are too few.
- **H10c again** (small .rx no worse than frontier React): +1.8, interval [−0.3, 4.2]. **Not shown.**

## Correct apps and cost

Two apps have no defect, the first in any round: frontier `.rx-1` and frontier `React-1`.

**Cost per correct app:** frontier .rx $6.14 and frontier React $4.15 (each arm's three runs over its one correct app). Every other cell has no correct app, so its cost per correct app is undefined.

Cost per run is billed tokens (`cost.json`) at round 10's rates (`usd.json`):

| arm | frontier $/run (round 10) | small $/run (round 10) | frontier calls | small calls |
|---|---|---|---|---|
| .rx | **2.05** (2.90) | 0.092 (0.085) | 21.3 | 15.5 |
| React | 1.38 (1.35) | 0.069 (0.066) | 12.7 | 11.8 |
| plain | 1.27 (1.24) | 0.077 (0.082) | 12.3 | 10.0 |
| Foldkit | 3.63 (3.56) | 0.175 (0.155) | 37.3 | 38.5 |

Frontier .rx cost about 30% less than in round 10. The other arms cost about the same.

## Change from round 10, same tests

Round 10's desks were regraded on this round's toolchain and tests (`r10-regraded/`). As pre-registered, the rename-reload test that round 10's brief caused is dropped from those numbers.

| arm, tier | round 10 regraded | round 11 |
|---|---|---|
| .rx frontier | 2.7 | 2.0 |
| .rx small | 4.7 | 3.8 |
| React frontier | 3.0 | 2.0 |
| React small | 13.7 | 9.7 |
| plain frontier | 2.7 | 3.0 |
| plain small | 8.8 | 14.5 |
| Foldkit frontier | 4.0 | 2.7 |
| Foldkit small | 13.5 | 13.2 |

Round 10 as first published had .rx at 6.7 in both tiers. The Solid fixes since then account for most of the drop to 2.7 and 4.7.

## Defects by kind

- **.rx:** no failure repeats in more than two desks. The overlap tests that fail in most desks of the other arms ("labels / comments that load while an add is out keep it") fail in only one .rx desk (rx-2).
- **React, plain, Foldkit (small):** most defects are overlap. Each fails 7 to 11 of 38 overlap tests on average: loads that land while a change is out drop the change, and late saves overwrite newer ones.

### .rx fuzz holes, traced

- **rx-2, seeds 1, 3, 18: a Solid bug.** A keyed projection whose function returns another keyed projection loses a row when the question switches while a guess is out. In a browser this shows as a duplicated row. The plain Solid repro is filed as solidjs/solid#4762. It counts against .rx, as pre-registered.
- **rx-5, rx-6, rx-7, the "row assignee" holes: agent mistakes.** The assign guess is written only when the row is in the list on screen (`if (i >= 0)`), so a row hidden by a close guess never gets it.
- **rx-5 and rx-8, the "detail title" holes: agent mistakes.** rx-5 seed 15 was replayed; rx-8 has the same code shape and was not replayed. The rename guess is written to a source derived from the detail before the detail has loaded. Solid drops a guess that was never shown (D-34), so the old title stays on screen while the rename is out.
- **rx-4 seed 15 (detail title): agent mistake.** Rename guesses are kept in their own list, and the heading reads it with `find`, which returns the oldest guess. With two renames out, the heading shows the first one, while the row shows the second.

## Lines of source per app

Non-blank lines, without the given `api` and `flights` files.

| arm | frontier | small |
|---|---|---|
| .rx | 414 | 373 |
| React | 784 | 744 |
| plain | 873 | 967 |
| Foldkit | 2276 | 2340 |

## Deviations

- foldkit-6 and foldkit-7 ended without a final report and were graded as they stood.
- rx-5 used 16 check/test runs, one over the cap of 15.
- react-5 listed the file names in `eval/` (it read no contents) and said so.
- Three agents (rx-1, rx-2, react-4) added temporary visible tests and then restored the file.
- None of these affects grading.

## Reading

- **Small tier:** the round 10 result replicates. Small .rx writes fewer than half of React's defects and about a quarter of plain's and Foldkit's.
- **Frontier tier:** all four arms are now close, between 2 and 3 defects. .rx is level with React, but three runs per arm cannot show "no worse".
- **Cost:** frontier .rx is still dearer than React and plain ($2.05 against about $1.30) but cheaper than Foldkit. With 3 of its 6 frontier defects from one Solid bug, the frontier .rx count is partly Solid's still.

## Files

- `summary.txt`: per desk.
- `graded/`: `grade.sh` output.
- `r10-regraded/`: round 10's desks on this toolchain.
- `stats.py`, `stats.txt`.
- `cost.json`, `usd.json`.
- `agents-11.txt`: desk to transcript.
36 changes: 36 additions & 0 deletions eval/results/desk-2026-11-11/agents-11.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
desk11-rx-1 a84e1656cb0ba6df6
desk11-react-1 a2f24068924bc4fa4
desk11-vanilla-1 afe87582e2a06182e
desk11-foldkit-1 adefd18cee0357245
desk11-rx-4 aa00b48b4d63e1f4f
desk11-react-4 a9ad76d0fefab4fb9
desk11-vanilla-4 a44863179ca8f66d2
desk11-foldkit-4 ae41cd1296a281f94
desk11-rx-2 a22838ba99d99ae05
desk11-react-2 ae4cfba0be84e5fde
desk11-vanilla-2 abbd5e1185c32e849
desk11-foldkit-2 a86728ef1f49707e3
desk11-rx-5 aca8962526f6412eb
desk11-react-5 accf3998eda68b899
desk11-vanilla-5 acdf73c2cc217936e
desk11-foldkit-5 a267361eee853d3d5
desk11-rx-6 aacc6363979cdbc04
desk11-react-6 a7113a69ad5ba8dc5
desk11-vanilla-6 a3aa74d9de8c92134
desk11-foldkit-6 a6f3d7ae6bddace40
desk11-rx-3 a981489adbd708190
desk11-react-3 afd3f0953f7a4f889
desk11-vanilla-3 a3a9a9c11a9591692
desk11-foldkit-3 abf32ed818e83da60
desk11-rx-7 ac6e74d314c0c60b7
desk11-react-7 a354f1917af348048
desk11-vanilla-7 ad6fbfda274aef3d6
desk11-foldkit-7 afe411d8be72215e2
desk11-rx-8 ad97b6d0dc3a855a3
desk11-react-8 adef2bade38df0577
desk11-vanilla-8 ae61d4a5ff5667c62
desk11-foldkit-8 a68e6145c8637d77a
desk11-rx-9 aa3711f9a5483d068
desk11-react-9 adf12270373046254
desk11-vanilla-9 ac7ec2aeac924ce56
desk11-foldkit-9 aafaabb4bcd01167f
Loading
Loading