|
| 1 | +# Round 11 results: defects and cost by model tier, four approaches (2026-11-11) |
| 2 | + |
| 3 | +**Pre-registration:** `eval/PREREG-desk-11.md`, frozen at b8b9e0b. |
| 4 | + |
| 5 | +**Runs:** 36, which is 4 arms × (3 frontier + 6 small). Frontier runs used the current Opus model, small runs the current Haiku model; the model is read from each transcript. |
| 6 | + |
| 7 | +**Toolchain:** Solid `next` 6751472f, built from source; rxc from b8b9e0b. |
| 8 | + |
| 9 | +**Grading:** `harness/grade.sh` (hidden 77, overlap 38, 20 fuzz seeds). Defects are failed hidden tests, plus failed overlap tests, plus fuzz seeds that found a hole. Only defects are counted. No fuzz hole was traced to a legal hold, so nothing is excluded. |
| 10 | + |
| 11 | +## Defects per app |
| 12 | + |
| 13 | +| arm | frontier mean | small mean | small per desk | |
| 14 | +|---|---|---|---| |
| 15 | +| .rx (inline docs) | 2.0 | **3.8** | 3, 3, 4, 2, 9, 2 | |
| 16 | +| React | 2.0 | 9.7 | 13, 2, 17, 14, 1, 11 | |
| 17 | +| plain TypeScript | 3.0 | 14.5 | 12, 16, 11, 9, 20, 19 | |
| 18 | +| Foldkit (inline docs) | 2.7 | 13.2 | 17, 13, 13, 12, 10, 14 | |
| 19 | + |
| 20 | +Frontier per desk: .rx 0, 5, 1; React 0, 2, 4; plain 3, 3, 3; Foldkit 4, 2, 2. |
| 21 | + |
| 22 | +## Tests |
| 23 | + |
| 24 | +All tests are on the small tier unless named otherwise; full output is in `stats.txt`. |
| 25 | + |
| 26 | +- **H11a** (.rx < React): permutation p = 0.047, Holm 0.047. **Supported.** |
| 27 | +- **H11b** (.rx < plain): p = 0.0021, Holm 0.0041. **Supported.** |
| 28 | +- **H11b** (.rx < Foldkit): p = 0.0010, Holm 0.0031. **Supported.** |
| 29 | +- **H11c** (frontier .rx no worse than frontier React): difference 0.0, 90% bootstrap interval [−2.3, 2.3]. The upper end is not under 1, so it is **not shown**. Both arms average 2.0, but three runs each are too few. |
| 30 | +- **H10c again** (small .rx no worse than frontier React): +1.8, interval [−0.3, 4.2]. **Not shown.** |
| 31 | + |
| 32 | +## Correct apps and cost |
| 33 | + |
| 34 | +Two apps have no defect, the first in any round: frontier `.rx-1` and frontier `React-1`. |
| 35 | + |
| 36 | +**Cost per correct app:** frontier .rx $6.14 and frontier React $4.15 (each arm's three runs over its one correct app). Every other cell has no correct app, so its cost per correct app is undefined. |
| 37 | + |
| 38 | +Cost per run is billed tokens (`cost.json`) at round 10's rates (`usd.json`): |
| 39 | + |
| 40 | +| arm | frontier $/run (round 10) | small $/run (round 10) | frontier calls | small calls | |
| 41 | +|---|---|---|---|---| |
| 42 | +| .rx | **2.05** (2.90) | 0.092 (0.085) | 21.3 | 15.5 | |
| 43 | +| React | 1.38 (1.35) | 0.069 (0.066) | 12.7 | 11.8 | |
| 44 | +| plain | 1.27 (1.24) | 0.077 (0.082) | 12.3 | 10.0 | |
| 45 | +| Foldkit | 3.63 (3.56) | 0.175 (0.155) | 37.3 | 38.5 | |
| 46 | + |
| 47 | +Frontier .rx cost about 30% less than in round 10. The other arms cost about the same. |
| 48 | + |
| 49 | +## Change from round 10, same tests |
| 50 | + |
| 51 | +Round 10's desks were regraded on this round's toolchain and tests (`r10-regraded/`). As pre-registered, the rename-reload test that round 10's brief caused is dropped from those numbers. |
| 52 | + |
| 53 | +| arm, tier | round 10 regraded | round 11 | |
| 54 | +|---|---|---| |
| 55 | +| .rx frontier | 2.7 | 2.0 | |
| 56 | +| .rx small | 4.7 | 3.8 | |
| 57 | +| React frontier | 3.0 | 2.0 | |
| 58 | +| React small | 13.7 | 9.7 | |
| 59 | +| plain frontier | 2.7 | 3.0 | |
| 60 | +| plain small | 8.8 | 14.5 | |
| 61 | +| Foldkit frontier | 4.0 | 2.7 | |
| 62 | +| Foldkit small | 13.5 | 13.2 | |
| 63 | + |
| 64 | +Round 10 as first published had .rx at 6.7 in both tiers. The Solid fixes since then account for most of the drop to 2.7 and 4.7. |
| 65 | + |
| 66 | +## Defects by kind |
| 67 | + |
| 68 | +- **.rx:** no failure repeats in more than two desks. The overlap tests that fail in most desks of the other arms ("labels / comments that load while an add is out keep it") fail in only one .rx desk (rx-2). |
| 69 | +- **React, plain, Foldkit (small):** most defects are overlap. Each fails 7 to 11 of 38 overlap tests on average: loads that land while a change is out drop the change, and late saves overwrite newer ones. |
| 70 | + |
| 71 | +### .rx fuzz holes, traced |
| 72 | + |
| 73 | +- **rx-2, seeds 1, 3, 18: a Solid bug.** A keyed projection whose function returns another keyed projection loses a row when the question switches while a guess is out. In a browser this shows as a duplicated row. The plain Solid repro is filed as solidjs/solid#4762. It counts against .rx, as pre-registered. |
| 74 | +- **rx-5, rx-6, rx-7, the "row assignee" holes: agent mistakes.** The assign guess is written only when the row is in the list on screen (`if (i >= 0)`), so a row hidden by a close guess never gets it. |
| 75 | +- **rx-5 and rx-8, the "detail title" holes: agent mistakes.** rx-5 seed 15 was replayed; rx-8 has the same code shape and was not replayed. The rename guess is written to a source derived from the detail before the detail has loaded. Solid drops a guess that was never shown (D-34), so the old title stays on screen while the rename is out. |
| 76 | +- **rx-4 seed 15 (detail title): agent mistake.** Rename guesses are kept in their own list, and the heading reads it with `find`, which returns the oldest guess. With two renames out, the heading shows the first one, while the row shows the second. |
| 77 | + |
| 78 | +## Lines of source per app |
| 79 | + |
| 80 | +Non-blank lines, without the given `api` and `flights` files. |
| 81 | + |
| 82 | +| arm | frontier | small | |
| 83 | +|---|---|---| |
| 84 | +| .rx | 414 | 373 | |
| 85 | +| React | 784 | 744 | |
| 86 | +| plain | 873 | 967 | |
| 87 | +| Foldkit | 2276 | 2340 | |
| 88 | + |
| 89 | +## Deviations |
| 90 | + |
| 91 | +- foldkit-6 and foldkit-7 ended without a final report and were graded as they stood. |
| 92 | +- rx-5 used 16 check/test runs, one over the cap of 15. |
| 93 | +- react-5 listed the file names in `eval/` (it read no contents) and said so. |
| 94 | +- Three agents (rx-1, rx-2, react-4) added temporary visible tests and then restored the file. |
| 95 | +- None of these affects grading. |
| 96 | + |
| 97 | +## Reading |
| 98 | + |
| 99 | +- **Small tier:** the round 10 result replicates. Small .rx writes fewer than half of React's defects and about a quarter of plain's and Foldkit's. |
| 100 | +- **Frontier tier:** all four arms are now close, between 2 and 3 defects. .rx is level with React, but three runs per arm cannot show "no worse". |
| 101 | +- **Cost:** frontier .rx is still dearer than React and plain ($2.05 against about $1.30) but cheaper than Foldkit. With 3 of its 6 frontier defects from one Solid bug, the frontier .rx count is partly Solid's still. |
| 102 | + |
| 103 | +## Files |
| 104 | + |
| 105 | +- `summary.txt`: per desk. |
| 106 | +- `graded/`: `grade.sh` output. |
| 107 | +- `r10-regraded/`: round 10's desks on this toolchain. |
| 108 | +- `stats.py`, `stats.txt`. |
| 109 | +- `cost.json`, `usd.json`. |
| 110 | +- `agents-11.txt`: desk to transcript. |
0 commit comments