Skip to content

Commit b90331e

Browse files
committed
Merge remote-tracking branch 'origin/main' into claude/project-thread-2sdg89
# Conflicts: # ref/oracle/lower.ts
2 parents bffcff8 + 46ede7a commit b90331e

85 files changed

Lines changed: 1918 additions & 5 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎eval/PREREG-desk-11.md‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Round 11: round 10 again, on a Solid with every eval bug fixed and the current guide
2+
3+
Pre-registration, committed before any round 11 agent runs. Status: FROZEN (Ryan, 2026-11-10: "freeze").
4+
5+
## Why
6+
7+
Round 10 (`results/desk-2026-11-02`) confirmed H10a: small .rx desks had 6.7 defects, small React desks 14.7. It
8+
rejected H10c: frontier React desks had 3.0. Most of the .rx frontier defects traced to Solid bugs, and those have
9+
all been fixed since (#4627 #4629 #4631 #4638 #4661 #4666 #4689). Regrading round 10's own apps on the fixed Solid
10+
moved frontier .rx from 7.3 to 4.7 raw defects. Since round 10 the guide has gained `guide/reference.md`, the rules on
11+
one page (PR #29), and the instant model (PR #34). A 3-desk check with `reference.md` (`results/cost-2026-11-03`, not
12+
counted) gave frontier .rx 2.0 defects. Round 11 asks whether that holds with fresh agents, all four arms, both tiers,
13+
and the same design as round 10, so the two rounds compare directly.
14+
15+
Round 10's plan named maintenance as round 11. It moves to round 12 and runs on round 11's apps, so the apps it extends
16+
were written with the current guide on a fixed Solid.
17+
18+
## Task and arms (as round 10 unless stated)
19+
20+
- Task: `tasks/desk9` with `brief-plain.md`. **One change from round 10**: the detail's `retry` button is now always
21+
present while an issue is selected, not only after a failure. In round 10, 25 of 36 desks across all arms failed
22+
"a detail that loads while a rename is out keeps the new title" because the brief left this unclear. Graded by
23+
`hidden.mjs` (77) and `hidden-overlap.mjs` (38), unchanged.
24+
- Arms: `rx` with `--inline-docs` (now `language.md`, `async.md` and `reference.md`), `foldkit` with `--inline-docs`,
25+
`react`, `vanilla`. `rxc` is built from the commit that holds this file once it is frozen.
26+
- Solid: `solid-next` at `next` 6751472f (2026-11-10; includes the fixes for #4689, #4724, #4737, #4738, #4739, #4748
27+
and #4751, none yet in an rc), built from source. The same build is used for every .rx run and grade. rc.22 isn't out; if it ships before the round starts and
28+
holds every fix, it replaces this pin.
29+
- Harness: subagents, as in round 10. Numbers are comparable only within one harness: on the Foldkit side check
30+
(2026-11-08), the CLI harness alone moved Foldkit from 4.7 to 2.7 defects.
31+
32+
## Runs, grading
33+
34+
As round 10: 3 frontier and 6 small runs per arm (36), one interleaved batch, at most 15 `./check`/`./test` calls,
35+
graded by `harness/grade.sh`. The definitions of correct and cost are round 10's. **Only defects are counted**: no
36+
legal hold and no failure the brief caused is in any count or table.
37+
38+
- A legal hold is a consistent hold that settles (2026-10-30 ruling). The one hidden test that asserted one (overlap,
39+
"two issues assigned at once": it required the declined row to revert while the other assignment was still out,
40+
which a list held whole until both settle does not do) now checks only what is a defect, that the decline does not
41+
take the other row with it and that both rows are right once everything has answered. Committed with this file.
42+
- A fuzz hole counts unless it is traced to a legal hold or to behaviour Solid's rules make legal; each exclusion is
43+
listed with its trace.
44+
- Round 10's numbers, wherever they are compared with round 11's, are regraded with the same tests, and without "a
45+
detail that loads while a rename is out keeps the new title", which round 10's brief caused (above).
46+
47+
## Hypotheses and tests
48+
49+
- **H11a** (primary, replication of H10a): small .rx runs have fewer defects than small React runs. One-sided
50+
permutation test on mean defects, 100,000 relabellings, seed 11.
51+
- **H11b**: small .rx has fewer defects than small plain, and fewer than small Foldkit. Same test, Holm-corrected with
52+
H11a.
53+
- **H11c**: frontier .rx runs have no more defects than frontier React runs. Difference in means with a 90% bootstrap
54+
interval (10,000 resamples, seed 11); the claim holds if the upper end is under 1 defect. (Round 10's H10c compared
55+
small .rx with frontier React. That comparison is reported again, as H10c was.)
56+
- **Cost per correct app** (descriptive), by arm and tier. In round 10 no app was correct, so this was undefined.
57+
- Reported alongside: the change from round 10 for each arm and tier, defects by kind, turns, cost per run, and lines.
58+
59+
The direction is the claim being tested. A null or reversed result is reported as plainly.
60+
61+
## Gate before freezing
62+
63+
The 12 existing .rx desks (round 10 1-9, cost check 1-3) are regraded on the pinned Solid. A regression against
64+
rc.21 is filed upstream and fixed or recorded before the round starts.
65+
66+
Result (2026-11-08): on 2789ca9c one desk had a new deterministic hole (round 10 rx-8, fuzz seed 3: a close guess
67+
during another action's refresh, then the refresh fails). It came from the lowering, not from the app: a whole guess
68+
(`optimistic rows := rs => rs.map(...)`) was written into the draft row by row, which replaces each matched row. It now
69+
lowers to the setter's return, which Solid diffs shallowly against the view as guesses (its default). With that, the
70+
other 11 desks grade as before the change (52 defects over the 12; round 10 rx-9 fuzz seed 8, a known hole, is
71+
deterministic on both lowerings and was missed once by timing), and seed 3 is gone. The plain Solid case is solidjs/solid#4724.
72+
73+
Result (2026-11-10), repinned to 6751472f: regrading on the intermediate next builds (c9816808, 79b6541d) turned up
74+
five more Solid bugs, each reduced to a plain Solid repro and fixed upstream (#4724 by #4729, #4737/#4738/#4739 by
75+
#4740, #4748 by #4749, #4751 by #4753). On 6751472f the 12 desks total 51 defects, against 52 on 2789ca9c: no desk
76+
gained a hole, and round 10 rx-6 fuzz seed 12 is gone. The gate passes.
77+
78+
## Cost
79+
80+
About $30 at round 10's prices: frontier $2.90 (.rx), $1.35 (React), $1.24 (plain), $3.56 (Foldkit) per run;
81+
small under $0.16.
Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Round 11 results: defects and cost by model tier, four approaches (2026-11-11)
2+
3+
**Pre-registration:** `eval/PREREG-desk-11.md`, frozen at b8b9e0b.
4+
5+
**Runs:** 36, which is 4 arms × (3 frontier + 6 small). Frontier runs used the current Opus model, small runs the current Haiku model; the model is read from each transcript.
6+
7+
**Toolchain:** Solid `next` 6751472f, built from source; rxc from b8b9e0b.
8+
9+
**Grading:** `harness/grade.sh` (hidden 77, overlap 38, 20 fuzz seeds). Defects are failed hidden tests, plus failed overlap tests, plus fuzz seeds that found a hole. Only defects are counted. No fuzz hole was traced to a legal hold, so nothing is excluded.
10+
11+
## Defects per app
12+
13+
| arm | frontier mean | small mean | small per desk |
14+
|---|---|---|---|
15+
| .rx (inline docs) | 2.0 | **3.8** | 3, 3, 4, 2, 9, 2 |
16+
| React | 2.0 | 9.7 | 13, 2, 17, 14, 1, 11 |
17+
| plain TypeScript | 3.0 | 14.5 | 12, 16, 11, 9, 20, 19 |
18+
| Foldkit (inline docs) | 2.7 | 13.2 | 17, 13, 13, 12, 10, 14 |
19+
20+
Frontier per desk: .rx 0, 5, 1; React 0, 2, 4; plain 3, 3, 3; Foldkit 4, 2, 2.
21+
22+
## Tests
23+
24+
All tests are on the small tier unless named otherwise; full output is in `stats.txt`.
25+
26+
- **H11a** (.rx < React): permutation p = 0.047, Holm 0.047. **Supported.**
27+
- **H11b** (.rx < plain): p = 0.0021, Holm 0.0041. **Supported.**
28+
- **H11b** (.rx < Foldkit): p = 0.0010, Holm 0.0031. **Supported.**
29+
- **H11c** (frontier .rx no worse than frontier React): difference 0.0, 90% bootstrap interval [−2.3, 2.3]. The upper end is not under 1, so it is **not shown**. Both arms average 2.0, but three runs each are too few.
30+
- **H10c again** (small .rx no worse than frontier React): +1.8, interval [−0.3, 4.2]. **Not shown.**
31+
32+
## Correct apps and cost
33+
34+
Two apps have no defect, the first in any round: frontier `.rx-1` and frontier `React-1`.
35+
36+
**Cost per correct app:** frontier .rx $6.14 and frontier React $4.15 (each arm's three runs over its one correct app). Every other cell has no correct app, so its cost per correct app is undefined.
37+
38+
Cost per run is billed tokens (`cost.json`) at round 10's rates (`usd.json`):
39+
40+
| arm | frontier $/run (round 10) | small $/run (round 10) | frontier calls | small calls |
41+
|---|---|---|---|---|
42+
| .rx | **2.05** (2.90) | 0.092 (0.085) | 21.3 | 15.5 |
43+
| React | 1.38 (1.35) | 0.069 (0.066) | 12.7 | 11.8 |
44+
| plain | 1.27 (1.24) | 0.077 (0.082) | 12.3 | 10.0 |
45+
| Foldkit | 3.63 (3.56) | 0.175 (0.155) | 37.3 | 38.5 |
46+
47+
Frontier .rx cost about 30% less than in round 10. The other arms cost about the same.
48+
49+
## Change from round 10, same tests
50+
51+
Round 10's desks were regraded on this round's toolchain and tests (`r10-regraded/`). As pre-registered, the rename-reload test that round 10's brief caused is dropped from those numbers.
52+
53+
| arm, tier | round 10 regraded | round 11 |
54+
|---|---|---|
55+
| .rx frontier | 2.7 | 2.0 |
56+
| .rx small | 4.7 | 3.8 |
57+
| React frontier | 3.0 | 2.0 |
58+
| React small | 13.7 | 9.7 |
59+
| plain frontier | 2.7 | 3.0 |
60+
| plain small | 8.8 | 14.5 |
61+
| Foldkit frontier | 4.0 | 2.7 |
62+
| Foldkit small | 13.5 | 13.2 |
63+
64+
Round 10 as first published had .rx at 6.7 in both tiers. The Solid fixes since then account for most of the drop to 2.7 and 4.7.
65+
66+
## Defects by kind
67+
68+
- **.rx:** no failure repeats in more than two desks. The overlap tests that fail in most desks of the other arms ("labels / comments that load while an add is out keep it") fail in only one .rx desk (rx-2).
69+
- **React, plain, Foldkit (small):** most defects are overlap. Each fails 7 to 11 of 38 overlap tests on average: loads that land while a change is out drop the change, and late saves overwrite newer ones.
70+
71+
### .rx fuzz holes, traced
72+
73+
- **rx-2, seeds 1, 3, 18: a Solid bug.** A keyed projection whose function returns another keyed projection loses a row when the question switches while a guess is out. In a browser this shows as a duplicated row. The plain Solid repro is filed as solidjs/solid#4762. It counts against .rx, as pre-registered.
74+
- **rx-5, rx-6, rx-7, the "row assignee" holes: agent mistakes.** The assign guess is written only when the row is in the list on screen (`if (i >= 0)`), so a row hidden by a close guess never gets it.
75+
- **rx-5 and rx-8, the "detail title" holes: agent mistakes.** rx-5 seed 15 was replayed; rx-8 has the same code shape and was not replayed. The rename guess is written to a source derived from the detail before the detail has loaded. Solid drops a guess that was never shown (D-34), so the old title stays on screen while the rename is out.
76+
- **rx-4 seed 15 (detail title): agent mistake.** Rename guesses are kept in their own list, and the heading reads it with `find`, which returns the oldest guess. With two renames out, the heading shows the first one, while the row shows the second.
77+
78+
## Lines of source per app
79+
80+
Non-blank lines, without the given `api` and `flights` files.
81+
82+
| arm | frontier | small |
83+
|---|---|---|
84+
| .rx | 414 | 373 |
85+
| React | 784 | 744 |
86+
| plain | 873 | 967 |
87+
| Foldkit | 2276 | 2340 |
88+
89+
## Deviations
90+
91+
- foldkit-6 and foldkit-7 ended without a final report and were graded as they stood.
92+
- rx-5 used 16 check/test runs, one over the cap of 15.
93+
- react-5 listed the file names in `eval/` (it read no contents) and said so.
94+
- Three agents (rx-1, rx-2, react-4) added temporary visible tests and then restored the file.
95+
- None of these affects grading.
96+
97+
## Reading
98+
99+
- **Small tier:** the round 10 result replicates. Small .rx writes fewer than half of React's defects and about a quarter of plain's and Foldkit's.
100+
- **Frontier tier:** all four arms are now close, between 2 and 3 defects. .rx is level with React, but three runs per arm cannot show "no worse".
101+
- **Cost:** frontier .rx is still dearer than React and plain ($2.05 against about $1.30) but cheaper than Foldkit. With 3 of its 6 frontier defects from one Solid bug, the frontier .rx count is partly Solid's still.
102+
103+
## Files
104+
105+
- `summary.txt`: per desk.
106+
- `graded/`: `grade.sh` output.
107+
- `r10-regraded/`: round 10's desks on this toolchain.
108+
- `stats.py`, `stats.txt`.
109+
- `cost.json`, `usd.json`.
110+
- `agents-11.txt`: desk to transcript.
Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
desk11-rx-1 a84e1656cb0ba6df6
2+
desk11-react-1 a2f24068924bc4fa4
3+
desk11-vanilla-1 afe87582e2a06182e
4+
desk11-foldkit-1 adefd18cee0357245
5+
desk11-rx-4 aa00b48b4d63e1f4f
6+
desk11-react-4 a9ad76d0fefab4fb9
7+
desk11-vanilla-4 a44863179ca8f66d2
8+
desk11-foldkit-4 ae41cd1296a281f94
9+
desk11-rx-2 a22838ba99d99ae05
10+
desk11-react-2 ae4cfba0be84e5fde
11+
desk11-vanilla-2 abbd5e1185c32e849
12+
desk11-foldkit-2 a86728ef1f49707e3
13+
desk11-rx-5 aca8962526f6412eb
14+
desk11-react-5 accf3998eda68b899
15+
desk11-vanilla-5 acdf73c2cc217936e
16+
desk11-foldkit-5 a267361eee853d3d5
17+
desk11-rx-6 aacc6363979cdbc04
18+
desk11-react-6 a7113a69ad5ba8dc5
19+
desk11-vanilla-6 a3aa74d9de8c92134
20+
desk11-foldkit-6 a6f3d7ae6bddace40
21+
desk11-rx-3 a981489adbd708190
22+
desk11-react-3 afd3f0953f7a4f889
23+
desk11-vanilla-3 a3a9a9c11a9591692
24+
desk11-foldkit-3 abf32ed818e83da60
25+
desk11-rx-7 ac6e74d314c0c60b7
26+
desk11-react-7 a354f1917af348048
27+
desk11-vanilla-7 ad6fbfda274aef3d6
28+
desk11-foldkit-7 afe411d8be72215e2
29+
desk11-rx-8 ad97b6d0dc3a855a3
30+
desk11-react-8 adef2bade38df0577
31+
desk11-vanilla-8 ae61d4a5ff5667c62
32+
desk11-foldkit-8 a68e6145c8637d77a
33+
desk11-rx-9 aa3711f9a5483d068
34+
desk11-react-9 adf12270373046254
35+
desk11-vanilla-9 ac7ec2aeac924ce56
36+
desk11-foldkit-9 aafaabb4bcd01167f

0 commit comments

Comments
 (0)