Repository navigation
Commit edb76be
Round 10 eval (cost and defects by model tier) + FOREIGN_FAILURE (R-f-11) (#27)
* eval: vanilla arm (plain TypeScript and DOM, no framework)
src/main.ts default-exports mount(root); checker is strict tsc; scored like the other arms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: fuzz fills time out after 2 s like clicks
An app that disables an input while a request is out made fill wait 30 s per move, so whole sessions hit the 120 s
hang limit and were scored as page hangs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* latest of a store copies it by reading through the store (S-store-8)
S-store-7 snapshotted with Solid's deep(), which serves an older frame than reads through the store: after one of two
guesses on a row was declined, the snapshot still showed it. Found by desk9-rx-7's overlap tests (32 -> 28 of 38 after
4898ee3). Effect store inputs use the same copy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: PREREG-desk-10 draft (cost per correct app by model tier, four arms); rx-inline prompt fits a new app
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: PREREG-desk-10 counts legal holds out of defects
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: fuzz 'dropped guess' honours newer accepted writes and A1
A guess superseded by a newer write to the same field that was already accepted (so no longer in flight) was counted
as dropped, and so was a declined newer write still showing while the older one is out, which A1 (H-11) allows.
Found classifying the round 9 .rx desk holes: 9 of 32 were these. React and plain desk holes are unchanged by it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: ./test runs a dev build and shows the page's console warnings and errors
Agents never saw Solid's dev diagnostics (SILENT_HOLD and others) or React's warnings because ./test dropped console
output. Now the agent's test build is a development build and the warnings and errors print after the results, each
once with the tests that logged it. Grading stays a production build. Visible tests pass unchanged on round 9 React,
plain and .rx desks.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: freeze PREREG-desk-10 (Solid next 81803325, rxc from this commit)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 10 results (defects and cost by tier, four arms)
H10a and H10b (vs Foldkit) supported in the small tier; H10b vs plain and H10c not.
No app is defect-free in any arm, so cost per correct app is undefined; cost per app reported.
Solid issues filed from the run: #4627, #4629, #4631. cost.mjs accepts the vanilla arm;
replay.mjs replays the focus move.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* check: FOREIGN_FAILURE (R-f-11), a failure shown by another section's boundary
A question's failure that reaches an `errored` region guarding an unrelated
question, while it also surfaces in its own non-nested region, is refused:
the other section would show it as its own and retry the wrong question.
Found by round 10 (desk rx-2: the list passed to the detail as `rows`).
Fires on that program only across the repository's and eval runs' .rx files.
Same pass in compiler/src/foreign.rs and ref/src/foreign.ts, over the
resolved model after every module is checked. Spec 04 R-f-11 (taken by
default), diagnostics table, scenario S-fail-4, guide idiom.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
---------
Co-authored-by: Claude <noreply@anthropic.com>1 parent 0adcdad commit edb76be
213 files changed
Lines changed: 7079 additions & 24 deletions
File tree
- compiler/src
- eval
- harness
- results/desk-2026-11-02
- full
- graded
- legal
- guide
- ref
- compile
- oracle
- src
- scenarios
- spec
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
557 | 557 | | |
558 | 558 | | |
559 | 559 | | |
| 560 | + | |
| 561 | + | |
| 562 | + | |
| 563 | + | |
| 564 | + | |
| 565 | + | |
| 566 | + | |
| 567 | + | |
| 568 | + | |
| 569 | + | |
| 570 | + | |
| 571 | + | |
| 572 | + | |
| 573 | + | |
560 | 574 | | |
561 | 575 | | |
562 | 576 | | |
| |||
850 | 864 | | |
851 | 865 | | |
852 | 866 | | |
| 867 | + | |
| 868 | + | |
853 | 869 | | |
854 | 870 | | |
855 | 871 | | |
856 | 872 | | |
857 | 873 | | |
858 | 874 | | |
| 875 | + | |
| 876 | + | |
| 877 | + | |
| 878 | + | |
859 | 879 | | |
860 | 880 | | |
861 | 881 | | |
| |||
864 | 884 | | |
865 | 885 | | |
866 | 886 | | |
867 | | - | |
| 887 | + | |
868 | 888 | | |
869 | 889 | | |
870 | 890 | | |
| |||
877 | 897 | | |
878 | 898 | | |
879 | 899 | | |
| 900 | + | |
| 901 | + | |
880 | 902 | | |
881 | 903 | | |
882 | 904 | | |
883 | 905 | | |
884 | 906 | | |
| 907 | + | |
| 908 | + | |
885 | 909 | | |
| 910 | + | |
886 | 911 | | |
887 | 912 | | |
888 | 913 | | |
| |||
925 | 950 | | |
926 | 951 | | |
927 | 952 | | |
| 953 | + | |
| 954 | + | |
928 | 955 | | |
929 | 956 | | |
930 | 957 | | |
| |||
1637 | 1664 | | |
1638 | 1665 | | |
1639 | 1666 | | |
| 1667 | + | |
| 1668 | + | |
| 1669 | + | |
| 1670 | + | |
| 1671 | + | |
| 1672 | + | |
1640 | 1673 | | |
1641 | 1674 | | |
1642 | 1675 | | |
| |||
1655 | 1688 | | |
1656 | 1689 | | |
1657 | 1690 | | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
| 1696 | + | |
1658 | 1697 | | |
1659 | 1698 | | |
1660 | 1699 | | |
| |||
4110 | 4149 | | |
4111 | 4150 | | |
4112 | 4151 | | |
| 4152 | + | |
| 4153 | + | |
| 4154 | + | |
| 4155 | + | |
| 4156 | + | |
| 4157 | + | |
| 4158 | + | |
| 4159 | + | |
| 4160 | + | |
| 4161 | + | |
| 4162 | + | |
| 4163 | + | |
| 4164 | + | |
| 4165 | + | |
| 4166 | + | |
| 4167 | + | |
| 4168 | + | |
| 4169 | + | |
| 4170 | + | |
| 4171 | + | |
| 4172 | + | |
| 4173 | + | |
| 4174 | + | |
| 4175 | + | |
| 4176 | + | |
| 4177 | + | |
| 4178 | + | |
| 4179 | + | |
| 4180 | + | |
| 4181 | + | |
| 4182 | + | |
| 4183 | + | |
| 4184 | + | |
| 4185 | + | |
| 4186 | + | |
| 4187 | + | |
| 4188 | + | |
| 4189 | + | |
| 4190 | + | |
| 4191 | + | |
4113 | 4192 | | |
4114 | 4193 | | |
4115 | 4194 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
| 11 | + | |
11 | 12 | | |
12 | 13 | | |
13 | 14 | | |
| |||
0 commit comments