You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 607b51c
Browse filesBrowse the repository at this point in the historyBrowse files
GUESS_WHILE_LOADING (AC-3a), with round 12's results and the XState arm (#53)
* Round 11 prereg draft; brief: the detail's retry is always present
PREREG-desk-11 (DRAFT): round 10's design again on Solid next 2789ca9c
(every eval bug fixed, #4689 included) with the current guide. H11a
replicates H10a; H11c compares frontier .rx with frontier React. The
brief now says the detail's retry button is always present while an
issue is selected, which round 10 left unclear (25 of 36 desks failed
the rename-reload overlap test over it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Lower a whole optimistic guess to the store setter's return
A whole guess on an optimistic store (optimistic rows := rs => rs.map(...)) was written into the draft index by index,
which replaces each matched row object. It is now the setter's return, which Solid diffs shallowly against the view as
guesses. Clears round 10 rx-8 fuzz seed 3 on next 2789ca9c; the other 11 desks grade unchanged. Gate result recorded
in PREREG-desk-11.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* PREREG-desk-11: state the gate result precisely
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* PREREG-desk-11: repin Solid to next 6751472f, record the gate result
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Count only defects: the assign-two overlap test accepts the H-11 hold; prereg says so
The overlap test 'two issues assigned at once' required the declined row to revert while the other assignment was
still out. A list held whole until both settle is a legal hold, not a defect, so the test now checks that the decline
leaves the other row alone and that both rows are right once everything has answered. PREREG-desk-11 states that only
defects are counted and that round 10's numbers are regraded the same way (without the brief-caused retry test).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Freeze PREREG-desk-11 (Ryan, 2026-11-10)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 11 results (defects, cost, round 10 regraded)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 11, rx-4 seed 15 traced
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: xstate arm (React 19 + XState v5, Stately's own docs)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: XState side check results; cost.mjs knows the xstate arm
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 12 pre-registration (draft)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* guide: guessing a prop goes through the child's own equation; prereg 12 notes it
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Round 12 prereg: pin Solid next bcef1041, record the gate
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Round 12 prereg: frozen
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 12 results (graded, stats, costs, first traces)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* eval: round 12 traces (solidjs/solid#4786, #4787), sensitivity
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Guide: guess only on a value that is shown; name the three ways to keep a write from going out early
The rule was listed under 'things that look like bugs and aren't' with an
optional fix, so agents treated a void guess (D-34) as acceptable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Guide: prefer keeping a loading value uneditable over a separate guessed layer
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Revert the guide's preference for an uneditable loading value
58 of 60 round 12 apps (every arm) keep the rename editable while the detail
loads, so that is fair UX; the guide lists the three fixes without ranking them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Guide: a write before its value is shown needs a place to show (not 'don't send it')
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Guide: state the shown-value and refresh rules without the eval task's nouns
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* Prototype GUESS_BEFORE_SHOWN (D-34) behind RX_PROTO_GBS; round 12 browser probe and plain-Solid repro
The check fires when a click starts a guess on a value that may have no answer yet,
whose question reads what the click is about, and the control is neither inside that
value's loading region nor disabled on it. Over round 12's .rx apps it flags 8 sites,
all of which lose a change in the browser probe. Off unless RX_PROTO_GBS is set.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
* GUESS_WHILE_LOADING (AC-3a): a guess a click can make while its value loads a new question
Follows Solid's ruling on solidjs/solid#4795 (OPTIMISTIC_WRITE_WHILE_LOADING, at
compile time): while x loads a new question a guess composes on the previous answer
and the new answer replaces it (D-34). A handler that guesses x, where x's question
reads what the click reads, must be covered as in R-8: inside x's loading region, or
disabled on x (for a form's onSubmit, its submit buttons).
Round 12: flags 8 handlers in 5 .rx apps; the browser probe loses the change in every
one and in no unflagged app. It also fired on the eval's own references and on
examples/ai/feed.rx, which now disable the control while it loads; every reference
still passes its visible, hidden and overlap tests. Spec 06 AC-3a, both diagnostics
tables, a diagnostics.md section, a unit test; the guide line follows the ruling.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX
---------
Co-authored-by: Claude <noreply@anthropic.com>
"{what} {} `guess {n}` while {n} is loading a new question (its question reads what the click reads): the guess composes on the previous answer, and the new answer replaces it, so the change disappears while it is out (D-34) — put the element inside the `loading` region whose content reads {n}, or give it `disabled={{pending? {n}}}`, or keep the pending change in a source of its own until it lands",
3175
+
match&via {Some(a) => format!("starts `{a}`, which can run"),None => "can run".into()}
3176
+
),
3177
+
);
3178
+
}
3179
+
}
3180
+
}
3181
+
}
3182
+
st.extend(children(n,false));
3183
+
}
3184
+
}
3185
+
}
3186
+
}
3187
+
/// AC-3a: `t`'s value comes from an `await` (with or without a first paint), here or in what it reads.
-**Small tier:** .rx had 3.8 defects per app and React 9.7. H11a holds, at p = 0.047.
10
+
-**Frontier tier:** every arm was between 2 and 3. With three runs per arm, H11c ("no worse than React") could not be shown: the interval was [−2.3, 2.3].
11
+
-**XState:** a side check the same day (`results/xstate-2026-11-11`, not pre-registered) put small XState at 7.0, between .rx and React and below Foldkit's 13.2.
12
+
13
+
Round 12 asks whether these hold with fresh agents. It adds XState as a fifth arm and doubles the frontier runs, so the frontier comparison has a chance to resolve.
14
+
15
+
Ryan, 2026-11-11: replication now, maintenance next. Maintenance moves to round 13 and runs on round 12's apps.
16
+
17
+
## Task and arms (as round 11 unless stated)
18
+
19
+
**Task:**`tasks/desk9` with `brief-plain.md`, unchanged. Graded by `hidden.mjs` (77) and `hidden-overlap.mjs` (38), unchanged.
20
+
21
+
**Arms:**
22
+
-**`rx`** with `--inline-docs` (`language.md`, `async.md`, `reference.md`).
23
+
-**`foldkit`** with `--inline-docs`.
24
+
-**`react`**.
25
+
-**`vanilla`**.
26
+
-**`xstate`**, new: React 19 with xstate 5.32.6 and @xstate/react 6.1.0, with no TanStack Query. It uses `--inline-docs`, which here means Stately's own v5 cheatsheet and @xstate/react's README in the prompt. Every Stately v5 docs page is readable in `docs/xstate`, from statelyai/docs at 7ee0c60. The checker is the same as React's.
27
+
28
+
**Language and guide changes since round 11's freeze:**
29
+
-`guess` replaces `optimistic` (#47).
30
+
- Errors are cleared with `onFlush` (#47).
31
+
- A guess over an equation is allowed (#48).
32
+
- Destructuring works (#49).
33
+
34
+
The guide follows all four. `rxc` is built from the commit that holds this file once it is frozen. Round 11's .rx desks use `optimistic`, so the gate below grades them with round 11's `rxc`, which isolates the Solid change.
35
+
36
+
**Guide addition** (principle only, no task code): `async.md` says how to guess a prop. You guess over the child's own equation of it, never `props.x`.
37
+
38
+
**Solid:**`solid-next` at `next` bcef1041, built from source. It holds the fixes for both cases of solidjs/solid#4762: cf32a50b (a keyed projection of a projection losing a row on a switch) and #4769 (two guesses, one landing after the switch, now hold together as D-33 says).
39
+
40
+
**Harness:** subagents, as in round 11.
41
+
42
+
## Runs, grading
43
+
44
+
**Runs:** 6 frontier and 6 small per arm, 60 in all, in one interleaved batch. Each run gets at most 15 `./check`/`./test` calls.
45
+
46
+
**Grading:**`harness/grade.sh`, with round 11's definitions of correct and cost. Only defects are counted. A fuzz hole counts unless it is traced to a legal hold, and each exclusion is listed with its trace.
47
+
48
+
## Hypotheses and tests
49
+
50
+
All tests use seed 12.
51
+
52
+
-**H12a** (primary, replication of H11a): small .rx runs have fewer defects than small React runs. One-sided permutation test on mean defects, 100,000 relabellings.
53
+
-**H12b**: three comparisons, Holm-corrected together with H12a:
54
+
- small .rx has fewer defects than small XState;
55
+
- small .rx has fewer defects than small plain;
56
+
- small .rx has fewer defects than small Foldkit.
57
+
-**H12c**: frontier .rx runs have no more defects than frontier React runs. Difference in means with a 90% bootstrap interval, 10,000 resamples. The claim holds if the upper end is under 1 defect.
58
+
-**H12d**: small XState runs have fewer defects than small Foldkit runs. Same test as H12a, reported apart from the .rx family. This is the PROCESS question: state machines against Effect/TEA.
59
+
-**Descriptive:** cost per correct app, by arm and tier.
60
+
-**Reported alongside:**
61
+
- the change from round 11 for each arm and tier;
62
+
- defects by kind;
63
+
- turns, cost per run and lines.
64
+
65
+
The direction is the claim being tested. A null or reversed result is reported as plainly.
66
+
67
+
## Gate before freezing
68
+
69
+
Round 11's nine .rx desks are regraded on the pinned Solid, with round 11's `rxc`. Any desk that gains a hole is traced. A Solid regression is filed upstream, then fixed or recorded, before the round starts.
70
+
71
+
**Gate result** (2026-11-11, on `next` bcef1041's source, round 11's `rxc`): 26 defects across the nine desks, against 29 in round 11. Only rx-2 changed, from 5 to 2: its three fuzz holes (seeds 1, 3, 18, solidjs/solid#4762) are gone, and its two overlap failures, which are its own (a pending source in a component that remounts), remain. The other eight desks grade exactly as in round 11. No desk gained a hole, so nothing needed tracing.
72
+
73
+
## Cost
74
+
75
+
About $67 at round 11's prices: 6 × ($2.05 + $1.38 + $1.27 + $3.63 + $2.30) for the frontier runs, plus about $3.30 for the small ones.
0 commit comments