Skip to content

Commit 607b51c

Browse files
ryansolidclaude
andauthored
GUESS_WHILE_LOADING (AC-3a), with round 12's results and the XState arm (#53)
* Round 11 prereg draft; brief: the detail's retry is always present PREREG-desk-11 (DRAFT): round 10's design again on Solid next 2789ca9c (every eval bug fixed, #4689 included) with the current guide. H11a replicates H10a; H11c compares frontier .rx with frontier React. The brief now says the detail's retry button is always present while an issue is selected, which round 10 left unclear (25 of 36 desks failed the rename-reload overlap test over it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Lower a whole optimistic guess to the store setter's return A whole guess on an optimistic store (optimistic rows := rs => rs.map(...)) was written into the draft index by index, which replaces each matched row object. It is now the setter's return, which Solid diffs shallowly against the view as guesses. Clears round 10 rx-8 fuzz seed 3 on next 2789ca9c; the other 11 desks grade unchanged. Gate result recorded in PREREG-desk-11. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * PREREG-desk-11: state the gate result precisely Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * PREREG-desk-11: repin Solid to next 6751472f, record the gate result Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Count only defects: the assign-two overlap test accepts the H-11 hold; prereg says so The overlap test 'two issues assigned at once' required the declined row to revert while the other assignment was still out. A list held whole until both settle is a legal hold, not a defect, so the test now checks that the decline leaves the other row alone and that both rows are right once everything has answered. PREREG-desk-11 states that only defects are counted and that round 10's numbers are regraded the same way (without the brief-caused retry test). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Freeze PREREG-desk-11 (Ryan, 2026-11-10) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: round 11 results (defects, cost, round 10 regraded) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: round 11, rx-4 seed 15 traced Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: xstate arm (React 19 + XState v5, Stately's own docs) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: XState side check results; cost.mjs knows the xstate arm Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: round 12 pre-registration (draft) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * guide: guessing a prop goes through the child's own equation; prereg 12 notes it Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Round 12 prereg: pin Solid next bcef1041, record the gate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Round 12 prereg: frozen Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: round 12 results (graded, stats, costs, first traces) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * eval: round 12 traces (solidjs/solid#4786, #4787), sensitivity Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Guide: guess only on a value that is shown; name the three ways to keep a write from going out early The rule was listed under 'things that look like bugs and aren't' with an optional fix, so agents treated a void guess (D-34) as acceptable. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Guide: prefer keeping a loading value uneditable over a separate guessed layer Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Revert the guide's preference for an uneditable loading value 58 of 60 round 12 apps (every arm) keep the rename editable while the detail loads, so that is fair UX; the guide lists the three fixes without ranking them. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Guide: a write before its value is shown needs a place to show (not 'don't send it') Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Guide: state the shown-value and refresh rules without the eval task's nouns Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * Prototype GUESS_BEFORE_SHOWN (D-34) behind RX_PROTO_GBS; round 12 browser probe and plain-Solid repro The check fires when a click starts a guess on a value that may have no answer yet, whose question reads what the click is about, and the control is neither inside that value's loading region nor disabled on it. Over round 12's .rx apps it flags 8 sites, all of which lose a change in the browser probe. Off unless RX_PROTO_GBS is set. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX * GUESS_WHILE_LOADING (AC-3a): a guess a click can make while its value loads a new question Follows Solid's ruling on solidjs/solid#4795 (OPTIMISTIC_WRITE_WHILE_LOADING, at compile time): while x loads a new question a guess composes on the previous answer and the new answer replaces it (D-34). A handler that guesses x, where x's question reads what the click reads, must be covered as in R-8: inside x's loading region, or disabled on x (for a form's onSubmit, its submit buttons). Round 12: flags 8 handlers in 5 .rx apps; the browser probe loses the change in every one and in no unflagged app. It also fired on the eval's own references and on examples/ai/feed.rx, which now disable the control while it loads; every reference still passes its visible, hidden and overlap tests. Spec 06 AC-3a, both diagnostics tables, a diagnostics.md section, a unit test; the guide line follows the ruling. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0117BoWKEv4MghW8Ns9LeUUX --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 08255de commit 607b51c

108 files changed

Lines changed: 2896 additions & 28 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎compiler/src/check.rs‎

Lines changed: 194 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -338,6 +338,25 @@ fn snapshot_reads<'a>(n: Node<'a>, caught: bool, locals: &mut HashSet<String>, o
338338
snapshot_reads(c, caught, locals, out);
339339
}
340340
}
341+
/// AC-3a: the submit buttons of a form's markup (a `<button>` without `type`, or `type="submit"`), not through components.
342+
fn submit_buttons<'a>(ms: &'a [Markup], out: &mut Vec<&'a [Attr]>) {
343+
for m in ms {
344+
match &m.kind {
345+
MarkupKind::El { tag, attrs, children } => {
346+
if tag == "button" && !attrs.iter().any(|a| a.name == "type" && !matches!(&a.e.k, ExprKind::Lit(Lit::Str(v)) if v == "submit")) {
347+
out.push(attrs);
348+
}
349+
submit_buttons(children, out);
350+
}
351+
MarkupKind::Mif { t, f, .. } => {
352+
submit_buttons(t, out);
353+
submit_buttons(f, out);
354+
}
355+
MarkupKind::Mfor { body, .. } | MarkupKind::Loading { body, .. } | MarkupKind::Errored { body, .. } | MarkupKind::Portal { body, .. } => submit_buttons(body, out),
356+
_ => {}
357+
}
358+
}
359+
}
341360
/// A node the reference walks generically (`Object.values`): an expression, or a statement inside one.
342361
#[derive(Clone, Copy)]
343362
pub(crate) enum Node<'a> {
@@ -770,6 +789,8 @@ struct ReaderRec<'a> {
770789
/// R-8: an event handler on an element, with the element's attributes (its `disabled` covers the handler).
771790
struct HandlerRec<'a> {
772791
attrs: &'a [Attr],
792+
/// the element's children: a `<form>`'s submit buttons cover its `onSubmit` (AC-3a)
793+
kids: &'a [Markup],
773794
h: &'a Expr,
774795
env: EId,
775796
region: RId,
@@ -1028,6 +1049,7 @@ impl<'a> Checker<'a> {
10281049
self.resolve_tree();
10291050
self.loading_on_outside();
10301051
self.snapshot_guards();
1052+
self.guess_while_loading();
10311053
}
10321054
fn run_later(&mut self, job: Later<'a>) {
10331055
match job {
@@ -2765,7 +2787,7 @@ impl<'a> Checker<'a> {
27652787
if !h {
27662788
self.readers.push(ReaderRec { region, e: &a.e, env, line: self.line });
27672789
} else {
2768-
self.handlers.push(HandlerRec { attrs, h: &a.e, env, region, line: self.line, what: format!("<{}> {}", tag, a.name) });
2790+
self.handlers.push(HandlerRec { attrs, kids: children, h: &a.e, env, region, line: self.line, what: format!("<{}> {}", tag, a.name) });
27692791
}
27702792
}
27712793
self.markup(children, env, false);
@@ -3068,6 +3090,132 @@ impl<'a> Checker<'a> {
30683090
}
30693091
}
30703092
}
3093+
3094+
// ---- AC-3a: a guess a click can make while its target loads a new question ----
3095+
/// AC-3a (D-34, Solid's `OPTIMISTIC_WRITE_WHILE_LOADING`): a handler starts a guess (in its own body, an inferred
3096+
/// or inline action, or a named action it calls) on an equation or source that awaits, whose question reads what
3097+
/// the click reads (a selection, a row), from an element that is not covered (R-8: inside a `loading` region whose
3098+
/// content reads the target, or `disabled` reading it). While that question is out the guess composes on the
3099+
/// previous answer, and the new answer replaces it: the user's change disappears while its action is out.
3100+
fn guess_while_loading(&mut self) {
3101+
let hs: Vec<(&'a [Attr], &'a [Markup], &'a Expr, EId, RId, i64, String)> = self.handlers.iter().map(|h| (h.attrs, h.kids, h.h, h.env, h.region, h.line, h.what.clone())).collect();
3102+
let acts: Vec<(String, i64, &'a [Stmt], EId)> = self.actions.iter().map(|a| (a.name.clone(), a.line, a.body, a.env)).collect();
3103+
let mut done: HashSet<*const Expr> = HashSet::new();
3104+
for (attrs, kids, h, env, region, line, what) in hs {
3105+
if !done.insert(h as *const Expr) {
3106+
continue;
3107+
}
3108+
// a form submits only through its buttons: disabled submit buttons block it (HTML implicit submission)
3109+
let mut submits: Vec<&'a [Attr]> = vec![];
3110+
if what == "<form> onSubmit" {
3111+
submit_buttons(kids, &mut submits);
3112+
}
3113+
// the bodies the click runs: the handler, and the actions it calls by name
3114+
let mut bodies: Vec<(Node<'a>, EId, Option<String>)> = vec![(Node::E(h), env, None)];
3115+
let mut stack = vec![Node::E(h)];
3116+
while let Some(n) = stack.pop() {
3117+
if let Node::E(Expr { k: ExprKind::Call { f, .. }, .. }) = n {
3118+
if let ExprKind::Id(name) = &f.k {
3119+
if let Some(a) = self.lookup(env, name) {
3120+
if self.syms[a].kind == SymKind::Action {
3121+
let l = self.syms[a].line;
3122+
for (an, al, body, aenv) in &acts {
3123+
if an == name && *al == l {
3124+
bodies.extend(body.iter().map(|st| (Node::S(st), *aenv, Some(name.clone()))));
3125+
}
3126+
}
3127+
}
3128+
}
3129+
}
3130+
}
3131+
stack.extend(children(n, false));
3132+
}
3133+
// what the click is about: the names those bodies read, outside their own writes and guesses
3134+
let mut inputs: Vec<SId> = vec![];
3135+
for (root, benv, _) in &bodies {
3136+
let mut st = vec![*root];
3137+
while let Some(n) = st.pop() {
3138+
match n {
3139+
Node::S(x) if matches!(x.k, StmtKind::Optimistic { .. } | StmtKind::Write { .. }) => continue,
3140+
Node::E(x) => {
3141+
if let ExprKind::Id(name) = &x.k {
3142+
if let Some(v) = self.lookup(*benv, name) {
3143+
if matches!(self.syms[v].kind, SymKind::Source | SymKind::Eq | SymKind::Value | SymKind::Props) && !inputs.contains(&v) {
3144+
inputs.push(v);
3145+
}
3146+
}
3147+
}
3148+
}
3149+
_ => {}
3150+
}
3151+
st.extend(children(n, false));
3152+
}
3153+
}
3154+
let mut seen: HashSet<SId> = HashSet::new();
3155+
for (root, benv, via) in bodies {
3156+
let mut st = vec![root];
3157+
while let Some(n) = st.pop() {
3158+
if let Node::S(x) = n {
3159+
if let StmtKind::Optimistic { target, .. } = &x.k {
3160+
if let Some(t) = self.lookup(benv, &target.name) {
3161+
if seen.insert(t)
3162+
&& self.awaits(t, &mut HashSet::new())
3163+
&& self.asks_about(t, &inputs)
3164+
&& !attrs.iter().any(|a| a.name == "disabled" && self.depends(&a.e, env, t, &mut HashSet::new()))
3165+
&& !(!submits.is_empty() && submits.iter().all(|b| b.iter().any(|a| a.name == "disabled" && self.depends(&a.e, env, t, &mut HashSet::new()))))
3166+
&& !self.region_reads(region, t, |r| r.provides.contains(HOLD))
3167+
{
3168+
let n = &target.name;
3169+
self.line = line;
3170+
self.err(
3171+
"GUESS_WHILE_LOADING",
3172+
"AC-3a",
3173+
format!(
3174+
"{what} {} `guess {n}` while {n} is loading a new question (its question reads what the click reads): the guess composes on the previous answer, and the new answer replaces it, so the change disappears while it is out (D-34) — put the element inside the `loading` region whose content reads {n}, or give it `disabled={{pending? {n}}}`, or keep the pending change in a source of its own until it lands",
3175+
match &via { Some(a) => format!("starts `{a}`, which can run"), None => "can run".into() }
3176+
),
3177+
);
3178+
}
3179+
}
3180+
}
3181+
}
3182+
st.extend(children(n, false));
3183+
}
3184+
}
3185+
}
3186+
}
3187+
/// AC-3a: `t`'s value comes from an `await` (with or without a first paint), here or in what it reads.
3188+
fn awaits(&self, t: SId, busy: &mut HashSet<SId>) -> bool {
3189+
let Some((e, env)) = self.syms[t].expr else { return false };
3190+
if !busy.insert(t) {
3191+
return false;
3192+
}
3193+
let mut stack = vec![Node::E(e)];
3194+
while let Some(n) = stack.pop() {
3195+
if let Node::E(x) = n {
3196+
match &x.k {
3197+
ExprKind::Fn { .. } | ExprKind::ServerFn { .. } | ExprKind::Peek(_) => continue,
3198+
ExprKind::Await { .. } => return true,
3199+
ExprKind::Id(name) => {
3200+
if let Some(u) = self.lookup(env, name) {
3201+
if u != t && matches!(self.syms[u].kind, SymKind::Eq | SymKind::Source) && self.awaits(u, busy) {
3202+
return true;
3203+
}
3204+
}
3205+
continue;
3206+
}
3207+
_ => {}
3208+
}
3209+
}
3210+
stack.extend(children(n, false));
3211+
}
3212+
false
3213+
}
3214+
/// AC-3a: `t`'s question reads one of `inputs` (what a click reads) that does not itself follow `t`.
3215+
fn asks_about(&self, t: SId, inputs: &[SId]) -> bool {
3216+
let Some((e, env)) = self.syms[t].expr else { return false };
3217+
inputs.iter().any(|&i| i != t && !self.syms[i].expr.is_some_and(|(ie, ienv)| self.depends(ie, ienv, t, &mut HashSet::new())) && self.depends(e, env, i, &mut HashSet::new()))
3218+
}
30713219
/// R-8: `s` can be read before its first answer: an `await` with no first paint (`-> v`, R-7) in it or what it reads.
30723220
fn absent(&self, s: SId, memo: &mut HashMap<SId, bool>, busy: &mut HashSet<SId>) -> bool {
30733221
self.sym_absent(s, false, memo, busy)
@@ -4123,6 +4271,51 @@ mod tests {
41234271
check_modules(&set, spa).diags.iter().map(|d| (d.line, d.code.clone())).collect()
41244272
}
41254273

4274+
/// 06 AC-3a: a guess a click can make while its value loads a new question, unless the element is covered.
4275+
#[test]
4276+
fn guess_while_loading() {
4277+
let src = r#"export default component App() {
4278+
source sel := 1
4279+
source filter := "open"
4280+
source detail = await fetch("issue:" + sel)
4281+
source rows = await fetch("rows:" + filter)
4282+
source note := ""
4283+
action rename(t: string) {
4284+
try {
4285+
guess detail := d => d == null ? d : { ...d, title: t }
4286+
guess rows := rs => rs
4287+
await fetch("rename:" + sel + "," + t)
4288+
refresh detail
4289+
} catch (e) {
4290+
note := e.message
4291+
}
4292+
}
4293+
errored fallback(e) { <p>failed</p> } {
4294+
loading on(sel) fallback { <p>loading</p> } {
4295+
<h2>{detail.title}</h2>
4296+
<button onClick={() => { rename("in") }}>in</button>
4297+
}
4298+
loading on(filter) fallback { <p>loading</p> } {
4299+
<p>{rows.length}</p>
4300+
}
4301+
}
4302+
<button onClick={() => { sel := sel + 1 }}>next</button>
4303+
<button onClick={() => { rename("out") }}>out</button>
4304+
<button disabled={pending? detail} onClick={() => { rename("off") }}>off</button>
4305+
<form onSubmit={() => { rename("form") }}>
4306+
<button disabled={pending? detail}>go</button>
4307+
</form>
4308+
<form onSubmit={() => { rename("form2") }}>
4309+
<button>go</button>
4310+
</form>
4311+
}
4312+
"#;
4313+
let got: Vec<i64> = codes(src, false).iter().filter(|(_, c)| c == "GUESS_WHILE_LOADING").map(|(l, _)| *l).collect();
4314+
// `out` and the second form: `detail` asks about `sel`, which `rename` reads; `rows` asks about `filter`,
4315+
// which the click never reads; `in` sits in detail's region, `off` and the first form are disabled on it
4316+
assert_eq!(got, vec![27, 32], "{:?}", codes(src, false));
4317+
}
4318+
41264319
/// 08 CN-3d: in a client-only build, a reader the first load surely waits on needs a `loading` above it.
41274320
#[test]
41284321
fn first_load_unheld_in_client_only_builds() {

‎eval/PREREG-desk-12.md‎

Lines changed: 75 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,75 @@
1+
# Round 12: round 11 again with five arms and twice the frontier runs
2+
3+
Pre-registration, committed before any round 12 agent runs. Status: FROZEN (Ryan, 2026-11-11 "Go").
4+
5+
## Why
6+
7+
Round 11 (`results/desk-2026-11-11`) replicated round 10's small-tier result.
8+
9+
- **Small tier:** .rx had 3.8 defects per app and React 9.7. H11a holds, at p = 0.047.
10+
- **Frontier tier:** every arm was between 2 and 3. With three runs per arm, H11c ("no worse than React") could not be shown: the interval was [−2.3, 2.3].
11+
- **XState:** a side check the same day (`results/xstate-2026-11-11`, not pre-registered) put small XState at 7.0, between .rx and React and below Foldkit's 13.2.
12+
13+
Round 12 asks whether these hold with fresh agents. It adds XState as a fifth arm and doubles the frontier runs, so the frontier comparison has a chance to resolve.
14+
15+
Ryan, 2026-11-11: replication now, maintenance next. Maintenance moves to round 13 and runs on round 12's apps.
16+
17+
## Task and arms (as round 11 unless stated)
18+
19+
**Task:** `tasks/desk9` with `brief-plain.md`, unchanged. Graded by `hidden.mjs` (77) and `hidden-overlap.mjs` (38), unchanged.
20+
21+
**Arms:**
22+
- **`rx`** with `--inline-docs` (`language.md`, `async.md`, `reference.md`).
23+
- **`foldkit`** with `--inline-docs`.
24+
- **`react`**.
25+
- **`vanilla`**.
26+
- **`xstate`**, new: React 19 with xstate 5.32.6 and @xstate/react 6.1.0, with no TanStack Query. It uses `--inline-docs`, which here means Stately's own v5 cheatsheet and @xstate/react's README in the prompt. Every Stately v5 docs page is readable in `docs/xstate`, from statelyai/docs at 7ee0c60. The checker is the same as React's.
27+
28+
**Language and guide changes since round 11's freeze:**
29+
- `guess` replaces `optimistic` (#47).
30+
- Errors are cleared with `onFlush` (#47).
31+
- A guess over an equation is allowed (#48).
32+
- Destructuring works (#49).
33+
34+
The guide follows all four. `rxc` is built from the commit that holds this file once it is frozen. Round 11's .rx desks use `optimistic`, so the gate below grades them with round 11's `rxc`, which isolates the Solid change.
35+
36+
**Guide addition** (principle only, no task code): `async.md` says how to guess a prop. You guess over the child's own equation of it, never `props.x`.
37+
38+
**Solid:** `solid-next` at `next` bcef1041, built from source. It holds the fixes for both cases of solidjs/solid#4762: cf32a50b (a keyed projection of a projection losing a row on a switch) and #4769 (two guesses, one landing after the switch, now hold together as D-33 says).
39+
40+
**Harness:** subagents, as in round 11.
41+
42+
## Runs, grading
43+
44+
**Runs:** 6 frontier and 6 small per arm, 60 in all, in one interleaved batch. Each run gets at most 15 `./check`/`./test` calls.
45+
46+
**Grading:** `harness/grade.sh`, with round 11's definitions of correct and cost. Only defects are counted. A fuzz hole counts unless it is traced to a legal hold, and each exclusion is listed with its trace.
47+
48+
## Hypotheses and tests
49+
50+
All tests use seed 12.
51+
52+
- **H12a** (primary, replication of H11a): small .rx runs have fewer defects than small React runs. One-sided permutation test on mean defects, 100,000 relabellings.
53+
- **H12b**: three comparisons, Holm-corrected together with H12a:
54+
- small .rx has fewer defects than small XState;
55+
- small .rx has fewer defects than small plain;
56+
- small .rx has fewer defects than small Foldkit.
57+
- **H12c**: frontier .rx runs have no more defects than frontier React runs. Difference in means with a 90% bootstrap interval, 10,000 resamples. The claim holds if the upper end is under 1 defect.
58+
- **H12d**: small XState runs have fewer defects than small Foldkit runs. Same test as H12a, reported apart from the .rx family. This is the PROCESS question: state machines against Effect/TEA.
59+
- **Descriptive:** cost per correct app, by arm and tier.
60+
- **Reported alongside:**
61+
- the change from round 11 for each arm and tier;
62+
- defects by kind;
63+
- turns, cost per run and lines.
64+
65+
The direction is the claim being tested. A null or reversed result is reported as plainly.
66+
67+
## Gate before freezing
68+
69+
Round 11's nine .rx desks are regraded on the pinned Solid, with round 11's `rxc`. Any desk that gains a hole is traced. A Solid regression is filed upstream, then fixed or recorded, before the round starts.
70+
71+
**Gate result** (2026-11-11, on `next` bcef1041's source, round 11's `rxc`): 26 defects across the nine desks, against 29 in round 11. Only rx-2 changed, from 5 to 2: its three fuzz holes (seeds 1, 3, 18, solidjs/solid#4762) are gone, and its two overlap failures, which are its own (a pending source in a component that remounts), remain. The other eight desks grade exactly as in round 11. No desk gained a hole, so nothing needed tracing.
72+
73+
## Cost
74+
75+
About $67 at round 11's prices: 6 × ($2.05 + $1.38 + $1.27 + $3.63 + $2.30) for the frontier runs, plus about $3.30 for the small ones.

‎eval/harness/build.mjs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// Builds one app (a run directory or a reference) into `<dir>/.build/dist` with Vite, for the tests to drive.
22
// rx arm: `rxc compile src/main.rx` → JavaScript over Solid rc.19, mounted with the .rx runtime (ref/compile/rt-web.ts)
3-
// react arm: `src/App.tsx` (default export) mounted with createRoot under StrictMode, as Vite's React template does
3+
// react and xstate arms: `src/App.tsx` (default export) mounted with createRoot under StrictMode, as Vite's React template does
44
// solid arm: `src/App.tsx` (default export) compiled by @solidjs/compiler (Solid 2's JSX compiler) and mounted with
55
// @solidjs/web's `render`
66
// vanilla arm: `src/main.ts` default-exports `mount(root: HTMLElement)`, called once with the `#app` container
@@ -57,7 +57,7 @@ export async function build(dir, arm = armOf(dir)) {
5757
writeFileSync(path.join(work, "entry.js"), entry);
5858
writeFileSync(path.join(work, "index.html"), `<!doctype html><html><head><meta charset="utf-8"><title>app</title></head><body><div id="app"></div><script type="module" src="./entry.js"></script></body></html>\n`);
5959
const vite = await import(pathToFileURL(path.join(evalDir, "node_modules/vite/dist/node/index.js")).href);
60-
const plugins = arm === "react" ? [(await import(pathToFileURL(path.join(evalDir, "node_modules/@vitejs/plugin-react/dist/index.js")).href)).default()]
60+
const plugins = arm === "react" || arm === "xstate" ? [(await import(pathToFileURL(path.join(evalDir, "node_modules/@vitejs/plugin-react/dist/index.js")).href)).default()]
6161
: arm === "solid" ? [await solidPlugin()] : [];
6262
try {
6363
await vite.build({

‎eval/harness/check.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
// The arm's compiler feedback, as the agent runs it.
22
// rx: rxc check src/main.rx
3-
// react: tsc --noEmit -p . and eslint (react-hooks rules)
3+
// react, xstate: tsc --noEmit -p . and eslint (react-hooks rules)
44
// solid: tsc --noEmit -p . and eslint (eslint-plugin-solid's v2 config)
55
// vanilla: tsc --noEmit -p . (strict)
66
// foldkit: tsc --noEmit -p . with Foldkit's template tsconfig (its lint, oxlint + effect-tsgo, is not on npm)

‎eval/harness/cost.mjs‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -45,7 +45,7 @@ export function permutation(a, b, n = 100000, seed = 5) {
4545
if (import.meta.url === pathToFileURL(process.argv[1]).href) {
4646
const [list, dir, out] = process.argv.slice(2);
4747
const runs = readFileSync(list, "utf8").trim().split("\n").map(l => l.trim().split(/\s+/)).map(([run, id]) => {
48-
const [, task, arm, n] = run.match(/^(\w+)-(rx|react|solid|foldkit|vanilla)-(\d+)$/);
48+
const [, task, arm, n] = run.match(/^(\w+)-(rx|react|xstate|solid|foldkit|vanilla)-(\d+)$/);
4949
return { run, task, arm, n: Number(n), ...usage(path.join(dir, `agent-${id}.jsonl`)) };
5050
});
5151
for (const r of runs) console.log(`${r.run.padEnd(16)} billed ${Math.round(r.billed).toString().padStart(7)} calls ${r.calls} out ${r.output} cacheRead ${r.cacheRead} cacheWrite ${r.cacheWrite}`);

0 commit comments

Comments
 (0)