test: nothing runs the ignored simlin-engine tests #915

Activity

1. bpowers

   added a commit that references this issue on Aug 2, 2026

   engine: make implicit-helper identity a name, not a hash-ordered posi…

   ...

   4fdb940
2. bpowers commented on Aug 10, 2026

   

   bpowers

   on Aug 10, 2026

   OwnerAuthor

   More actions

   Re-measured on perf/tests (off the 2026-08 engine perf branch). The failure list in the description is now partly stale: there are 4 failures in the simulate.rs ignored tier today, not 7, and two of the three named non-DELAY-FIXED tests no longer exist. Posting the corrected state so nobody starts from the old list.

   What the ignored simulate:: tier looks like now

   --list --ignored on the engine integration binary reports 7 simulate:: entries:

   simulate::clearn_ltm_partials_all_parse
   simulate::compiles_and_runs_clearn_structural
   simulate::simulates_clearn_wasm
   simulate::simulates_delayfixed_xmile
   simulate::simulates_delayfixed_mdl
   simulate::simulates_delayfixed2_xmile
   simulate::simulates_delayfixed2_mdl
   

   Running them (--ignored simulate::, debug build):

   test result: FAILED. 3 passed; 4 failed
   failures:
       simulate::simulates_delayfixed_xmile
       simulate::simulates_delayfixed_mdl
       simulate::simulates_delayfixed2_xmile
       simulate::simulates_delayfixed2_mdl
   

   Reconciliation with the description's list of 7

   description's entry	status today
   simulates_delayfixed_xmile / _mdl / delayfixed2_xmile / _mdl	still #[ignore]d, still failing — #346 (DELAY FIXED ring-buffer semantics)
   simulates_except_xmile	no longer exists as a test. It is now a comment at src/simlin-engine/tests/integration/simulate.rs:2024-2028 recording that xmutil drops EXCEPT semantics and subscript mappings in XMILE conversion, so only the MDL path is covered (simulates_except / simulates_except2, both running).
   simulates_getdata_xmile / _mdl	no longer exist as tests. Also a comment now, at simulate.rs:2240-2244, giving the same reason the issue does (xmutil strips GET DATA BETWEEN TIMES; #347).

   So the "seventh failure with no live tracking issue" that this issue was largely written around — simulates_except_xmile — has since been converted from a permanently-failing #[ignore]d test into a documented comment. That is arguably the right resolution for an xmutil-side gap, and it removes the specific instance, but it does not address what this issue is actually about: nothing runs the tier, so nobody would have learned about that conversion (or a regression) either way. The issue should stay open on its own terms.

   Runtime of the whole ignored tier, for the lane design

   The other half of the problem is that a naive lane would need a budget. Timed at 8 threads on a 16c/32t machine (debug build), the full -p simlin-engine --test integration -- --ignored set:

   test	s
   simulate_ltm_wasm::discovery_clearn_matches_vm_wasm	329.8
   simulate_ltm::clearn_with_ltm_simulates_model_vars_identically	51.8
   simulate_ltm_pinned::clearn_pinned_climate_loop_is_scored	49.3
   simulate::simulates_clearn_wasm	47.1
   wrld3_ltm_panic::wrld3_element_level_enumeration_is_uncapped	30.5
   ltm_discovery_large_models::clearn_ltm_discovery_compiles	22.4
   simulate::clearn_ltm_partials_all_parse	18.6
   simulate_ltm_wasm::discovery_world3_matches_vm_wasm	16.3
   simulate::compiles_and_runs_clearn_structural	8.2
   mdl_roundtrip::write_mdl_for_vensim_validation	0.005

   One test is 330 s on its own, so this is a #657-shaped release lane, not something that fits the default debug run — which also answers the "are these still ignored for time?" question: yes, and by a wide margin. (metasd_expansion_tier_heavy was also on this list; it is gone as of perf/tests, where the metasd expansion tier became one test per model and its five models now run by default.)

   Suggested edits to this issue: correct the failure list to the four DELAY FIXED tests, drop the simulates_except_xmile "untracked seventh" framing (or reframe it as "the conversion to a comment happened unobserved, which is the same failure mode"), and note that any lane needs an expected-fail set covering #346 plus a budget that tolerates a 330 s test.
3. bpowers

   added a commit that references this issue on Aug 11, 2026

   engine: performance pass on compile, simulation, and the LTM path (#1021

   ...

   a17be13
4. bpowers

   changed the title [-]test: nothing runs the simlin-engine `#[ignore]`d tier; 7 tests fail there today (one untracked)[/-] [+]test: nothing runs the ignored simlin-engine tests[/+] on Oct 12, 2026
5. bpowers

   added

   ciCI, build pipeline, test hygieneCI, build pipeline, test hygiene

   hygieneToil, but its useful to get get too behind on itToil, but its useful to get get too behind on it

   on Oct 12, 2026
6. bpowers commented on Oct 12, 2026

   

   bpowers

   on Oct 12, 2026

   OwnerAuthor

   More actions

   Closing: already fixed on main.

   scripts/gates.sh runs every ignored simlin-engine test (--lib and --test integration, --ignored) under the optimized gates profile, and the gates job in .github/workflows/ci.yaml:161-192 runs it on every push and PR to main (PR #1080). Every remaining engine #[ignore] is a runtime-only gate; the DELAY FIXED models now run by default as refuses_delayfixed_* (tests/integration/simulate.rs:3152-3178) and DELAY FIXED / GET DATA are known-mismatch rows in mdl_vensim_truth.rs:257-334.
7. bpowers

   closed this as completedon Oct 12, 2026