ci: run the ignored C-LEARN-scale engine tests in an optimized build #657

Activity

1. bpowers

   added

   ciCI, build pipeline, test hygieneCI, build pipeline, test hygiene

   on Jun 8, 2026
2. bpowers commented on Jul 9, 2026

   

   bpowers

   on Jul 9, 2026

   OwnerAuthor

   More actions

   Confirmed rot: this is no longer hypothetical

   Discovered while working #912/#913 on branch conveyor-engine. This issue predicted that ignored C-LEARN-scale tests "only run when someone remembers"; a pin has now silently gone stale, which raises the severity from "no regression coverage" to "an active gate is broken and nobody noticed."

   clearn_ltm_var_count_guardrail fails on a clean checkout of conveyor-engine HEAD (c1cc86d5): it produces 6709 where the pin asserts 6713. Some earlier commit on the branch moved LTM emission by 4 variables. The test never executed, so nothing reported it.

   This test is not in the inventory above -- it postdates this issue. It lives at src/simlin-engine/tests/integration/simulate.rs:5730, and its own doc comment states the pin exists so that

   any change to LTM emission on a real large model is a conscious, reviewed flip of this number -- the #654 layout-slot ceiling (65,536 u16 slots) is the resource it protects.

   An exact-value pin guarding a resource ceiling, which is only ever run by hand, provides no protection. It rots into a landmine for the next person who runs --ignored and cannot tell an intended emission change from a regression. (The stale number itself is being flipped deliberately in the #912/#913 PR; the unmonitored gate is what this issue should fix.)

   The #[ignore] tier is entirely unmonitored today

   Neither scripts/pre-commit nor .github/workflows/ci.yaml runs the engine's --ignored tier. The only --ignored invocation in CI is scoped to a different crate:

   .github/workflows/ci.yaml:410
     cargo test -p simlin-serve --release --test integration -- --ignored --nocapture smoke::
   

   The only schedule: trigger in .github/workflows/ is memory.yml (unrelated). So there is no nightly lane to attach this to yet.

   Current inventory is also much larger than tech-debt.md item 13 records (23 tests / 5 files, last reviewed 2026-03-12). Actual count in src/simlin-engine/tests/integration/ today is 72 across 12 files:

   simulate.rs                    34    ltm_discovery_large_models.rs   9
   simulate_ltm_wasm.rs            7    metasd_macros.rs                6
   simulate_ltm.rs                 3    wrld3_ltm_panic.rs              3
   test_helpers.rs                 3    clearn_unit_errors.rs           2
   simulate_ltm_pinned.rs          2    mdl_equivalence.rs              1
   mdl_roundtrip.rs                1    wrld3_unit_errors.rs            1
   

   Item 13's Count field should be refreshed when this lands.

   Two distinct populations, which argues for the nightly framing

   The ignored tests are not homogeneous, and the proposed lane needs to handle both:

   1. Passing gates that must keep passing -- clearn_ltm_var_count_guardrail, simulates_clearn, simulates_clearn_wasm, compiles_and_runs_clearn_structural, simulates_wrld3_03_wasm. These are pins and correctness oracles. They are ignored purely for runtime (the guardrail alone is ~4 min on a 53k-line / 1.4 MB model). These are the ones that rot silently. A nightly --ignored sweep, or a sweep gated on changes to src/simlin-engine/, would make them gate something.
   2. Known-failing feature-gap tests -- delayfixed (engine: DELAY FIXED is refused instead of simulated #346), getdata (mdl: GET DATA BETWEEN TIMES and .dat data variables are unsupported #347), except_xmile. These are expected-red and must not fail the lane. They need #[ignore = "reason"] strings or a filter/feature so the sweep can exclude them, otherwise the job is red on day one and gets muted.

   Population 2 is already tracked by #346/#347 (and #345, closed); a sibling issue may be filed covering the currently-failing #[ignore]d feature-gap tests. This issue should stay scoped to the unmonitored gate, i.e. building the lane and deciding which tests it enforces.

   Suggested acceptance criteria

   • A scheduled (nightly or weekly) or engine-path-gated CI job runs the release-mode --ignored tier for simlin-engine.
   • The job's test selection explicitly excludes the known-failing feature-gap tests, by filter or by an #[ignore = "..."] convention, so a red run means a real regression.
   • A red run notifies someone (the failure mode being fixed here is "nobody noticed").
   • tech-debt.md item 13 Count / Last reviewed refreshed.
3. bpowers commented on Sep 10, 2026

   

   bpowers

   on Sep 10, 2026

   OwnerAuthor

   More actions

   New evidence from the LTM fidelity work of 2026-09-07..09 (branch ltm-fidelity) that this lane is needed, plus a concrete scope for it. Filed here rather than as a new issue because the root problem is this one (and #915's tier-wide half).

   What drifted

   • clearn_ltm_slot_maxima_digest (src/simlin-engine/tests/integration/simulate_ltm.rs) and the other C-LEARN #[ignore]d release-only gates (clearn_with_ltm_simulates_model_vars_identically in the same file, clearn_ltm_discovery_compiles in ltm_discovery_large_models.rs) are run by neither the pre-commit hook nor cargo test. .github/workflows/ci.yaml invokes --ignored in exactly one place (line 554) and it is scoped to the simlin-serve smoke tests.
   • Commit 05e4dae ("engine: score flow-to-stock links off a net-flow aux") moved the digest's pins -- slots 20,221 -> 20,337, nonzero slots 3,106 -> 3,198 -- and landed without re-pinning the gate. It stayed red for a day until f5e0325 ("engine: freeze the clock inside the ceteris-paribus partial") re-derived both steps (20,337 -> 20,338; 3,198 -> 2,876) and wrote the two-step derivation into the gate's rustdoc.
   • That digest is the reproducible, checked-in form of the GH ltm: link-score generation is quadratic in an arrayed target's size #977 value-neutrality differential over all C-LEARN LTM slots. A red digest nobody sees is exactly the "a change that zeroed 149 C-LEARN LTM slots passed every named C-LEARN gate" failure it was added to catch.

   Concrete shape

   A CI job (or a pre-push step) that builds release and runs

   cargo test -p simlin-engine --release --test integration -- --ignored clearn
   

   gated on test/xmutil_test_models/C-LEARN v77 for Vensim.mdl being present, with a documented time budget (the digest itself is ~3 s in release on top of the build; the other C-LEARN gates were measured at 6-15 s release above). The clearn filter keeps the lane green on day one by sidestepping #915's seven known-failing non-C-LEARN ignored tests; the tier-wide monitoring in #915 is the larger follow-on. Cost: small.
4. bpowers

   added a commit that references this issue on Sep 10, 2026

   engine: LTM fidelity fixes from the paper audit (#1062)

   ...

   1996601
5. bpowers

   changed the title [-]ci: add a release-mode lane so C-LEARN-scale tests can be un-ignored[/-] [+]ci: run the ignored C-LEARN-scale engine tests in an optimized build[/+] on Oct 12, 2026
6. bpowers

   added

   enhancementNew feature or requestNew feature or request

   on Oct 12, 2026
7. bpowers commented on Oct 12, 2026

   

   bpowers

   on Oct 12, 2026

   OwnerAuthor

   More actions

   Closing: already fixed on main.

   CI's gates job (.github/workflows/ci.yaml:161-192, on every push and PR to main) runs scripts/gates.sh, which runs every #[ignore]d simlin-engine test (--lib --test integration -- --ignored) under the optimized gates profile (Cargo.toml:82), added in ea8367a (#1080). Every remaining ignore carries the "run under the gates profile" reason, and the tests the issue named are either in the default suite (simulates_clearn, clearn_ltm_var_count_guardrail) or gated (clearn_ltm_slot_maxima_digest, simulate_ltm.rs:11700).
8. bpowers

   closed this as completedon Oct 12, 2026