Skip to content

Registry: validate at the default tolerance over the experiment's horizon - #13

Merged
JKRT merged 1 commit into
masterfrom
registry-strict
Oct 27, 2026
Merged

JKRT merged 1 commit into
masterfrom
registry-strict

Conversation

@JKRT

@JKRT JKRT commented Oct 27, 2026

Copy link
Copy Markdown
Member

Takes the relaxed validation settings (atol/reltol above 0.01/3e-3, a stopTime short of the experiment's) off the relaxed entries of models that validated: 113 for MSL 4.1.0, 123 for 3.2.3. Solvers, observed filters, mappings and reference files are unchanged; a stopTime stays only where the reference ends before the experiment (ChuaCircuit 4.1.0 goes to its reference's end, 3000 s).

Checked by running each relaxed model in OM.jl through the harness at the default rule over the full horizon, and against omc on a sample:

  • Strict passes (95 for 4.1.0, 105 for 3.2.3): the relaxations were stale, the count does not change.
  • Strict failures, 18 per version: they now count as failing; each issue notes the finding.
    • Both versions: Electrical.Digital BUF3S, DFFREG, DFFREGL, DFFREGSRH, DFFREGSRL, HalfAdder, INV3S, NRXFER, NXFER, WiredX (OM.jl's logic outputs differ, omc matches the reference); OpAmps.SchmittTrigger; MultiBody PointGravity; FluidHeatFlow OneMass (3-5 K off), PumpDropOut and ParallelPumpDropOut (Unstable after the drop-out).
    • Disputed (omc also misses the reference): OpAmps.Multivibrator, MultiBody DoublePendulumInitTip, and FluxTubes ForceCurrentBehaviour (4.1.0) or ChuaCircuit's 50000 s chaotic reference (3.2.3).

Expected from the next coverage runs: MSL 4.1.0 about 482/534 validate (was 500), 3.2.3 about 370/425 (was 388).

🤖 Generated with Claude Code

…izon

The relaxed validation settings (atol/reltol above 0.01/3e-3, a stopTime short
of the experiment's) are off the relaxed entries of models that validated:
113 for MSL 4.1.0, 123 for 3.2.3. 95 and 105 of them validate strictly (OM.jl
runs at the default rule over the full horizon, cross-checked with omc); 18
per version do not: their issue notes the finding, and they count as failing
until fixed. A stopTime stays only where the reference ends before the
experiment; ChuaCircuit 4.1.0 goes to its reference's end, 3000 s.

Expected from the next coverage runs: MSL 4.1.0 about 482/534 validate
(was 500), 3.2.3 about 370/425 (was 388).

Assisted-by: Claude Opus 5.5
@JKRT
JKRT merged commit 7e35fd3 into master Oct 27, 2026
2 checks passed
@JKRT
JKRT deleted the registry-strict branch October 27, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant