Repository navigation
Conversation
…dependent validation gaps Restore the full compatibility workflow after focused before/after diagnostics. Add ranking and group integration regressions, keep raw and gain-corrected benchmark readouts separate, and document the scientific validation needed beyond implementation checks.
…entage of individual differences
…held-out refitting
…d shard new-seed calibration
…permit summary-only recovery
… isolate evaluation failures
…validation summaries
…ng validation on regression checks
…ong refits finish
…itional ranking scores
… replace superseded summary waiters
… benchmark clocks
…ns and label restricted-family controls
…ort validation coverage
… tail mass and freeze fresh confirmation cohorts
…lic and synthetic results
…act-risk uncertainty
…ult repair and frozen transfer support
…wn and explain sensitivity in results
…with explicit legacy controls
… legacy controls and preserve completed fits
…y before optional ICA
…run paired methods on one backend
…hout weakening validation
…provenance Inspect the final samples with a full terminal window while retaining the original baseline. Reject overlapping or repeated participant identities before group transfer, validate complete imported result schemas and digest manifests, publish recovery artifacts transactionally, and classify invalid cached RNG state as checkpoint corruption. Run isolated MATLAB R2026a / EEGLAB 2026.0.0 regressions on a separate review branch. Keep PR #50 and recovery run 40282756637 unchanged.
…ble score archives Keep frozen pooled and legacy controls intact. Require simultaneous paired evidence before an explicit baseline is replaced, preserve per-trial statistics after checkpoint compaction, and validate affected algorithms against independent sampling and existing development artifacts without fitting EEG again.
…ge reference Retain feasible interpolation when an unprotected average-reference candidate has only an uncertain score advantage. Apply the same rule to selection frequencies, keep explicit opt-out controls, and reject imported Balanced recommendations from the old rule. Validate only the affected decision with existing fitted candidates, bootstrap bounds and shrinkage.
Reuse the successful MATLAB R2026a / EEGLAB 2026.0.0 audits without restarting completed experiments. Add the new regression suite to future numerical CI and keep variance-only replay controls explicit. Record the improvement on seed 40103, the cost on 40102, and the limits of cached-candidate validation; do not promote the old admission result or open confirmation.
…operators Preserve legacy simulator bytes and RNG behavior while exposing optional source components. Replay cached ICA, channel, regression, rejection and weighting decisions without refitting. Separate sampling, retention, signal transformation and residual sources for development cases 40102 and 40103; do not tune the selector or open confirmation.
…ip ci] Keep the successful fixed-operator source audit without restarting it. Preserve its numerical evidence, distinguish residual artifact from signal loss and finite-sample cancellation, and refine the interpretation of the reference guard. Make default-versus-traced tests self-contained, keep the frozen-simulator comparison in the audit workflow, and add the three portable tests to future numerical CI. Operational ranking is unchanged.
…ckpoint completeness, replay critical - Rank.guardUpgrade now maintains the proposal unless the baseline is shown significantly better (reverse bound < 0); indistinguishable and missing evidence are told apart as 'maintain' vs 'unassessed' - Rank.factorialModel gate relaxed from every-draw-complete to >= 20 complete draws; shrink handles non-finite entries per element instead of vetoing the whole vector (NQC-053); choose excludes non-finite candidates before min() - selectionFrequency splits repair-skipped resamples into their own share - Artifact-variance fallback unified: objective mode, else ref.artifactVariance, else pooled (Measure default); Executor run() and execute() share one default - Group.rank now runs Rank.guardReference per recording and vetoes group candidates blocked on every recording, marking unprotected recordings instead of silently skipping - Executor saves filterCheck-excluded and adaptive (refineCutoffs / eogCheck) leaves, saves final ranking.mat, and resume() skips adaptive leaves the manifest predates - replay_reference_guard recovers the production bestSet critical from all feasible candidates and reports productionGuardCritical - METHODS 0.495976 / +1.83% now carry the full descriptive-mixed-estimator label; diagnose_paired_regression reports unavailable records instead of asserting - Tests: factorial gate expectations updated; new guardUpgrade and shrink tests Timer counterexample (reportedFitSeconds) deferred: the contradiction is in historical artifacts, not current code; needs a decision on re-running the old shard, not a code change.
- tests/test_algorithm_regressions.m: update 4 guard tests to the round-1 guard semantics (maintain proposal unless baseline significantly better); share comparisons use >= because the new guard only moves shares toward the baseline (equality when no resample flips). No production code changed. - benchmarks/pipecompare_open_summary.m: pad missing top-level struct fields with [] before checkpoint merge and cross-folder concatenation, so frozen archives predating the 'levels' field (source run fa295f9) concatenate with the current 8-field schema instead of horzcat failing. No stored values are dropped or rewritten; downstream never reads .levels.
MATLAB rejects S.(f) = [] on a nonscalar struct array ("Scalar structure
required for this assignment"), which is what failed the round-2 head at
pipecompare_open_summary line 71 when part held the 30 frozen records.
Pad missing top-level fields element by element instead, at both the
checkpoint merge and the cross-folder concatenation. No stored values
are dropped or rewritten; downstream still never reads .levels.
…rking - Direct guardReference cases: significantly better repaired baseline replaces the skip (retain-baseline), indistinguishable evidence maintains the proposal, a biased repaired candidate is still used when it is the only eligible one, and the guard is idle when the proposal is already repaired or no repaired candidate is eligible. - Group.rank case: a recording whose ranking table lacks diffHi is marked unprotected with reason 'ranking table lacks guard inputs', the group status is partially-unguarded, and no candidate is vetoed on that recording's behalf. - Test-only change; no production logic touched.
… untouched) - benchmarks/pipecompare_shadow_selection.m: per held-out seed (fresh range >= 20031, never design seeds 40101-40104, never locked confirmation 41001-41030, asserted in script and workflow prep), fit one Balanced family and decide it twice: guarded (protectReferenceCleanup=true) vs unguarded diagnostic (protectReferenceCleanup=false), plus Standard (legacy-total) as control. Writes per-seed fragment JSON immediately and run-level JSON/CSV with paired differences, record-bootstrap CI, coverage, provenance passport and labels (fullyHeldOut=true, preprocessingRefitted= false, adaptiveLeavesIncluded=false meaning adaptive leaves are not re-derived per arm; a variant A verdict is conditional and cannot close P0-5). Guarded rerun must match the stored recommendation exactly or the script fails; same-pick pairs are asserted as a negative control. - benchmarks/check_shadow_selection.py: standalone independent recompute checker (stdlib only). It implements the same unavailable semantics as diagnose_paired_regression.py, which cannot read shadow JSON (it reads development-optimization calibration CSV); modifying it risked existing CI, so the check lives in a new file instead. - tests/test_shadow_selection.m: 4 tests through the mode=test hook (arm MSE gain-corrected/raw/labels, paired intersection/unavailable/ verdicts, negative control/default gain, CSV/JSON consistency). - .github/workflows/shadow-selection.yml: pull_request with path filter plus workflow_dispatch (defaults first_seed 20031, 8 seeds); prep validates the fresh range, per-seed matrix jobs (340 min job / 330 min MATLAB step, R2026a + EEGLAB 2026.0.0, checkouts pinned to the prep head SHA), collect merges fragments, runs the checker, and publishes shadow-selection only when the check passes. - .github/workflows/matlab-tests.yml: register test_shadow_selection (1 line). - docs/METHODS.md: append shadow section only (96 lines, 0 deletions); all previously published numbers unchanged. Production files zero-change. Variant B (full-process shadow, the only evidence that can close P0-5) is deliberately not in this commit.
- contractMatch used cellfun on a struct array (conditions), which MATLAB rejects, and its try/catch returned false on every call, so the Standard arm was unavailable on every Round 5 seed. It now uses arrayfun. - Execution is split by search: one matrix job per seed per search (Standard-only and Balanced-only, 360-minute job / 350-minute step timeouts, fragment upload with if: always()). The Balanced-only job derives both Balanced arms from its single fitted family via a shared balancedArms helper (runSeed now calls the same helper, so the two paths cannot drift). The Standard-only job's MSE is kept only when its contract fingerprint matches the Balanced fingerprint at merge. - Merge now combines arm fragments, allows partial coverage (run seeds = present seeds, nSeedsPlanned = planned seeds), and the collect job runs with if: always(), so a partial run is checked and published instead of being skipped. The checker reads arm fragments (single arm as JSON object or array) and verifies the fingerprint condition. - Statistics, seeds, arm definitions and frozen criteria are unchanged; no production logic is touched. METHODS.md gains only a trailing Round 6 amendment; the Round 5 run is void as paired evidence. - Tests: three new merge tests (matching fingerprint, mismatched fingerprint, partial only-Balanced) in test_shadow_selection.m.
Measure: add eyeBiasApplicable; scalar keeps old gate behaviour (including missing-eye rejection), non-scalar gets NaN eyeBias/ eyeBiasSq and assessed=false with applicable=false, so 0 can only mean 'assessed and no bias', never 'not assessed'. Rank: gate only checks assessed when applicable; table default for eyeBias is NaN, not 0; biasOf replaces non-finite eyeSq with 0 only for non-applicable objectives, giving an explicit shape-only ranking bias instead of silently treating unknown bias as zero. Tests: new testNonScalarEyeBiasIsNotZeroAndDoesNotBlock covers the peak case (recommended, table NaN, finite objective) and the scalar counterpart (still rejected as 'artifact bias unassessed'). Pushed now at user direction, cancelling in-progress Round 6 shadow jobs 20037/20038 Balanced; Round 6 paired evidence remains partial (2 pairs) and is not completed by this commit. Not verified until matlab-tests proves it in CI.
Single bundle (user 2026-10-28: do all doable items in this round, LLM modules also bundled, attribution loss accepted; fast loop is the round gate, full shadow is background proof, not the gate). - Round 7 test fix: eyeBias_P3mean (was eyeBias_obj1). - A1-A7 (crossfit finite guard, guard note, group single label, error list, calibration gain log, duplicate trial warning, parallel=false switch), bestSetUndefined flag (report-only), calibration coverage + summarizeByStratum (report-only), calibration rows target (single-target P3 unchanged), strict checkpoint opt-in, Executor per-step seconds. - Shadow tools: planned coverage, filename-seed assert, duplicate assert, guard-reproduction/negative-control to unavailable, negative gain unavailable, per-seed CSV pick/selectedKey, workflow max-parallel 10->16, dispatch seed_list, EEGLAB SHA provenance. - B3 zero-weight bootstrap handling (Measure). - LLM transferable methods as new arms/prototypes, NOT weights: group-relative score, diffusion report, calibration-gate report (report-only helpers); successive halving, surrogate screen, bestSet strict, variant B M3-first design, LLM-assisted log review (local template, no LLM call, no external API). DeepSeek/Kimi weights are NOT embedded or called. - Design-only, explicitly not production: checkpoint slicing resume (guarded error), CI cache activation, matlab-tests suite split, shadow fragment timing wiring, shadow merge MATLAB tests, crossfit dedicated test, successive-halving / surrogate / variant B shadow wiring. - Sibling round8/ A-group work is superseded by this bundle (coordinator version already includes A items); do not merge both copies. Not verified until CI proves it in MATLAB; cloud has no MATLAB. Fast gate: matlab-tests + offline re-rank + 1-hour micro search; full shadow runs in background and is reported separately.
…ent timing, add merge/crossfit/B3 tests
- tests/test_methodology.m: testNonScalarEyeBiasIsNotZeroAndDoesNotBlock now derives the column with matlab.lang.makeValidName('eyeBias_P3.mean') (eyeBias_P3_mean) instead of the Round 8 guess eyeBias_P3mean; plus crossfit finite-value guard test and B3 zero-weight test.
- benchmarks/pipecompare_shadow_selection.m: report-only timing wiring (runSearch wallSeconds + per-step-type median/max, never a sum); no pick/MSE change.
- tests/test_shadow_selection.m: 3 merge-mode tests (filename seed mismatch, duplicate seed-arm, CSV pick/selectedKey match).
- docs/METHODS.md: append-only Round 9 note.
…earch - Layered fast diagnostics, timeout slice manifests, variant B gate default closed, reporting-only failure visibility, new MATLAB/Python tests. - Slice-diagnostic only: micro-search workflow fragments are labelled partialSearch and are never held-out evidence. - executionReady=false: reduced-family execution is NOT wired. pop_pipecompare has no icaSeeds/refineCutoffs/eogCheck options and Presets.searchOptions() hardcodes production values; Presets pass-through + trial-subset preregistration + MATLAB proof remain a separate next round. - No production pick change: Rank, Measure and Calibration untouched.
Integrate the reviewed changes as one PR commit, retaining the isolated development history and completed artifacts on their review/archive branches. All 42 production blobs are unchanged. Align the current d908041 guard description, label historical evidence by source SHA, repair MATLAB singleton/empty summary contracts, archive eight prototypes and four one-off workflows, and add strict checkpointed search delivery. Preserve 101 passing affected cases and the separately successful native resume proof; 13 Python tests and all original paired values/CIs/verdicts/CSVs passed in 41053921374. Reuse completed outputs without restarting MATLAB or ICA after the native proof. Final documentation/workflow registration is not claimed as a new full-head compatibility run. No new full-record experiment, confirmation, merge or release.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem and resulting behavior
PR #50 corrects weighted artifact uncertainty, paired covariance, missing HMM observations, spectral endpoints and validation provenance. Its review also exposed a growing mismatch between current behavior, historical diagnostics and execution status. This consolidation keeps the recommendation algorithm unchanged and makes those boundaries explicit.
Current recommendation rule
The current rule was introduced at
d908041ff5aa5f8c62cf111e5b75eae397d2f950: retain the proposal unless the feasible baseline is significantly better under the reverse simultaneous bound (baseline - proposal + critical < 0). Indistinguishable evidence is labelledmaintain; missing evidence isunassessed. An explicit baseline takes precedence over the average-reference repair comparison.The earlier
b27b435rule instead retained the baseline when an upgrade was uncertain. Its cached replay numbers are historical diagnostics and do not establish the current rule's accuracy. Current scope and evidence ledger and methods now agree.Consolidation
6b26931remains distinct from the processing revision.docs/RESEARCH_READINESS.mdabsent.Scientific limits
The original four-record development comparison at
4116ea1failed its engineering admission tolerance; the paired difference interval included zero. Its historical accuracy/calibration values do not certify later revisions. The latest original shadow run has eight Standard results and only one complete pair. A one-pair bootstrap interval cannot establish population performance.Fixed-family reranking and source decomposition do not validate the complete adaptive procedure, stable calibration or cross-record generalization. Independent confirmation remains unopened.
Validation
MATLAB validation uses only R2026a + EEGLAB 2026.0.0. The integration uses
[skip ci]to preserve completed checks and avoid a full rerun. Linked checks certify their stated source revisions and scope, rather than a new full compatibility run on the final documentation/integration SHA. No merge or release is performed.