Skip to content

Align EEG scoring fixes with versioned, recoverable validation - #50

Closed
abwoo wants to merge 55 commits into
mainfrom
codex/methodology-audit-20261023
Closed

abwoo wants to merge 55 commits into
mainfrom
codex/methodology-audit-20261023

Conversation

@abwoo

@abwoo abwoo commented Oct 23, 2026 •

Copy link
Copy Markdown
Owner

Problem and resulting behavior

PR #50 corrects weighted artifact uncertainty, paired covariance, missing HMM observations, spectral endpoints and validation provenance. Its review also exposed a growing mismatch between current behavior, historical diagnostics and execution status. This consolidation keeps the recommendation algorithm unchanged and makes those boundaries explicit.

Current recommendation rule

The current rule was introduced at d908041ff5aa5f8c62cf111e5b75eae397d2f950: retain the proposal unless the feasible baseline is significantly better under the reverse simultaneous bound (baseline - proposal + critical < 0). Indistinguishable evidence is labelled maintain; missing evidence is unassessed. An explicit baseline takes precedence over the average-reference repair comparison.

The earlier b27b435 rule instead retained the baseline when an upgrade was uncertain. Its cached replay numbers are historical diagnostics and do not establish the current rule's accuracy. Current scope and evidence ledger and methods now agree.

Consolidation

  • Normalize MATLAB empty/singleton/vector JSON representations only for schema-defined arrays. Keep source values, missing arms, planned coverage and provenance intact.
  • Re-aggregate the nine completed artifacts from 40805638604, without EEG fitting. Original algorithm SHA 6b26931 remains distinct from the processing revision.
  • Make full shadow searches explicit, so normal PR edits do not launch sixteen long searches or cancel an ongoing shadow run.
  • Add one-search checkpoint/slice recovery with code, seed, arm, variance and toolchain identity checks. Preserve completed leaves and ICA work, expose partial completion, and cache completed retries. Checkpointed executions remain separate from old unsliced paired means.
  • Preserve eight disconnected prototypes, their tests and four completed one-off audit workflows on archive/pr50-prototypes-20261030, and remove them from the delivery branch.
  • Replace appended round-by-round planning notes with maintained methods and an explicit evidence index. Keep docs/RESEARCH_READINESS.md absent.

Scientific limits

The original four-record development comparison at 4116ea1 failed its engineering admission tolerance; the paired difference interval included zero. Its historical accuracy/calibration values do not certify later revisions. The latest original shadow run has eight Standard results and only one complete pair. A one-pair bootstrap interval cannot establish population performance.

Fixed-family reranking and source decomposition do not validate the complete adaptive procedure, stable calibration or cross-record generalization. Independent confirmation remains unopened.

Validation

  • 101 affected cases passed in 41052090671; that run also exposed a new fixture failure, corrected before integration.
  • The remaining native interrupted/resumed-search proof passed, and the original-fragment MATLAB merge completed, in the successful steps of 41053098369. Its later Python failure is superseded below.
  • 41053921374 is successful: 13 Python tests passed, and all original paired values, intervals, verdicts and CSVs match (8 seeds, 10 assessable arm records, one complete pair). No MATLAB or ICA was restarted for this verification.
  • Across the scoped runs, 102 MATLAB cases passed. The native proof checks recommendation/score parity, unchanged cached candidate/ICA bytes, cached completion retry and identity mismatch rejection.
  • Final tree checks confirm all 42 production implementation blobs are unchanged, the tested helper/checker/test sources match their validation revisions, relative documentation links resolve, and the deleted research-readiness document remains absent.
  • Machine-readable provenance records the source, merge and checking commits separately. The native cross-worker workflow is prepared; a full scientific run through it is still pending.

MATLAB validation uses only R2026a + EEGLAB 2026.0.0. The integration uses [skip ci] to preserve completed checks and avoid a full rerun. Linked checks certify their stated source revisions and scope, rather than a new full compatibility run on the final documentation/integration SHA. No merge or release is performed.

abwoo added 8 commits October 23, 2026 16:50
…dependent validation gaps

Restore the full compatibility workflow after focused before/after diagnostics. Add ranking and group integration regressions, keep raw and gain-corrected benchmark readouts separate, and document the scientific validation needed beyond implementation checks.
@abwoo abwoo changed the title Preserve paired uncertainty in group shrinkage and correct artifact/PSD estimates Fix paired uncertainty and add independent EEG validation on MATLAB 2026 / EEGLAB 2026 Oct 23, 2026
abwoo added 19 commits October 23, 2026 18:17
… tail mass and freeze fresh confirmation cohorts
@abwoo abwoo changed the title Fix paired uncertainty and add independent EEG validation on MATLAB 2026 / EEGLAB 2026 Compare cleanup paths, expose artifact-risk uncertainty and validate optimized EEG processing Oct 25, 2026
abwoo added 14 commits October 25, 2026 11:04
… legacy controls and preserve completed fits
…provenance

Inspect the final samples with a full terminal window while retaining the original baseline. Reject overlapping or repeated participant identities before group transfer, validate complete imported result schemas and digest manifests, publish recovery artifacts transactionally, and classify invalid cached RNG state as checkpoint corruption.

Run isolated MATLAB R2026a / EEGLAB 2026.0.0 regressions on a separate review branch. Keep PR #50 and recovery run 40282756637 unchanged.
…ble score archives

Keep frozen pooled and legacy controls intact. Require simultaneous paired evidence before an explicit baseline is replaced, preserve per-trial statistics after checkpoint compaction, and validate affected algorithms against independent sampling and existing development artifacts without fitting EEG again.
…ge reference

Retain feasible interpolation when an unprotected average-reference candidate has only an uncertain score advantage. Apply the same rule to selection frequencies, keep explicit opt-out controls, and reject imported Balanced recommendations from the old rule. Validate only the affected decision with existing fitted candidates, bootstrap bounds and shrinkage.
Reuse the successful MATLAB R2026a / EEGLAB 2026.0.0 audits without restarting completed experiments. Add the new regression suite to future numerical CI and keep variance-only replay controls explicit. Record the improvement on seed 40103, the cost on 40102, and the limits of cached-candidate validation; do not promote the old admission result or open confirmation.
@abwoo abwoo changed the title Compare cleanup paths, expose artifact-risk uncertainty and validate optimized EEG processing Guard EEG pipeline selection and preserve reusable validation evidence Oct 27, 2026
abwoo added 13 commits October 27, 2026 14:16
…operators

Preserve legacy simulator bytes and RNG behavior while exposing optional source components. Replay cached ICA, channel, regression, rejection and weighting decisions without refitting. Separate sampling, retention, signal transformation and residual sources for development cases 40102 and 40103; do not tune the selector or open confirmation.
…ip ci]

Keep the successful fixed-operator source audit without restarting it. Preserve its numerical evidence, distinguish residual artifact from signal loss and finite-sample cancellation, and refine the interpretation of the reference guard. Make default-versus-traced tests self-contained, keep the frozen-simulator comparison in the audit workflow, and add the three portable tests to future numerical CI. Operational ranking is unchanged.
…ckpoint completeness, replay critical

- Rank.guardUpgrade now maintains the proposal unless the baseline is
  shown significantly better (reverse bound < 0); indistinguishable and
  missing evidence are told apart as 'maintain' vs 'unassessed'
- Rank.factorialModel gate relaxed from every-draw-complete to >= 20
  complete draws; shrink handles non-finite entries per element instead
  of vetoing the whole vector (NQC-053); choose excludes non-finite
  candidates before min()
- selectionFrequency splits repair-skipped resamples into their own share
- Artifact-variance fallback unified: objective mode, else
  ref.artifactVariance, else pooled (Measure default); Executor run()
  and execute() share one default
- Group.rank now runs Rank.guardReference per recording and vetoes
  group candidates blocked on every recording, marking unprotected
  recordings instead of silently skipping
- Executor saves filterCheck-excluded and adaptive (refineCutoffs /
  eogCheck) leaves, saves final ranking.mat, and resume() skips
  adaptive leaves the manifest predates
- replay_reference_guard recovers the production bestSet critical from
  all feasible candidates and reports productionGuardCritical
- METHODS 0.495976 / +1.83% now carry the full descriptive-mixed-estimator
  label; diagnose_paired_regression reports unavailable records instead
  of asserting
- Tests: factorial gate expectations updated; new guardUpgrade and
  shrink tests

Timer counterexample (reportedFitSeconds) deferred: the contradiction is
in historical artifacts, not current code; needs a decision on re-running
the old shard, not a code change.
- tests/test_algorithm_regressions.m: update 4 guard tests to the round-1
  guard semantics (maintain proposal unless baseline significantly better);
  share comparisons use >= because the new guard only moves shares toward
  the baseline (equality when no resample flips). No production code changed.
- benchmarks/pipecompare_open_summary.m: pad missing top-level struct
  fields with [] before checkpoint merge and cross-folder concatenation,
  so frozen archives predating the 'levels' field (source run fa295f9)
  concatenate with the current 8-field schema instead of horzcat failing.
  No stored values are dropped or rewritten; downstream never reads .levels.
MATLAB rejects S.(f) = [] on a nonscalar struct array ("Scalar structure
required for this assignment"), which is what failed the round-2 head at
pipecompare_open_summary line 71 when part held the 30 frozen records.
Pad missing top-level fields element by element instead, at both the
checkpoint merge and the cross-folder concatenation. No stored values
are dropped or rewritten; downstream still never reads .levels.
…rking

- Direct guardReference cases: significantly better repaired baseline replaces the skip
  (retain-baseline), indistinguishable evidence maintains the proposal, a biased repaired
  candidate is still used when it is the only eligible one, and the guard is idle when
  the proposal is already repaired or no repaired candidate is eligible.
- Group.rank case: a recording whose ranking table lacks diffHi is marked unprotected
  with reason 'ranking table lacks guard inputs', the group status is
  partially-unguarded, and no candidate is vetoed on that recording's behalf.
- Test-only change; no production logic touched.
… untouched)

- benchmarks/pipecompare_shadow_selection.m: per held-out seed (fresh range
  >= 20031, never design seeds 40101-40104, never locked confirmation
  41001-41030, asserted in script and workflow prep), fit one Balanced
  family and decide it twice: guarded (protectReferenceCleanup=true) vs
  unguarded diagnostic (protectReferenceCleanup=false), plus Standard
  (legacy-total) as control. Writes per-seed fragment JSON immediately and
  run-level JSON/CSV with paired differences, record-bootstrap CI, coverage,
  provenance passport and labels (fullyHeldOut=true, preprocessingRefitted=
  false, adaptiveLeavesIncluded=false meaning adaptive leaves are not
  re-derived per arm; a variant A verdict is conditional and cannot close
  P0-5). Guarded rerun must match the stored recommendation exactly or the
  script fails; same-pick pairs are asserted as a negative control.
- benchmarks/check_shadow_selection.py: standalone independent recompute
  checker (stdlib only). It implements the same unavailable semantics as
  diagnose_paired_regression.py, which cannot read shadow JSON (it reads
  development-optimization calibration CSV); modifying it risked existing
  CI, so the check lives in a new file instead.
- tests/test_shadow_selection.m: 4 tests through the mode=test hook
  (arm MSE gain-corrected/raw/labels, paired intersection/unavailable/
  verdicts, negative control/default gain, CSV/JSON consistency).
- .github/workflows/shadow-selection.yml: pull_request with path filter
  plus workflow_dispatch (defaults first_seed 20031, 8 seeds); prep validates
  the fresh range, per-seed matrix jobs (340 min job / 330 min MATLAB step,
  R2026a + EEGLAB 2026.0.0, checkouts pinned to the prep head SHA), collect
  merges fragments, runs the checker, and publishes shadow-selection only
  when the check passes.
- .github/workflows/matlab-tests.yml: register test_shadow_selection (1 line).
- docs/METHODS.md: append shadow section only (96 lines, 0 deletions);
  all previously published numbers unchanged.

Production files zero-change. Variant B (full-process shadow, the only
evidence that can close P0-5) is deliberately not in this commit.
- contractMatch used cellfun on a struct array (conditions), which MATLAB
  rejects, and its try/catch returned false on every call, so the Standard
  arm was unavailable on every Round 5 seed. It now uses arrayfun.
- Execution is split by search: one matrix job per seed per search
  (Standard-only and Balanced-only, 360-minute job / 350-minute step
  timeouts, fragment upload with if: always()). The Balanced-only job
  derives both Balanced arms from its single fitted family via a shared
  balancedArms helper (runSeed now calls the same helper, so the two
  paths cannot drift). The Standard-only job's MSE is kept only when
  its contract fingerprint matches the Balanced fingerprint at merge.
- Merge now combines arm fragments, allows partial coverage (run seeds =
  present seeds, nSeedsPlanned = planned seeds), and the collect job
  runs with if: always(), so a partial run is checked and published
  instead of being skipped. The checker reads arm fragments (single
  arm as JSON object or array) and verifies the fingerprint condition.
- Statistics, seeds, arm definitions and frozen criteria are unchanged;
  no production logic is touched. METHODS.md gains only a trailing
  Round 6 amendment; the Round 5 run is void as paired evidence.
- Tests: three new merge tests (matching fingerprint, mismatched
  fingerprint, partial only-Balanced) in test_shadow_selection.m.
Measure: add eyeBiasApplicable; scalar keeps old gate behaviour
(including missing-eye rejection), non-scalar gets NaN eyeBias/
eyeBiasSq and assessed=false with applicable=false, so 0 can only
mean 'assessed and no bias', never 'not assessed'.
Rank: gate only checks assessed when applicable; table default
for eyeBias is NaN, not 0; biasOf replaces non-finite eyeSq with 0
only for non-applicable objectives, giving an explicit shape-only
ranking bias instead of silently treating unknown bias as zero.
Tests: new testNonScalarEyeBiasIsNotZeroAndDoesNotBlock covers
the peak case (recommended, table NaN, finite objective) and the
scalar counterpart (still rejected as 'artifact bias unassessed').

Pushed now at user direction, cancelling in-progress Round 6
shadow jobs 20037/20038 Balanced; Round 6 paired evidence remains
partial (2 pairs) and is not completed by this commit. Not
verified until matlab-tests proves it in CI.
Single bundle (user 2026-10-28: do all doable items in this round,
LLM modules also bundled, attribution loss accepted; fast loop is
the round gate, full shadow is background proof, not the gate).
- Round 7 test fix: eyeBias_P3mean (was eyeBias_obj1).
- A1-A7 (crossfit finite guard, guard note, group single label,
  error list, calibration gain log, duplicate trial warning,
  parallel=false switch), bestSetUndefined flag (report-only),
  calibration coverage + summarizeByStratum (report-only),
  calibration rows target (single-target P3 unchanged),
  strict checkpoint opt-in, Executor per-step seconds.
- Shadow tools: planned coverage, filename-seed assert, duplicate
  assert, guard-reproduction/negative-control to unavailable,
  negative gain unavailable, per-seed CSV pick/selectedKey,
  workflow max-parallel 10->16, dispatch seed_list,
  EEGLAB SHA provenance.
- B3 zero-weight bootstrap handling (Measure).
- LLM transferable methods as new arms/prototypes, NOT weights:
  group-relative score, diffusion report, calibration-gate report
  (report-only helpers); successive halving, surrogate screen,
  bestSet strict, variant B M3-first design, LLM-assisted log
  review (local template, no LLM call, no external API).
  DeepSeek/Kimi weights are NOT embedded or called.
- Design-only, explicitly not production: checkpoint slicing
  resume (guarded error), CI cache activation, matlab-tests
  suite split, shadow fragment timing wiring, shadow merge
  MATLAB tests, crossfit dedicated test, successive-halving /
  surrogate / variant B shadow wiring.
- Sibling round8/ A-group work is superseded by this bundle
  (coordinator version already includes A items); do not merge
  both copies.
Not verified until CI proves it in MATLAB; cloud has no MATLAB.
Fast gate: matlab-tests + offline re-rank + 1-hour micro search;
full shadow runs in background and is reported separately.
…ent timing, add merge/crossfit/B3 tests

- tests/test_methodology.m: testNonScalarEyeBiasIsNotZeroAndDoesNotBlock now derives the column with matlab.lang.makeValidName('eyeBias_P3.mean') (eyeBias_P3_mean) instead of the Round 8 guess eyeBias_P3mean; plus crossfit finite-value guard test and B3 zero-weight test.
- benchmarks/pipecompare_shadow_selection.m: report-only timing wiring (runSearch wallSeconds + per-step-type median/max, never a sum); no pick/MSE change.
- tests/test_shadow_selection.m: 3 merge-mode tests (filename seed mismatch, duplicate seed-arm, CSV pick/selectedKey match).
- docs/METHODS.md: append-only Round 9 note.
…earch

- Layered fast diagnostics, timeout slice manifests, variant B gate
  default closed, reporting-only failure visibility, new MATLAB/Python tests.
- Slice-diagnostic only: micro-search workflow fragments are labelled
  partialSearch and are never held-out evidence.
- executionReady=false: reduced-family execution is NOT wired.
  pop_pipecompare has no icaSeeds/refineCutoffs/eogCheck options and
  Presets.searchOptions() hardcodes production values; Presets pass-through
  + trial-subset preregistration + MATLAB proof remain a separate next round.
- No production pick change: Rank, Measure and Calibration untouched.
Integrate the reviewed changes as one PR commit, retaining the isolated development history and completed artifacts on their review/archive branches. All 42 production blobs are unchanged. Align the current d908041 guard description, label historical evidence by source SHA, repair MATLAB singleton/empty summary contracts, archive eight prototypes and four one-off workflows, and add strict checkpointed search delivery.

Preserve 101 passing affected cases and the separately successful native resume proof; 13 Python tests and all original paired values/CIs/verdicts/CSVs passed in 41053921374. Reuse completed outputs without restarting MATLAB or ICA after the native proof. Final documentation/workflow registration is not claimed as a new full-head compatibility run.

No new full-record experiment, confirmation, merge or release.
@abwoo abwoo changed the title Guard EEG pipeline selection and preserve reusable validation evidence Align EEG scoring fixes with versioned, recoverable validation Oct 30, 2026
@abwoo abwoo closed this Nov 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant