chore(tiers): release kit — discovered test suite, bench validation kit, review before validation

.agents/skills/g6-release/SKILL.md

@@ -29,6 +29,9 @@ Routine pushes to a feature branch need no question.
   (`.github/pull_request_template.md`) pre-fills it.
 - New localStorage keys / migration markers / window names follow the coexistence rules
   (typed monotonic markers, never change a key's meaning, tier-suffixed window names).
+- Keep the PR **mergeable with main**: GitHub runs no CI on a PR with merge conflicts, so a
+  conflicting PR shows no checks at all. Merge main in (no force-push needed) when it drifts.
+- New tests only need the `tests/test-*` / `tests/validate-*` name: `pixi run test` discovers them.
 
 ## Commands (run from a checkout of the repo; `pixi run gh auth login` once)
 
@@ -37,12 +40,28 @@ Routine pushes to a feature branch need no question.
 | Build + publish a candidate | `pixi run candidate -- 219 178 [--headline "…"]` | pins each PR head, merges `--no-ff`, release commit, pushes `release/<date>`, opens a draft release PR, waits for CI, then points `next` at it and dispatches the deploy |
 | Conflict | resolve in `.tiers/candidate`, `git add`, `git commit --no-edit`, then `pixi run candidate -- --continue` (or `--abort`) | two PRs that keep colliding belong in one `batch/<name>` branch |
 | Dry run | `pixi run candidate -- 219 --no-push` then `pixi run next-preview -- --next release/<date>` and `python -m http.server -d _site 8080` | nothing leaves the machine |
+| Adversarial review | `codex-diff-review` skill on `origin/main...origin/release/<date>` | **before** asking for bench time; fix or drop, rebuild, then validate (release-process.md) |
+| Bench check | `pixi run bench-check -- LOG --hold epoch_a_frame25=25 --hold epoch_b_frame75=75 --channel next --build <sha>` | protocol `protocols/validation/bench_closed_loop_2x10.yaml`; steps in release-process.md "Bench validation kit" |
 | What's live | `pixi run tiers -- status` | head drift / main drift → rebuild (rc+1, validation resets) |
 | Validation note | `pixi run tiers -- validate --rig rig5 --by Isabel --notes "…"`, or `--notes-file /abs/path.md` for anything multi-line or with `` ` `` (pixi re-parses `--notes`) | names the exact candidate SHA; test **between sessions** |
 | Promote | `pixi run release` | re-checks everything, prints the Slack draft, asks for the release name, merge commit + tag |
 | Hotfix | `pixi run candidate -- --hotfix "why" 230` → `pixi run release -- --pr N` | never moves /next/ |
 | Roll back | `pixi run tiers -- rollback --to release-<prev> --revert release-<bad>` | the next push to main undoes the emergency deploy — land the revert first |
 
+## Validating through the Chrome extension
+
+The lab's github.io origin has the course pipeline switched on, so a completed Run experiment
+auto-commits its log to the course repo. For a validation run, override it **in the test tab only**
+(lost on reload; never change stored settings), then load the validation protocol:
+
+```js
+const orig = Studio.courseSettings;
+Studio.courseSettings = () => Object.assign({}, orig(), { direct: false });
+```
+
+Native dialogs (the NEXT confirm, the serial-port chooser, rig-mismatch confirms) block the
+extension, so the user clicks Connect, Run and OK. Check the result with `pixi run bench-check`.
+
 ## After a release
 
 - Post the Slack announcement (the tool prints a draft; the claude.ai Slack connector can post it

CLAUDE.md

@@ -115,7 +115,8 @@ Node, Prettier, Python, and `websockets` from conda-forge — **there is no npm,
 
 ```bash
 pixi install          # one-time: provisions Node + Prettier + Python
-pixi run test         # full JS test suite (node)
+pixi run test         # full suite: every tests/test-*.js|py + tests/validate-*.js (scripts/run-tests.py)
+pixi run test -- wire # only files whose name contains "wire"
 pixi run format       # Prettier --write over **/*.js
 pixi run format-check # Prettier --check (matches the old CI check)
 pixi run bridge       # FicTrac closed-loop bridge (see fictrac-bridge/)
@@ -127,7 +128,9 @@ only) at `js/vendor/yaml/browser/` and serves both the browser (via the import
 map in `experiment_designer_v3.html`) and Node — no Node-specific build is
 committed. The v3 modules `import 'yaml'` (a bare specifier); under Node that's
 resolved to the vendored browser build by `tests/vendor-yaml.register.mjs` (a
-`--import` resolve hook), which the `test` task wires in for the v3 suite.
+`--import` resolve hook), which `scripts/run-tests.py` passes to every Node test.
+A new test file only needs the `tests/test-*` (or `tests/validate-*`) name to be run; there is
+no list to edit, so parallel PRs no longer conflict in `pixi.toml`.
 `require()`-ing that ESM build is why `nodejs >= 22.12` is pinned. When bumping
 the vendored `yaml`, replace only `js/vendor/yaml/browser/` (+ `LICENSE`).
 

docs/development/release-process.md

@@ -100,6 +100,14 @@ same on macOS, Linux and Windows (CI runs the tooling tests on all three).
 - **Pre-tier PRs** that already bumped the footer and wrote their own notes entry are adopted, not
   double-bumped (their version and entry are kept; only the timestamp is refreshed).
 
+## Before validation: adversarial review of the candidate
+
+Once a candidate is built, and **before asking anyone for bench time**, get an adversarial review of
+exactly what would ship: the diff `origin/main...origin/release/<name>`. Claude sessions use the
+`codex-diff-review` skill (Codex standard + adversarial passes, reconciled with Claude's own review).
+Fix or drop what it finds, rebuild (rc+1), and only then validate. On 2026-09-29, rc1 and rc2 were
+bench-validated first; the review then held #219 back, so both bench runs had to be repeated on rc3.
+
 ## Validation
 
 - Validate on a rig **between sessions**. Never open `/next/` on a rig PC while an experiment is
@@ -113,6 +121,27 @@ same on macOS, Linux and Windows (CI runs the tooling tests on all three).
 - If a candidate touches `fictrac-bridge/`, testers run the bridge from a checkout of the
   candidate's `release/<name>` branch (a pinned commit, not a moving ref).
 
+### Bench validation kit (G6 2×10 fly-on-ball bench, ~5 min)
+
+1. Start the bridge with a log directory, and a stationary simulated fly:
+   `pixi run bridge -- --log-dir <dir>` and `pixi run sim -- --noise 0`.
+2. Open `https://reiserlab.github.io/webDisplayTools/next/arena_studio.html` and load
+   `protocols/validation/bench_closed_loop_2x10.yaml`. It needs the CSHL course pattern bundle on the
+   SD card. Connect the bridge, then Connect the arena.
+3. **Turn the course-repo upload off for the test.** The lab browsers have the course pipeline
+   configured (token + `reiserlab/cshl-2026-course` + direct commit), so a completed
+   **Run experiment** commits its log into the course data as the configured bench id. Use a browser
+   profile without the course token, or switch off "Commit directly" for the session. Never
+   validate with **Test experiment** instead: test runs don't write `run_metadata` to the bridge log.
+4. **Run experiment** (confirm the NEXT dialog). About 22 s.
+5. Check the log:
+   `pixi run bench-check -- <dir>/arena-log-*.jsonl --hold epoch_a_frame25=25 --hold epoch_b_frame75=75 --channel next --build <candidate sha>`.
+   It checks that the run completed, every arena command was ok, there were no error glyphs, the
+   closed loop held frames 25 and 75, and the log names this tier and build. It also prints the
+   controller firmware and the panel inventory. Exit status 0 = pass.
+6. Post the note with `--notes-file`, then free the bench: disconnect, close the tab, stop the
+   bridge and the simulator.
+
 ## Releasing
 
 `pixi run release` re-checks everything (heads unchanged, CI green, main not moved beyond docs,

pixi.toml

@@ -21,7 +21,10 @@ serve = "python -m http.server 8000 --bind 127.0.0.1"
 # The full test suite. yaml is vendored (js/vendor/yaml), so no package install.
 # The v3 suite pulls in ESM modules that `import 'yaml'`; --import maps that bare
 # specifier to the vendored copy (tests/vendor-yaml.register.mjs).
-test = "node tests/test-project-skills.js && node tests/validate-arena-calculations.js && node tests/test-protocol-roundtrip.js && node --import ./tests/vendor-yaml.register.mjs tests/test-protocol-roundtrip-v3.js && node --import ./tests/vendor-yaml.register.mjs tests/test-runtime-controls.js && node tests/test-plugin-registry.js && node tests/test-arena-wire-g6.js && node tests/test-arena-link.js && node tests/test-arena-runner-g6.js && node tests/test-arena-session.js && node tests/test-fictrac-bridge-client.js && node tests/test-run-log.js && node tests/test-bin-classifier.js && node tests/test-pattern-set.js && node tests/test-pat-preview.js && node tests/test-studio-meta.js && node tests/test-build-channel.js && node tests/test-studio-analog-in.js && node tests/test-studio-panel-inventory.js && node tests/test-studio-url-state.js && node tests/test-studio-runlog-adapter.js && node tests/test-studio-github.js && node tests/test-studio-canmutate.js && node tests/test-studio-protocols-index.js && node tests/test-g6-tall-rigs.js && node tests/test-arena-studio-alt.js && node tests/test-three-viewer-led-glow.js && node tests/test-three-viewer-preview-omissions.js && node tests/test-arena-replay-viewer-protocol.js && node tests/test-kinematics.js && node tests/test-runlog-replay.js && node tests/test-studio-replay.js && node tests/test-fly-gait.js && node tests/test-runlog-format.js && node tests/test-metadata-yaml.js && python tests/test-bridge-behavior.py && python tests/test-build-runlog-index.py && node tests/test-studio-postmortem.js && node tests/test-arena-telemetry.js && python tests/test-wedge-scan.py && python tests/test-runlog-check.py && node tests/test-trial-quality.js && python tests/test-telemetry-report.py && python tests/test-fictrac-sim.py && node --import ./tests/vendor-yaml.register.mjs tests/test-studio-runtime-vars.js && python tests/test-tiers-candidate.py"
+# Every tests/test-*.js|py and tests/validate-*.js, discovered by name (scripts/run-tests.py).
+# A new test only needs the naming convention; no shared list to edit, so PRs stop conflicting here.
+# pixi run test -- <substring…> runs a subset; -- --list shows what would run.
+test = "python scripts/run-tests.py"
 # ── Two-tier releases: Production + Next (docs/development/release-process.md) ──
 # pixi run candidate -- 219 178   build a frozen Next candidate from those PRs
 # pixi run release                 promote the validated candidate to Production
@@ -31,6 +34,8 @@ tiers = "python scripts/tiers/candidate.py"
 candidate = "python scripts/tiers/candidate.py build"
 release = "python scripts/tiers/candidate.py release"
 next-preview = "python scripts/tiers/stamp.py preview"
+# pixi run bench-check -- LOG [--hold COND=FRAME …] [--channel next] [--build SHA]   check a bench-validation run
+bench-check = "python scripts/bench-check.py"
 test-tiers = "python tests/test-tiers-candidate.py"
 format = "prettier --write \"**/*.js\""
 format-check = "prettier --check \"**/*.js\""

protocols/validation/bench_closed_loop_2x10.yaml

@@ -0,0 +1,102 @@
+version: 3
+
+# Release-candidate bench check for a G6 2x10 fly-on-ball arena (docs/development/release-process.md,
+# "Bench validation"). Closed-loop trials must OPEN AND HOLD on their trialParams frame_index:
+# epoch A on frame 25, epoch B on frame 75, 8 s each, with a STATIONARY simulated fly:
+#     pixi run bridge -- --log-dir <dir>      (bridge >= 3.4)
+#     pixi run sim -- --noise 0
+# Then check the bridge log:
+#     pixi run bench-check -- <dir>/arena-log-*.jsonl --hold epoch_a_frame25=25 --hold epoch_b_frame75=75
+# Needs the CSHL course pattern bundle on the SD card (pattern_ID 23 = course_bg, 36 =
+# p3_heisenberg_ts). Trimmed from rig7's p3-conditioning-closedloop-short-negative.yaml; first
+# used to bench-check #225 (2026-09-28). About 22 s.
+
+experiment_info:
+  name: "bench_closed_loop_2x10"
+  author: "Reiser Lab"
+  date_created: 2026-09-27
+
+rig: "./configs/rigs/cshl_g6_2x10_ball.yaml"
+
+plugins:
+  - name: "fictrac"
+    type: "class"
+    matlab:
+      class: "FicTracPlugin"
+    config:
+      bridge_url: "ws://localhost:8765"
+      fictrac_port: 60000
+      proto: "udp"
+      offset: 0
+
+variables:
+  pattern_name: &pattern_name "p3_heisenberg_ts"
+  pattern_id: &pattern_id 36
+  trial_dur: &trial_dur 8
+  start_bg_dur: &start_bg_dur 3
+  cl_gain: &cl_gain -1
+
+experiment:
+  - "start_bg"
+  - "epoch_a_frame25"
+  - "epoch_b_frame75"
+  - "start_bg"
+
+conditions:
+  - name: "start_bg"
+    commands:
+      - type: "controller"
+        command_name: "trialParams"
+        pattern: "course_bg"
+        pattern_ID: 23
+        duration: *start_bg_dur
+        mode: 2
+        frame_index: 0
+        frame_rate: 0
+        gain: 0
+      - type: "wait"
+        duration: *start_bg_dur
+
+  - name: "epoch_a_frame25"
+    commands:
+      - type: "controller"
+        command_name: "trialParams"
+        pattern: *pattern_name
+        pattern_ID: *pattern_id
+        duration: *trial_dur
+        mode: 3
+        frame_index: 25
+        frame_rate: 0
+        gain: 0
+      - type: "plugin"
+        plugin_name: "fictrac"
+        command_name: "startClosedLoop"
+        params:
+          coupling: *cl_gain
+      - type: "wait"
+        duration: *trial_dur
+      - type: "plugin"
+        plugin_name: "fictrac"
+        command_name: "stopClosedLoop"
+
+  - name: "epoch_b_frame75"
+    commands:
+      - type: "controller"
+        command_name: "trialParams"
+        pattern: *pattern_name
+        pattern_ID: *pattern_id
+        duration: *trial_dur
+        mode: 3
+        frame_index: 75
+        frame_rate: 0
+        gain: 0
+      - type: "plugin"
+        plugin_name: "fictrac"
+        command_name: "startClosedLoop"
+        params:
+          coupling: *cl_gain
+      - type: "wait"
+        duration: *trial_dur
+      - type: "plugin"
+        plugin_name: "fictrac"
+        command_name: "stopClosedLoop"