LIM-1457: Secondary usefulness experiment (independent replication of #70) by blocksorg[bot] · Pull Request #96 · Limen-Neural/brainstem-daemon

blocksorg · 2026-10-17T00:46:53Z

Summary

This adds a secondary usefulness experiment for #70 under experiments/usefulness-secondary/. The primary experiment, merged in #94, stays at experiments/usefulness/. This one is an independent replication with its own fixture, workload, drivers, and summarizer, kept as corroborating evidence as the maintainer requested. It shares no files with the primary experiment, which this PR leaves unchanged.

The branch is rebuilt on current main (Rust 1.99.0, neuromod 0.7.0), and all commits are by blocksorg[bot]:

0f15197 Pre-registration. It lands before any measurement. It adds:
- PROTOCOL.md with checks C1–C12, T1–T3, and P1–P2, the decision rules, and an explicitly pre-registered I1 run-inventory gate;
- a versioned synthetic, untrained 16 LIF × 8 channel fixture and its manifest;
- the workload and the live, simulation, and transport configs. The transport config uses loopback ports 15565–15567, which differ from the primary's.
0233773 Tooling. This is the measured commit.
- examples/usefulness_secondary_inprocess.rs runs daemon vs direct-engine baseline, with live, simulation, and mask variants.
- examples/usefulness_secondary_transport.rs runs the real binary over ZeroMQ with /readyz and SIGTERM.
- examples/usefulness_secondary_common/ holds shared helpers.
- The examples are [[example]] entries gated on corpus-ipc, and add no new dependencies.
- It also adds run.sh and summarize.py. The summarizer is stdlib only and fails both unevaluated checks and an incomplete run inventory.
08a2587 Results. Raw results under results/0233773…/ (gzipped, 1.9 MB), summary.md/.json, REPORT.md, and README and CHANGELOG entries placed next to the primary's.

No file in src/, CI, the qualification scripts, or experiments/usefulness/ changed.

Result

The decision rule fixed in advance gives supports further development (as runtime plumbing), the same decision as the primary experiment. All 18 gates pass.

Daemon output equals the direct engine exactly. 10,000 ticks × 5 separate processes match the baseline, and every repetition is identical.
Expected-behavior checks hold. The mask equivalence and sensitivity, drive, selectivity, quiet-period, and accounting checks pass. The simulation control emits 0 spikes.
In-process speed. Daemon p99 is 3.6–5.3 µs per tick against the 1 ms budget, at 265k–338k ticks/s. Derived overhead versus the baseline is about 1.4–2.0 µs at p50.
Real binary over ZeroMQ at 1 kHz.
- 600/600 probes got the correct response, with no stray output.
- Probe latency was p99 1.69 ms.
- SIGTERM gave exit 0 in under 5 ms.
- Daemon RSS was about 6.6 MB.
neuromod 0.7.0 behaves like 0.6.0 here. All 25 rasters are identical, file by file, to the earlier 0.6.0 / Rust 1.98.1 runs.

Findings (details in REPORT.md):

tempo suppresses input drive. Nominal tempo (1.0) maps to norepinephrine 1.0 and cuts input drive to 10 %; driven firing fell from 10/10 to 2/10 ticks.
Trained-model usefulness is still unproven. The fixture is synthetic, and the published sidecar carries output_weights, which live startup rejects.
No end-to-end tracing or acknowledgment. Outputs carry no input correlation, and PUB sends are not acknowledged.

Verification

Run on Rust 1.99.0, Debian 13 VM, 8 vCPU:

cargo fmt --check passed.
cargo clippy --locked --all-targets -- -D warnings passed.
cargo test --locked passed: 160 tests.
cargo clippy --locked --all-targets --all-features -- -D warnings passed. This covers the primary and secondary examples.
cargo test --locked --all-features passed: 193 tests.
Static analysis of the new files:
- lizard (CCN ≤ 9, ≤ 50 lines, ≤ 4 arguments) reports no warnings;
- pylint scores the summarizer 10/10;
- bandit, pydocstyle (D212 and D213), and pycodestyle report nothing.
experiments/usefulness-secondary/run.sh ran on a clean tree at 0233773, with every step exiting 0 and identity.json status complete.
To check reproducibility, I decompressed the committed raw files and re-ran summarize.py. It reproduced summary.json; the only difference is measured_commit.

Note: Cargo.toml on main does not exclude experiments/ from the published package, so both experiments' result files currently ship in the crate. I left this alone so it doesn't collide with the in-flight package-exclusions branch (LIM-1514).

Refs #70 (resolved by #94)
Refs LIM-1457

🤖 Generated with Claude Code

^{Need help on this PR? Tag @codesmith-bot with what you need.}

blocksorg · 2026-10-17T00:46:54Z

This pull request was created from a Blocks session.

View in dashboard | View on desktop

Created as ready for review. To change this setting visit Git Settings.

codeant-ai · 2026-10-17T00:46:58Z

Skipping PR review because a bot author is detected.

If you want to trigger CodeAnt AI, comment @codeant-ai review to trigger a manual review.

tenki-reviewer · 2026-10-17T00:46:59Z

Review complete. 🟡 2 medium

💬 Inline comments (2)

🟡 Dirty-tree guard conflicts with untracked results directory — run.sh:18
🟡 Missing runs make pre-registered checks vacuously pass — summarize.py:236

🧹 Nitpicks (3) — 🟢 3 low

🟢 Quadratic probe-to-receive matching in transport summary (summarize.py:308) — For every probe send, summarize_transport rescans the entire filtered receive list to build the matching window (window = [j for j, r in enumerate(nonempty) if p["send_ns"] <= r["recv_ns"] < end]) at experiments/usefulness/summarize.py:309.
🟢 Workload load never validates valid_mask width (mod.rs:71) — Workload::load checks only each segment's values width against channels (examples/usefulness_common/mod.rs:72), never valid_mask lengths.
🟢 SECONDS-based step timing has 1-second granularity (run.sh:120) — step() computes duration as $((SECONDS - start)) (experiments/usefulness/run.sh:120, recorded at line 124), and SECONDS is integer seconds.

This PR introduces an experiment package under experiments/usefulness/ plus two Rust example drivers (examples/usefulness_inprocess.rs, examples/usefulness_transport.rs) and shared driver code in examples/usefulness_common/mod.rs, wired into Cargo.toml. The runner captures provenance in identity.json, executes baseline and daemon runs in live/simulation modes with zeroed/unmasked variants, and the summarizer evaluates the pre-registered C1-C12/P1-P2 checks against raw JSONL artifacts, emitting summary.md and per-run summaries.

Files	Change
`Cargo.toml`, `examples/*`	Add example drivers implementing workload decode, ingress feeding, trace capture, and summary arithmetic for in-process and ZeroMQ transport modes.
`experiments/usefulness/run.sh`	Add the orchestrated runner with dirty-tree guard, per-step timing, and identity/provenance capture.
`experiments/usefulness/summarize.py`	Add the stdlib-only summarizer implementing percentile stats, variant re-expansion, raster comparisons, and transport latency matching.
`experiments/usefulness/fixtures/*`	Add SNN model, workload, and manifest fixtures with SHA-256 hashes and channel-neuron mapping.
`experiments/usefulness/{PROTOCOL,REPORT}.md`, configs	Add pre-registration criteria, measured results, and run configuration files.

_{Reviewed commit: ae91461}

linear-code · 2026-10-17T00:47:00Z

LIM-1457

coderabbitai · 2026-10-17T00:47:01Z

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

Source mode: Auto
Repository: Unavailable
Branch: Unavailable
File: No configuration file from this repository applied
Commit: Unavailable
Layers: Organization UI
Review profile: ASSERTIVE
Plan: Advanced
Run ID: 88f333ba-a128-4eee-a7aa-597d7efb6fbd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}

codacy-production · 2026-10-17T00:48:05Z

Up to standards ✅

🟢 Issues 0 issues

Results:
0 new issues

View in Codacy

🟢 Metrics 383 complexity · 6 duplication

Metric Results

Complexity 383

Duplication 6

View in Codacy

_{NEW Get contextual insights on your PRs based on Codacy's metrics, along with PR and Jira context, without leaving GitHub. Enable AI reviewer}
_{TIP This summary will be updated as you push new changes.}

deepsource-io · 2026-10-17T00:48:17Z

DeepSource Code Review

We reviewed changes in 35090db...08a2587 on this pull request. Below is the summary for the review, and you can see the individual issues we found as inline review comments.

See full review on DeepSource ↗

PR Report Card

Overall Grade	Security Reliability Complexity Hygiene

Code Review Summary

Analyzer	Status	Updated (UTC)	Details
Rust		Oct 25, 2026 9:00p.m.	Review ↗
Secrets		Oct 25, 2026 9:00p.m.	Review ↗

Important

AI Review is run only on demand for your team. We're only showing results of static analysis review right now. To trigger AI Review, comment @deepsourcebot review on this thread.

deepsource-io · 2026-10-17T00:49:10Z

+        stream,
+        "GET {path} HTTP/1.1\r\nHost: localhost\r\nConnection: close\r\n\r\n"
+    )?;
+    let mut response = String::new();


Empty call to `new()`

The new() function is used to initialise an object with specific data.
If no arguments are passed, the behaviour is identical to default().

deepsource-io · 2026-10-17T00:49:10Z

+    let args = parse_args()?;
+    std::fs::create_dir_all(&args.out)?;
+    let workload = Workload::load()?;
+    let cfg = DaemonConfig::load(Path::new(args.mode.config_path()))?;


The argument(s) to `DaemonConfig::load` contain the unit-type

This issue is raised when a unit value is passed to a function call without
using the unit literal: (). If this is intentional, extract the argument into
a separate expression and replace it with (). Otherwise, consider revisiting
this function call.

deepsource-io · 2026-10-17T00:49:10Z

+    values
+}
+
+fn main() -> Result<()> {


`fn main` has a cyclomatic complexity of 45 with "very-high" risk

A function with high cyclomatic complexity can be hard to understand and
maintain. Cyclomatic complexity is a software metric that measures the number of
independent paths through a function. A higher cyclomatic complexity indicates
that the function has more decision points and is more complex.

deepsource-io · 2026-10-17T00:49:10Z

+    std::fs::create_dir_all(&args.out)?;
+    let workload = Workload::load()?;
+    let spec = workload.transport.clone();
+    let cfg = DaemonConfig::load(Path::new(CONFIG))?;


The argument(s) to `DaemonConfig::load` contain the unit-type

This issue is raised when a unit value is passed to a function call without
using the unit literal: (). If this is intentional, extract the argument into
a separate expression and replace it with (). Otherwise, consider revisiting
this function call.

tenki-reviewer

Expand to see previous review

Adds a pre-registered usefulness experiment harness: Rust in-process/transport example drivers, a Python summarizer, a shell runner, fixtures, and protocol/report docs.

Key findings

🟡 Dirty-tree guard conflicts with untracked results directory — run.sh:18
🟡 Missing runs make pre-registered checks vacuously pass — summarize.py:236

tenki-reviewer · 2026-10-17T04:13:58Z

Review complete. 🟡 1 medium

Reviewed the latest changes; 2 prior findings remain open.

💬 Inline comments (1)

🟡 Bound the http_get body read and total request time — mod.rs:254

The delta reworks the usefulness-experiment harness and drivers rather than the daemon core. Validation is now fail-closed: Segment::check_width enforces both values and valid_mask widths, and the summarizer gains an inventory gate (I1) plus "not evaluated" failures so missing run families can no longer pass vacuously. Timing moves from integer SECONDS to millisecond date +%s%N deltas, and transport probe matching uses bisect bucketing instead of pairwise scans. The dirty-tree guard now excludes experiments/usefulness/results, and both drivers validate frames before encoding.

Files	Change
`examples/usefulness_common/mod.rs`, `examples/usefulness_inprocess.rs`, `examples/usefulness_transport.rs`	Fail-closed workload/mask validation, frame pre-encode checks, and transport driver lifecycle hardening.
`experiments/usefulness/run.sh`, `experiments/usefulness/summarize.py`	Millisecond step timing, results-aware dirty-tree guard, inventory gate I1, and linear bisect-based transport matching.
`experiments/usefulness/REPORT.md`, `results/3fe3f42e…/`	Updated protocol report plus committed logs, summaries, and raw artifacts for the recorded run.

_{Reviewed commit: 67cd297}

deepsource-io · 2026-10-17T04:15:30Z

+
+fn start_session(args: &Args, spec: TransportSpec, channels: usize) -> Result<(Session, u64)> {
+    let config_path = Path::new(CONFIG);
+    let cfg = DaemonConfig::load(config_path)?;


The argument(s) to `DaemonConfig::load` contain the unit-type

This issue is raised when a unit value is passed to a function call without
using the unit literal: (). If this is intentional, extract the argument into
a separate expression and replace it with (). Otherwise, consider revisiting
this function call.

tenki-reviewer

Expand to see previous review

This delta reworks the usefulness experiment harness with fail-closed validation, ms step timing, an inventory gate, and bisect-based transport matching, leaving one medium finding on the unbounded http_get body read.

Key findings

🟡 Bound the http_get body read and total request time — mod.rs:254

tenki-reviewer · 2026-10-17T04:17:50Z

+    stream.read_to_string(&mut response)?;
+    Ok(response)


🟡 bug · medium

Bound the http_get body read and total request time

http_get reads the response with unbounded read_to_string and only a per-read timeout (examples/usefulness_common/mod.rs:254); the 500ms set_read_timeout applies per syscall, not to the whole transfer. A control surface that trickles bytes or an unbounded /metrics producer keeps the read alive indefinitely while the String grows without limit. This stalls wait_ready polling, evidence capture, and SIGTERM shutdown of the daemon, and can balloon driver memory.

📋 Prompt for AI Agents

In examples/usefulness_common/mod.rs around lines 242-256, replace the unbounded stream.read_to_string(&mut response) with a size-limited read (e.g. stream.take(MAX_BODY_BYTES).read_to_end into a Vec<u8>, then validate UTF-8) and add an overall request deadline so a trickling or oversized control-surface response cannot stall the driver or grow memory without bound.

tenki-reviewer · 2026-10-17T04:20:39Z

Review complete. No new findings in these changes.

Reviewed the latest changes; 3 prior findings remain open.

This incremental delta applies minor cleanup to the usefulness experiment tooling and refreshes the reported results, with no new defects verified across any lens or domain scan.

Files	Change
`examples/usefulness_transport.rs`	Extracts a `config_path()` helper for the static config path; behavior unchanged.
`experiments/usefulness/summarize.py`	Lint-style refactor (naming/imports/constants); no behavioral drift found.
`experiments/usefulness/REPORT.md`	Regenerated summary document reflecting the refactor and current results.

_{Reviewed commit: d3d1a26}

Independent replication of the #70 experiment, kept alongside the primary experiments/usefulness/ (#94): protocol with the I1 run-inventory gate, synthetic versioned LIF fixture and manifest, deterministic workload, and mode configs under experiments/usefulness-secondary/, committed before any measurement. Refs #70.

In-process daemon/baseline and ZeroMQ transport drivers as corpus-ipc-gated examples (usefulness_secondary_*) using public APIs only, a locked runner with provenance capture, and a stdlib summarizer that fails unevaluated checks and an incomplete run inventory. Refs #70.

Raw results for measured commit 0233773 (parent of this commit) on Rust 1.99.0 / neuromod 0.7.0, summary, and REPORT.md. Link the secondary experiment from README and CHANGELOG. Refs #70.

tenki-reviewer · 2026-10-25T21:00:20Z

Review complete. No blocking issues — approved ✅; 3 nitpicks below.

Reviewed the latest changes; 5 prior findings resolved.

🧹 Nitpicks (3) — 🟢 3 low

🟢 Receiver thread leaks on early error return (usefulness_secondary_transport.rs:439) — measure spawns the receiver thread at examples/usefulness_secondary_transport.rs:442 and only sets shared.stop and joins it at lines 453-454.
🟢 Committed identity.json leaks absolute local path (identity.json:8) — The committed identity.json records active_toolchain with an absolute workstation path (experiments/usefulness-secondary/results/023377370f393121f23aa24355a038737ce5a163/identity.json:8), and run.sh's write_identity emits the same machine-specific value on future runs.
🟢 Transport driver skips channels consistency check (usefulness_secondary_transport.rs:391) — start_session in examples/usefulness_secondary_transport.rs:391 validates only tick_rate_hz against the workload, while the sibling in-process driver examples/usefulness_secondary_inprocess.rs:412 also bails when cfg.channels != workload.channels.

The delta ports the usefulness experiment harness to a secondary variant under experiments/usefulness-secondary/ with new example drivers in examples/, plus Cargo.toml, README.md, and CHANGELOG.md updates. The new summarizer emits pre-registered checks fail-closed, the runner excludes results from the dirty-tree guard and times steps in nanoseconds, probe matching uses bisect, and the workload loader validates mask widths — resolving the prior review's findings. Remaining issues are minor: an error-path thread leak and a missing channels check in the transport driver, an unbounded http_get read carried over from the prior version, and an absolute local path committed inside the recorded identity.json.

Files	Change
`examples/usefulness_secondary_*.rs`, `examples/usefulness_secondary_common/mod.rs`	New in-process and ZeroMQ transport experiment drivers plus shared workload/HTTP helpers, with mask-width validation and fixed timeouts.
`experiments/usefulness-secondary/run.sh`, `summarize.py`	New pre-registered experiment runner and summarizer with fail-closed checks, nanosecond step timing, and bisect-based probe matching.
`experiments/usefulness-secondary/config/`, `fixtures/`	Experiment TOML configs and SNN/workload fixture files.
`experiments/usefulness-secondary/results/...`	Committed run artifacts (logs, summaries, identity) backing the `REPORT.md` conclusions.
`PROTOCOL.md`, `REPORT.md`, `CHANGELOG.md`, `README.md`, `Cargo.toml`	Protocol/report documentation, changelog entry, and metadata updates.

_{Reviewed commit: 08a2587}

codescene-access

Gates Passed
6 Quality Gates Passed

See analysis details in CodeScene

Quality Gate Profile: Pay Down Tech Debt
Install CodeScene MCP: safeguard and uplift AI-generated code. Catch issues early with our IDE extension and CLI tool.

blocksorg Bot assigned rmems Oct 17, 2026

blocksorg Bot added documentation Improvements or additions to documentation testing labels Oct 17, 2026

blocksorg Bot mentioned this pull request Oct 17, 2026

experiment: validate brainstem-daemon usefulness with a reproducible workload #70

Closed

6 tasks

This comment was marked as outdated.

Sign in to view

deepsource-io Bot reviewed Oct 17, 2026

View reviewed changes

tenki-reviewer Bot reviewed Oct 17, 2026

View reviewed changes

Comment thread experiments/usefulness/run.sh Outdated

Comment thread experiments/usefulness/summarize.py Outdated

rmems added this to the 02 — v0.3.0 release qualification milestone Oct 17, 2026

blocksorg Bot force-pushed the feature/LIM-1457-usefulness-experiment branch from ae91461 to 67cd297 Compare October 17, 2026 04:13

This comment was marked as outdated.

Sign in to view

deepsource-io Bot reviewed Oct 17, 2026

View reviewed changes

tenki-reviewer Bot reviewed Oct 17, 2026

View reviewed changes

This comment was marked as outdated.

Sign in to view

rmems force-pushed the feature/LIM-1457-usefulness-experiment branch from d3d1a26 to 8bd498b Compare October 22, 2026 02:05

This comment was marked as outdated.

Sign in to view

rmems added this to Rust stack Oct 23, 2026

rmems moved this to In review in Rust stack Oct 23, 2026

blocksorg Bot added 3 commits October 25, 2026 20:56

LIM-1457: Record secondary usefulness results and report

08a2587

Raw results for measured commit 0233773 (parent of this commit) on Rust 1.99.0 / neuromod 0.7.0, summary, and REPORT.md. Link the secondary experiment from README and CHANGELOG. Refs #70.

blocksorg Bot force-pushed the feature/LIM-1457-usefulness-experiment branch from 8bd498b to 08a2587 Compare October 25, 2026 20:59

blocksorg Bot changed the title ~~LIM-1457: Reproducible usefulness experiment with recorded results~~ LIM-1457: Secondary usefulness experiment (independent replication of #70) Oct 25, 2026

codescene-access Bot approved these changes Oct 25, 2026

View reviewed changes

tenki-reviewer Bot approved these changes Oct 25, 2026

View reviewed changes

rmems linked an issue Oct 25, 2026 that may be closed by this pull request

experiment: validate brainstem-daemon usefulness with a reproducible workload #70

Closed

6 tasks

rmems merged commit d979e27 into main Oct 25, 2026
14 checks passed

rmems deleted the feature/LIM-1457-usefulness-experiment branch October 26, 2026 08:02

Conversation

blocksorg Bot commented Oct 17, 2026 • edited by blacksmith-sh Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Result

Verification

Uh oh!

blocksorg Bot commented Oct 17, 2026

Uh oh!

codeant-ai Bot commented Oct 17, 2026

Uh oh!

tenki-reviewer Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💬 Inline comments (2)

Uh oh!

linear-code Bot commented Oct 17, 2026

Uh oh!

coderabbitai Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

This comment was marked as outdated.

Uh oh!

codacy-production Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Up to standards ✅

Uh oh!

deepsource-io Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DeepSource Code Review

PR Report Card

Code Review Summary

Uh oh!

deepsource-io Bot Oct 17, 2026

Choose a reason for hiding this comment

Empty call to `new()`

Uh oh!

deepsource-io Bot Oct 17, 2026

Choose a reason for hiding this comment

The argument(s) to `DaemonConfig::load` contain the unit-type

Uh oh!

deepsource-io Bot Oct 17, 2026

Choose a reason for hiding this comment

`fn main` has a cyclomatic complexity of 45 with "very-high" risk

Uh oh!

deepsource-io Bot Oct 17, 2026

Choose a reason for hiding this comment

The argument(s) to `DaemonConfig::load` contain the unit-type

Uh oh!

tenki-reviewer Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

tenki-reviewer Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

💬 Inline comments (1)

Uh oh!

This comment was marked as outdated.

Uh oh!

deepsource-io Bot Oct 17, 2026

Choose a reason for hiding this comment

The argument(s) to `DaemonConfig::load` contain the unit-type

Uh oh!

tenki-reviewer Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tenki-reviewer Bot Oct 17, 2026

Choose a reason for hiding this comment

Uh oh!

tenki-reviewer Bot commented Oct 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

Uh oh!

tenki-reviewer Bot commented Oct 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

blocksorg Bot commented Oct 17, 2026 •

edited by blacksmith-sh Bot

Loading

tenki-reviewer Bot commented Oct 17, 2026 •

edited

Loading

coderabbitai Bot commented Oct 17, 2026 •

edited

Loading

codacy-production Bot commented Oct 17, 2026 •

edited

Loading

deepsource-io Bot commented Oct 17, 2026 •

edited

Loading

tenki-reviewer Bot left a comment •

edited

Loading

tenki-reviewer Bot commented Oct 17, 2026 •

edited

Loading

tenki-reviewer Bot left a comment •

edited

Loading

tenki-reviewer Bot commented Oct 17, 2026 •

edited

Loading

tenki-reviewer Bot commented Oct 25, 2026 •

edited

Loading