fix(bin): bound concurrent ShellCheck processes across the whole machine

[matthewstrud]

Problem

An 8-core, 15 GiB host hung under a ShellCheck storm while several checkouts of this repo ran their suites at once.
The cause is multiplication, not any one number: bin/fm-test-run.sh runs up to 8 test files, tests/fm-lint.test.sh's completeness sweep backgrounds 4 roots at a time, and bin/fm-lint.sh runs 2 workers.
Nothing at any layer knows how many other checkouts are doing the same on the same host.
Measured here: one ShellCheck of bin/fm-spawn.sh peaks at 1.06 GiB RSS, so the product exhausts memory well before it exhausts patience.

Change

bin/fm-lint-slot.sh is the single owner of a whole-machine bound: it runs a command only while holding one of N per-user, per-host slots.
Both ShellCheck launch sites (the fm-lint.sh worker and the lint test's sweep) now launch through it.
No layer's own parallelism number changes.

Coordination available between independent processes in separate worktrees, started at different times, under one user: the filesystem and the kernel.
A slot is a non-blocking flock(2) on /tmp/fm-lint-slots.<uid>/slot.<i>, taken via perl (already a lint requirement; flock(1) is absent on macOS).
The helper then execs the command with the locked descriptor inherited, so the command keeps the helper's pid (existing signal handling is unchanged) and the kernel frees the slot the instant that process dies, including on SIGKILL.
There is no lock file to go stale, no daemon, and no cleanup.

Default slots: half the online CPUs, at least 4 (one checkout's own peak, so a lone run never waits), capped at half the memory in GiB, never below 2.
FM_LINT_SLOTS overrides; 0 disables.

Degradation, all toward running rather than hanging

• A slot holder is always the exec'd command and never waits for a slot; a nested call runs directly. Deadlock is structurally impossible.
• Coordination unavailable (no perl; slot directory missing and uncreatable, a symlink, not a directory, or not owned by this user; slot file unopenable): the command runs immediately, unbounded, with one loud fm-lint-slot: ... running without the machine-wide bound line.
• No slot frees within FM_LINT_SLOT_WAIT_SECS (default 600): the command runs anyway with the same loud line.
• Known limit, stated in the header: the bound covers one user on one host. Another user, or a sandbox with a private /tmp, holds separate slots.

Measurements

Simulated worker = one full bash tests/fm-lint.test.sh in its own clone (real ShellCheck 0.11.0, includes the 408-root sweep and full-analysis partitions). 8 cores, 15 GiB. Sampled every 0.5 s: pgrep -x shellcheck count, summed ShellCheck RSS, /proc/loadavg. Ambient load1 on the host was 3-5 throughout from unrelated processes, so load figures carry that floor. The harness killed a run if load1 passed 11.

run	workers	slots	peak concurrent shellcheck	peak shellcheck RSS	peak load1	wall	result
alone, unbounded	1	off	4	1.9 GiB	4.3	293 s	pass
alone, default	1	4	4	1.9 GiB	7.0	297 s	pass
two, unbounded	2	off	8	3.7 GiB	6.4	312 s	pass
two, bounded	2	4	4	2.9 GiB	8.1	344 s	pass
three, bounded (first default tried)	3	7	7	2.7 GiB	11.1	-	killed by the load guard at 260 s
three, bounded (shipped default)	3	4	4	3.2 GiB	9.2	402 s	pass, all three

Reading:

• Unbounded, concurrent ShellCheck scales with worker count (4 -> 8). Bounded, it stays at 4 for one, two, and three workers.
• A lone worker is not slower: 297 s vs 293 s, inside run-to-run noise (a repeat lone bounded run took 299 s), because its own peak equals the bound and it never waits.
• The first default I tried (min of CPUs and memory/2 = 7) held its bound but still let three workers push load past the core count, so I tightened it to half the CPUs. That failed run is in the table on purpose.
• I did not run three or four workers unbounded. The 7-slot run already tripped the safety guard, and reproducing the hang was not worth the evidence.

What this does not bound

In the three-worker runs load1 reached ~10 while only 3 ShellChecks were alive; the rest was the test scripts themselves plus ambient load.
This PR bounds lint, which is what the card scopes and what holds the memory.
bin/fm-test-run.sh's up-to-8 test files per checkout is still multiplied by the number of checkouts, and the same slot helper pattern would apply there. That is a follow-up, not part of this change.

Verification

• tests/fm-lint.test.sh gains seven behavior tests: the total is bounded across independently started processes (and both slots are actually used), status and output pass through unchanged, a SIGKILLed holder frees its slot immediately, a full slot set runs loudly after the bounded wait, unavailable coordination (non-directory, symlink) runs loudly and keeps the command's stderr byte-identical, a holder never waits for a second slot, and fm-lint.sh really launches ShellCheck through the bound.
• bin/fm-test-run.sh --changed (39 scripts): 37 pass. tests/fm-calm-pi-extension.test.sh fails because this host has no Chrome. tests/fm-test-run.test.sh fails identically on the unmodified base commit (same 24 passing assertions, same comm: input is not in sorted order), so it predates this change.
• bin/fm-lint.sh and bin/fm-doc-audience-check.sh pass.

[Copilot]

Copilot review overview

🔵 Needs a closer look

bin/fm-lint-slot.sh can inadvertently close stdin/stdout when FM_LINT_SLOT_DIAG_FD is set to 0/1, and the PR description doesn’t currently call out the additional Beads captain-hold behavior change.

Review effort: Lite
Findings: None

What changed in this PR

Captain, this PR introduces a whole-machine concurrency bound for ShellCheck by routing lint invocations through a new slot-based locking helper, reducing the risk of host OOM/hangs when multiple checkouts run simultaneously. It also includes an additional (separate) behavior change around creating captain-hold rows on Beads backends with due.required.

Changes:

• Add bin/fm-lint-slot.sh to enforce a per-user, per-host global slot limit using flock(2) via Perl, with bounded wait and safe-degradation behavior.
• Route ShellCheck launches in bin/fm-lint.sh and the lint test’s per-root sweep through the new slot helper, and add behavioral tests for the bound.
• Update captain-hold creation to waive Beads due.required for newly created captain rows, with new tests and documentation.

File	Description
tests/​fm-lint.test.sh	Routes per-root ShellCheck through the slot helper and adds end-to-end behavior tests for slot bounding and degradation.
tests/​fm-captain-hold-lifecycle.test.sh	Adds a Beads fixture test ensuring hold can create a captain row when due.required is enforced without a custom captain type.
docs/​fm-test-portable-shards.md	Documents that fm-lint.sh’s per-invocation worker limit is distinct from the new whole-machine bound.
docs/​configuration.md	Documents captain-hold row creation semantics for Beads (native type mapping, due waiver).
bin/​fm-test-run.sh	Ensures changes to bin/fm-lint-slot.sh trigger the appropriate test family selection.
bin/​fm-lint.sh	Wraps ShellCheck worker invocations with the slot helper while preserving prior worker lifecycle/signaling behavior.
bin/​fm-lint-slot.sh	New implementation of the machine-wide slot bound with flock-based coordination and bounded-wait fallback.
bin/​fm-captain-hold.sh	Waives Beads due.required for tasks-axi add when creating a missing captain-hold row.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.