Skip to content

fix(bin): move Bearings candidate-PR rows off jq argv onto file transport - #4146

Closed
karotkriss wants to merge 2 commits into
kunchenguid:mainfrom
karotkriss:fm/fm-bearings-jq-argv
Closed

karotkriss wants to merge 2 commits into
kunchenguid:mainfrom
karotkriss:fm/fm-bearings-jq-argv

Conversation

@karotkriss

@karotkriss karotkriss commented Sep 10, 2026 •

Copy link
Copy Markdown
Contributor

Intent

Captain, 2026-09-10: the effort-impact triage picked three upstream issues and the captain ruled "pick 3 issues and get them done in the firstmate repo": ship this one as a PR to kunchenguid/firstmate. The hard rule from that triage: touch NO hot file (bin/fm-spawn.sh, bin/fm-test-run.sh, bin/fm-teardown.sh, bin/fm-watch.sh, bin/fm-bootstrap.sh, bin/fm-brief.sh, bin/fm-wake-lib.sh, tests/fm-brief.test.sh, tests/fm-teardown.test.sh, docs/scripts.md, docs/configuration.md, AGENTS.md, docs/architecture.md), add no new script and change no documented interface, so the PR cannot deadlock on rebases the way #2839 did. The upstream repo requires the no-mistakes attestation on every PR, and its "Behavior portable parallel" shards time out at their 10-minute cap until #4123 merges; if that shard is the only red check, close the gate on the evidence that its tests passed and say so in the PR. PR body: "Fixes #" lines, the reproduction, and the test evidence. No co-author. This pick: upstream issue #3056. bin/fm-bearings-snapshot.sh still passes unbounded JSON through jq argv: rows=$(jq -n --argjson a "$rows" --argjson b "$repo_rows" '$a + $b') accumulates PR rows across repos (around line 282), and the final projection passes the accumulated PR status as --arg prs "$PR_STATUS" (around line 310). Each jq argv argument is capped by the kernel's 128 KiB MAX_ARG_STRLEN, so large fleets kill the board. The sibling bin/fm-fleet-snapshot.sh fixed the same class with a temp-file transport dir and --slurpfile (around lines 1934-1968, upstream PR 3677).

What Changed

  • bin/fm-bearings-snapshot.sh now accumulates fetched candidate-PR rows by appending each repo's JSON array to a mktemp transport file (cleaned up on EXIT) and feeds the final projection with --slurpfile candidate_pr_batches, replacing the --argjson/jq -n '$a + $b' accumulation and the --arg-style pass of the full PR set; the projection reconstitutes $candidate_prs via $candidate_pr_batches | add // [], so large fleets no longer breach the 128 KiB MAX_ARG_STRLEN per-argument cap that killed the board.
  • Added tests/fm-bearings-snapshot.test.sh coverage: a FAKE_GH_HUGE fixture emitting 500 padded PRs per repo and test_pr_rows_above_arg_limit_still_project, asserting a 2-repo/1000-PR set projects completely with a single repo's rows exceeding 131072 bytes.
  • Bumped the CI Bearings snapshot test-count guard in .github/workflows/ci.yml from 59 to 60 to match the new test.

Risk Assessment

✅ Low: Small, well-bounded fix that moves the only unbounded jq argv value (accumulated candidate-PR rows) onto a temp-file/--slurpfile transport mirroring the established sibling pattern, with correct empty/multiple/empty-batch semantics verified and a real behavior-level regression test.

Testing

I derived the intent (accumulated PR rows must survive Linux's 128 KiB per-argv-argument cap by traveling through a temp file and --slurpfile instead of jq --argjson) and drove it against the real fm-bearings-snapshot.sh via its behavior test harness with a fake gh returning 500 padded PRs across 2 repos (1000 rows, one repo's slice >131072 bytes). The fix projects the full set (candidate_prs length 1000, "checked (2 repos, 1000 open)"); the pre-fix base code reproduces the exact reported crash, proving the regression. The adjacent transport paths (no-PR local-only default, normal enrichment, partial-failure degradation, and multi-repo cap accumulation) were also driven live and pass. I did not run the full 60-test suite to completion (it is dominated by several deliberate 30s sleep/timeout tests and exceeds the window with zero failures observed); the CI count-guard bump 59→60 in .github/workflows/ci.yml is a CI-only artifact validated by CI's own runner, not a live product surface.

  • Live validation: ✅ go - 5 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Fleet-sized PR set (1000 rows, one repo slice >128 KiB) projects completely instead of killing the board ✅ pass live test_pr_rows_above_arg_limit_still_project against target e93fc71: candidate_prs length 1000, checked (2 repos, 1000 open), repo slice >131072 bytes — jq-argv-regression.txt (AFTER)
Reported failure reproduces on pre-fix code (jq argv exceeds MAX_ARG_STRLEN) ✅ pass live Same regression driven against daaffdb:bin/fm-bearings-snapshot.sh: jq: Argument list too long then invalid JSON text passed to --argjson then projection failed — jq-argv-regression.txt (BEFOR…
No-PR local-only default still projects (empty --slurpfile transport → add // []) ✅ pass live test_default_is_bounded_and_local_only — pass
Normal --include-prs enrichment still correct after transport change ✅ pass live test_include_prs_is_the_only_fetch_path and test_partial_github_failure_degrades — pass
Rows accumulated across multiple repos via file transport respect caps/expansion ✅ pass live test_section_caps_and_expansion_flags, test_per_repository_pr_cap_is_disclosed, test_pr_repository_cap_and_expansion — pass
CI Bearings test-count guard equals actual emitted ok-count (60) ⏸️ untested no The full suite includes several deliberate 30s sleep/timeout tests and exceeds the local window; this is a CI-workflow count guard, not a live product surface, and remote CI owns broad-suite regressio…
Evidence: jq-argv regression: fail-before-fix vs pass-after-fix transcript

Source: jq-argv regression: fail-before-fix vs pass-after-fix transcript

## BEFORE fix (base daaffdb): line 319: jq: Argument list too long / jq: invalid JSON text passed to --argjson / fm-bearings-snapshot: projection failed / not ok ## AFTER fix (e93fc71, --slurpfile temp-file transport): ok - candidate-PR rows above the per-argument limit still project completely

# fm-bearings jq-argv regression: 1000 candidate PR rows (>128 KiB) must project

## BEFORE fix (base daaffdb bin/fm-bearings-snapshot.sh) -- reproduces the reported failure
    ~/.no-mistakes/worktrees/80aee654c94f/01M2V1TVWKX2J6A6MH3FJ2F05V/bin/fm-bearings-snapshot.sh: line 319: ~/.local/bin/jq: Argument list too long
    ~/.no-mistakes/worktrees/80aee654c94f/01M2V1TVWKX2J6A6MH3FJ2F05V/bin/fm-bearings-snapshot.sh: line 319: ~/.local/bin/jq: Argument list too long
    jq: invalid JSON text passed to --argjson
    Use jq --help for help with command-line options,
    or see the jq manpage, or online docs  at https://jqlang.github.io/jq
    fm-bearings-snapshot: projection failed
    not ok - projection failed on PR rows above the per-argument limit

## AFTER fix (target e93fc71, --slurpfile temp-file transport)
    ok - candidate-PR rows above the per-argument limit still project completely

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

🔧 **Rebase** - 1 issue found → auto-fixed ✅
  • ⚠️ bin/fm-bearings-snapshot.sh - merge conflict rebasing onto origin/main

🔧 Fix applied.
✅ Re-checked - no issues remain.

✅ **Review** - passed

✅ No issues found.

✅ **Test** - passed

✅ No issues found.

  • Live validation: ✅ go - 5 of 6 scenarios driven live against the product
Scenario Result Live Evidence
Fleet-sized PR set (1000 rows, one repo slice >128 KiB) projects completely instead of killing the board ✅ pass live test_pr_rows_above_arg_limit_still_project against target e93fc71: candidate_prs length 1000, checked (2 repos, 1000 open), repo slice >131072 bytes — jq-argv-regression.txt (AFTER)
Reported failure reproduces on pre-fix code (jq argv exceeds MAX_ARG_STRLEN) ✅ pass live Same regression driven against daaffdb:bin/fm-bearings-snapshot.sh: jq: Argument list too long then invalid JSON text passed to --argjson then projection failed — jq-argv-regression.txt (BEFOR…
No-PR local-only default still projects (empty --slurpfile transport → add // []) ✅ pass live test_default_is_bounded_and_local_only — pass
Normal --include-prs enrichment still correct after transport change ✅ pass live test_include_prs_is_the_only_fetch_path and test_partial_github_failure_degrades — pass
Rows accumulated across multiple repos via file transport respect caps/expansion ✅ pass live test_section_caps_and_expansion_flags, test_per_repository_pr_cap_is_disclosed, test_pr_repository_cap_and_expansion — pass
CI Bearings test-count guard equals actual emitted ok-count (60) ⏸️ untested no The full suite includes several deliberate 30s sleep/timeout tests and exceeds the local window; this is a CI-workflow count guard, not a live product surface, and remote CI owns broad-suite regressio…
  • bash tests/fm-bearings-snapshot.test.sh driver running only test_pr_rows_above_arg_limit_still_project against target e93fc71 — pass
  • Same regression test against base daaffdb:bin/fm-bearings-snapshot.sh — fails with jq: Argument list too long / invalid JSON text passed to --argjson / projection failed (fail-before-fix confirmed)
  • test_default_is_bounded_and_local_only (empty PR transport / slurpfile of empty file) — pass
  • test_include_prs_is_the_only_fetch_path — pass
  • test_partial_github_failure_degrades — pass
  • test_section_caps_and_expansion_flags, test_per_repository_pr_cap_is_disclosed, test_pr_repository_cap_and_expansion (multi-repo row accumulation via file) — pass
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

@kunchenguid

Copy link
Copy Markdown
Owner

Speaking as Kun's firstmate:

First look on HEAD c8d413599b496af8bbb81e57231a400d4537a7d2 vs main b1ad702fafdd. Attestation MATCH. Fork @karotkriss. Diff reviewed (bin/fm-bearings-snapshot.sh temp-file+--slurpfile transport; test; ci.yml Bearings count 56→57 for the new test).

What changed (inspected)

Contract-class: restore — bearings board already promised to project PR rows; argv death on large fleets is a broken promised path. Count-guard bump is mechanical with the added test (not a new default product path).

VISION.md (each rule)

  • One captain, one interface: aligns — board must not die on large fleets.
  • Authority is explicit: aligns.
  • Scripts own the mechanics: aligns.
  • A restart is a non-event: aligns.
  • Delegation with a spine: aligns.
  • The fleet outlives any vendor: aligns.
  • Scope: aligns.

CI: Require no-mistakes SUCCESS (latest). Behavior portable parallel 1 CANCELLED at known 10m cap (author: suite summary failed=0 then cancel in cleanup). Stock macOS Bash snapshot compatibility SUCCESS after count bump. Not merge-eligible while parallel-1 shows fail/cancelled. No captain card.

Security: clean (workflow touch is test-count only). No Firstmate flag.

@karotkriss
karotkriss force-pushed the fm/fm-bearings-jq-argv branch from c8d4135 to 98714d5 Compare September 11, 2026 15:04
@karotkriss karotkriss closed this Sep 11, 2026
@karotkriss karotkriss reopened this Sep 11, 2026
@karotkriss
karotkriss force-pushed the fm/fm-bearings-jq-argv branch from 98714d5 to 36f5927 Compare September 11, 2026 15:42
@karotkriss karotkriss changed the title fix(bearings): move accumulated candidate-PR rows off jq argv fix(bearings): move accumulated PR rows off jq argv onto file transport Sep 11, 2026
bin/fm-bearings-snapshot.sh accumulated per-repo PR rows through
jq --argjson and passed the accumulated set into the final projection
the same way. Each jq argv argument is capped by the kernel's
MAX_ARG_STRLEN (128 KiB), so a large fleet's accumulated rows killed
the board with 'projection failed'.

Rows now travel by file transport: each fetched repo appends its rows
array to a temp file (cleaned on exit) and the final projection merges
the batches with --slurpfile, the same pattern bin/fm-fleet-snapshot.sh
uses for its snapshot transport.

The new regression builds a two-repo fixture whose first repo's rows
alone serialize past 131072 bytes and asserts the projection preserves
all 1000 rows; it fails with 'projection failed' on the prior code.

Fixes kunchenguid#3056
…" check was caused by this PR's own change. The PR added one new Bearings snapshot test (test_pr_rows_above_arg_limit_still_project, the jq-argv regression), raising the suite from 56 to 57 `ok - ` lines, but the count guard in .github/workflows/ci.yml still asserted exactly 56 (CI reported `got 57`). Fixed by updating that guard from 56 to 57 (both the `[ -eq ]` test and its error message). Confirmed the count is correct: the CI log itself showed 57, the test file contains 57 `pass` calls, and the diff adds exactly one test driver invocation. .github/workflows/ci.yml is not on the intent's forbidden hot-file list, so touching it is allowed
@karotkriss
karotkriss force-pushed the fm/fm-bearings-jq-argv branch from 36f5927 to e93fc71 Compare September 18, 2026 20:31
@karotkriss karotkriss changed the title fix(bearings): move accumulated PR rows off jq argv onto file transport fix(bin): move Bearings candidate-PR rows off jq argv onto file transport Sep 18, 2026
@karotkriss

Copy link
Copy Markdown
Contributor Author

Superseded by #5985, which re-applies this fix on current main and closes #3056; closing.

@karotkriss karotkriss closed this Sep 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants