Skip to content

test(prompt-invariants): split random interleavings into 10 tests with per-test fake DOM (lr-248bff) - #484

Merged
clagentic-merger[bot] merged 1 commit into
mainfrom
chore/lr-248bff-split-random-interleavings
Oct 23, 2026
Merged

clagentic-merger[bot] merged 1 commit into
mainfrom
chore/lr-248bff-split-random-interleavings

Conversation

@clagentic-builder

Copy link
Copy Markdown
Contributor

What

Test-only. test/prompt-invariants.test.js: the single 150-seed random-interleavings test is split into 10 tests of 15 seeds each (seed values 1..150 unchanged, same mulberry32 PRNG, STEPS 40 unchanged). No product code touched.

Why

The one test sat at or over its 300 s timeout in a full local run, failing by wall time, and its still-running loop then failed the next test.

Changes

  • 10 tests, seeds 1 to 15 ... seeds 136 to 150, each with a 120 s timeout (measured max 68 s under full npm test, 35 s alone).
  • Tally: one shared tally, summed across the parts; a describe after hook asserts every ending kind > 0 and dead == 0 over all 150 seeds, once all 10 parts completed. This is the same assertion over the same union of runs as before, so it is equivalent (a per-slice assertion would be stricter than the original and could fail on a rare ending such as restart). If a part failed or timed out it is already reported and the partial tally is not judged.
  • Isolation: a fresh fake DOM and store (freshDom) per random part and per scenario test, replacing the file-level before. randomRun now stops on t.signal.aborted (node:test aborts it on timeout), so a timed-out part stops mutating process env, the module cache and the DOM instead of running on under the next test; that leak, not the DOM object alone, is what failed the sub-agent test.
  • Per-step cost: no obvious waste found (flush is one setImmediate; most cost is purgeLib re-requiring lib per seed and seed-dependent chatter/paging in product code driven by the test); left unchanged.

Evidence

Forced-timeout check (part timeout temporarily 2 s, then reverted): all 10 parts fail by timeout, every later test including a sub-agentsrequestsurvivesitsparents turn result passes. That test also passes alone (288 ms, --test-name-pattern).

File alone (10 parts): 3.2, 12.6, 30.6, 33.1, 35.4, 30.3, 31.3, 28.7, 29.4, 34.9 s (suite 269.6 s, spec has no single test over 36 s).

Three consecutive full npm test runs on this host (part times in s, parts 1..10):

  • Run 1: 7.3, 18.8, 45.5, 35.3, 37.9, 30.2, 29.6, 33.8, 30.9, 27.2. Failing set: web-UI shutdown_server reaches the announcer (ENOTEMPTY, lr-c3e80a flake). prompt-invariants absent.
  • Run 2: 5.9, 21.1, 46.0, 34.9, 43.0, 68.2, 64.8, 62.0, 53.5, 42.7. Failing set: web-UI shutdown_server reaches the announcer (ENOTEMPTY), codex: a loop iteration is recorded as the loop (ENOTEMPTY) (both lr-c3e80a), a gatekeeper session still pending at the timeout is cancelled... (identity dir, lr-7da984 late-settle). prompt-invariants absent.
  • Run 3: 6.0, 25.0, 52.8, 32.0, 36.4, 32.8, 33.9, 32.5, 32.2, 44.3. Failing set: empty (3590 pass, 0 fail).

Profile has no test block (clagentic-loadout test run exits 72), so npm test is the admitted command; no loadout run ids.

TASK: lr-248bff
class sweep: other 300 s+ single loops in test/ not examined (out of scope, test-only fix for this file).

🤖 Generated with Claude Code

…h per-test fake DOM (lr-248bff)

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@clagentic-reviewer

Copy link
Copy Markdown

PEACHES — clean (0 finding(s), 1 dropped)
verdict: CLEAN
range: 95b494a..46a361f
engine: carrier: gpt-5.6-sol

Dropped candidates (1):

  • test/prompt-invariants.test.js:773 [amos.code-craft.5] The existing invariant test is rewritten into timeout slices specifically to make it pass, which the rule prohibits. (reason: No failure mode named: seeds, steps, PRNG and the cross-seed tally are unchanged per the HOLDEN ruling, and the candidate shows no weakened assertion, so the Pre-Report Gate fails and the settled split shape is not re-argued.)
{"reviewer": "peaches", "review_status": "clean", "head_sha": "46a361f8d23275c4fad5783ff77336863235c213", "pr_number": 484, "dropped": [{"file": "test/prompt-invariants.test.js", "line": 773, "rule_id": "amos.code-craft.5", "message": "The existing invariant test is rewritten into timeout slices specifically to make it pass, which the rule prohibits.", "reason": "No failure mode named: seeds, steps, PRNG and the cross-seed tally are unchanged per the HOLDEN ruling, and the candidate shows no weakened assertion, so the Pre-Report Gate fails and the settled split shape is not re-argued."}], "range": {"basis": "base..head", "base": "95b494a8024f8e055e6546f9d88b684f1e0678ea", "head": "46a361f8d23275c4fad5783ff77336863235c213"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "95b494a8024f8e055e6546f9d88b684f1e0678ea", "evaluated_findings": [], "fence_schema_version": 3}

@clagentic-security

Copy link
Copy Markdown

BOBBIE — clean (0 finding(s))
verdict: CLEAN
range: 95b494a..46a361f
engine: carrier: gpt-5.6-sol

{"reviewer": "bobbie", "review_status": "clean", "head_sha": "46a361f8d23275c4fad5783ff77336863235c213", "pr_number": 484, "scanners_run": [{"scanner": "gitleaks", "status": "ran"}, {"scanner": "trufflehog", "status": "ran"}, {"scanner": "semgrep", "status": "ran"}, {"scanner": "osv-scanner", "status": "not_applicable", "reason": "no packages found to scan"}], "fence_schema_version": 3, "range": {"basis": "base..head", "base": "95b494a8024f8e055e6546f9d88b684f1e0678ea", "head": "46a361f8d23275c4fad5783ff77336863235c213"}, "engines": [{"engine": "carrier", "model": "gpt-5.6-sol", "chunks": 1}], "verdict_outcome": "CLEAN", "verdict_reasons": [], "evaluator_version": "2", "policy_base_sha": "95b494a8024f8e055e6546f9d88b684f1e0678ea", "evaluated_findings": []}

@clagentic-merger
clagentic-merger Bot merged commit e8e9833 into main Oct 23, 2026
6 checks passed
@clagentic-merger

Copy link
Copy Markdown
Contributor

Merged via clagentic-loadout v0.2.0

Field Value
Gated HEAD SHA 46a361f8d23275c4fad5783ff77336863235c213
Merged SHA 46a361f8d23275c4fad5783ff77336863235c213
Reviews clagentic-reviewer[bot], clagentic-security[bot]
Verdicts peaches: fence CLEAN, gate CLEAN
bobbie: fence CLEAN, gate CLEAN
CI status combined_state='pending' (ignored: 0 commit statuses, so the check-runs decide) (0 status(es), 6 run(s); source=check-runs)
task_id lr-248bff

@clagentic-merger
clagentic-merger Bot deleted the chore/lr-248bff-split-random-interleavings branch October 23, 2026 16:41
@clagentic-release-bot

Copy link
Copy Markdown
Contributor

This issue has been resolved in version 1.11.0-beta.2.

To update, run:

npx @clagentic/console@1.11.0-beta.2

@clagentic-release-bot

Copy link
Copy Markdown
Contributor

This issue has been resolved in version 1.11.0.

To update, run:

npx @clagentic/console@1.11.0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants