Skip to content

test: run regenerateLibs into a fixture dir and pin what it emits - #228

Merged
thedavidmeister merged 7 commits into
mainfrom
fix-207
Sep 16, 2026
Merged

thedavidmeister merged 7 commits into
mainfrom
fix-207

Conversation

@thedavidmeister

Copy link
Copy Markdown
Contributor

Closes #207.

What was wrong

Build's two generator hooks were executed by nothing. A revert() as the
first statement of regenerateLibs left the suite at 512 passed / 0 failed on
baa1a9c; the same in regenerateSnapshots left it at 508 passed / 4 failed,
the four being public-RPC rate limits in LibRainDeployTest that are there
without any mutation. Zero delta both ways, re-verified on current main rather
than on the e9a922a the issue was filed against.

Every check on the hooks was output-anchored: it compared a committed lib
against the emitters, so it saw drift only AFTER somebody re-ran the generator
and committed what came out. A loop bound that stopped one contract short, or a
constantPrefix taken from contracts[0] on every pass, was invisible until
the next release cut it into the append-only record.

The issue's own recommended residual — extend the emitter pins to
MigrationRegistry's two committed libs — has already landed, in 4eb5363
(PR #176), which postdates the commit the issue was verified against. That half
is closed; this PR closes the half the issue's title names.

What changed

regenerateLibs takes its directory from a new internal virtual libDir()
instead of hard-coding LibRainDeploySnapshot.LIB_DIR, for exactly the reason
BuildScript.recordRoot is already overridable: a hook that can only be pointed
at the committed tree can only be RUN by overwriting files the rest of the suite
compiles and reads, and forge runs test contracts in parallel. No parameter is
added to run() or cutRelease(), and nothing outside Build can see the
hook.

BuildHarness gains the fixture directory, an externalRegenerateLibs() that
refuses to run when it is left pointing at src/lib, and an externalLibDir()
so the production default is still assertable.

regenerateSnapshots gets no such seam and can have none: LibFs.dirForTag
confines every snapshot write to src/generated/<tag>/, which is the record
frozenSnapshotPaths walks, so there is no directory to drive it into that the
suites running beside it do not read.

Migration

None. libDir() is internal with the previous constant as its default, so a
downstream Build that overrides nothing behaves exactly as before. A consumer
repo with its own BuildHarness copy must pass the constructor's new argument;
"" reproduces the old behaviour.

QA

  • Discriminating tests: BuildTest.testRegenerateLibsEmitsExactlyTheCommittedLibs,
    BuildTest.testTheDefaultLibDirIsWhereTheCommittedLibsAre,
    BuildTest.testRegenerateLibsRefusesToWriteTheCommittedLibs - each fails on
    base because on base the hook cannot be run at all: the two mutations the
    issue names were applied to main itself and the whole suite stayed green
    (revert() in regenerateLibs: 512 passed, 0 failed; i + 1 < contracts.length in the same loop: 505 passed, 7 failed, every one of the 7
    a vm.createFork rate limit, which is the same kind of failure the unmutated
    baseline has and never an assertion). The same two mutations on this branch
    are killed, named below.
  • Mutations applied (each reverted before the next; killing test named by its
    own failure message):
    • script/Build.sol:78 regenerateLibs -> revert(); as its first statement
      -> testRegenerateLibsEmitsExactlyTheCommittedLibs (EvmError: Revert)
    • script/Build.sol:81 i < contracts.length -> i + 1 < contracts.length
      -> testRegenerateLibsEmitsExactlyTheCommittedLibs (emitted a different number of files than the repo commits: 3 != 5)
    • script/Build.sol:83 contracts[i].constantPrefix ->
      contracts[0].constantPrefix ->
      testRegenerateLibsEmitsExactlyTheCommittedLibs (emitted something other than the committed LibMigrationRegistryDeploy.sol)
    • script/Build.sol:83 LibRainDeploySnapshot.CANDIDATE -> "0_1_10" ->
      testRegenerateLibsEmitsExactlyTheCommittedLibs (emitted something other than the committed LibAddressRegistryDeploy.sol)
    • script/Build.sol:72 return LibRainDeploySnapshot.LIB_DIR; -> return "fixture-lib"; -> testTheDefaultLibDirIsWhereTheCommittedLibsAre
      (fixture-lib != src/lib)
    • test/concrete/BuildHarness.sol:48 the empty-dir refusal deleted ->
      testRegenerateLibsRefusesToWriteTheCommittedLibs (next call did not revert as expected). That run also took
      testEveryCommittedAliasLibIsWhatTheGeneratorEmits down with it, reading
      src/lib/LibAddressRegistryDeploy.sol as empty while the unguarded write
      was in it - the race the refusal exists for, observed. git status src/lib
      was clean afterwards, which is the no-op oracle above holding in the one
      place it can be checked directly.
  • Oracle: the committed tree, not the emitters. A regeneration of a clean
    checkout is a no-op, so every file regenerateLibs() writes must be
    byte-identical to the file already in src/lib/, and it must write no others.
    That is independent of the emitters in the way the existing pins are not: they
    say the committed files are what the emitters produce, and this says the hook
    asks the emitters for those files. testTheDefaultLibDirIsWhereTheCommittedLibsAre
    asserts the literal src/lib rather than the constant the default returns,
    so it is not the implementation compared against itself.
  • Category check: the issue asks for an argument, a loop bound and a call
    ordering. Arguments: every argument of all three calls in regenerateLibs is
    covered, the three mutations above being constantPrefix, CANDIDATE and the
    directory; contractName and candidate.snapshot reach the file name and the
    emitted entries, and recordRoot() reaches the released lib's import block, so
    a wrong value in any of them changes the bytes compared. Loop bound: the file
    COUNT is asserted as well as the contents, because a short loop writes nothing
    wrong - it writes nothing at all - and an extra write is a generated file
    nothing regenerates. Call ordering: there is no ordering property inside
    regenerateLibs to assert, and asserting one would be theatre - writeAliasLib
    and writeReleasedSuitesAggregate read nothing, writeReleasedSuitesLib reads
    only the record, so no permutation of the three changes a byte anywhere. The
    ordering that IS observable, regenerateSnapshots before regenerateLibs, is
    already asserted by BuildScriptTest.testRunRegeneratesAndFreezesNothing
    through BuildScriptHarness's markers. regenerateSnapshots itself stays
    undrivable for the LibFs reason above, which is a boundary in a pinned
    upstream package rather than a seam this repo declined to open.

nix develop -c forge test --threads 1 on baa1a9c: 507 passed / 5 failed, all
5 being public-RPC rate limits in LibRainDeployTest and
RainDeployBroadcastTest (that suite passes 79/79 on its own when the endpoint
is not rate-limiting). On this branch: 515 passed / 0 failed, 512 + the three
new tests.

🤖 Generated with Claude Code

https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN

`Build`'s generator hooks were executed by nothing: a `revert()` as the
first statement of either one left the whole suite green. Every check on
them was output-anchored -- the committed libs compared against the
emitters -- so a wrong argument or a short loop bound was invisible until
somebody re-ran the generator and committed the result.

`regenerateLibs` now takes its directory from an overridable `libDir()`,
for the reason `BuildScript.recordRoot` is overridable, and a harness
drives it into `fixture-lib/`. A regeneration of a clean checkout is a
no-op, so the committed tree is the oracle: every file the hook writes
must be byte-identical to the one already there, and it must write no
others.

`regenerateSnapshots` gets no such seam, because `LibFs` confines every
snapshot write to `src/generated/<tag>/`, which is the record the suites
running beside it walk.

Closes #207

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN
@coderabbitai

coderabbitai Bot commented Sep 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

Next included review available in 34 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Learn how review limits work.

Review configuration:

⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Advanced

Run ID: 674d7777-8ce4-4246-8d23-aca2ab127b0c

📥 Commits

Reviewing files that changed from the base of the PR and between 2fed7dc and 17c72cb.

📒 Files selected for processing (4)
  • script/Build.sol
  • test/concrete/BuildHarness.sol
  • test/concrete/BuildRecordRootHarness.sol
  • test/script/Build.t.sol

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN
baku-ccron and others added 2 commits September 15, 2026 22:37
…nd the two new tests

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN
What comes back is every sentence the code does not state:

- `Build.libDir`: why the hook is overridable at all — a lib hook that can only
  be pointed at the committed tree can only be RUN by overwriting files the
  suites forge runs in parallel are compiling, and why `regenerateSnapshots`
  needs no equivalent.
- `BuildHarnessWouldWriteTheCommittedLibs`: what the revert means.
- `BuildHarness`: that the harness is also a seam for RUNNING `regenerateLibs`
  somewhere nothing compiles, the empty-string sentinel on `sLibDir` and its
  constructor `@param`, and the precondition that the directory already exists.
- `BuildTest`: that `regenerateLibs()` IS run, into a fixture directory, which
  is the exception to the paragraph saying nothing here runs a write.
- `testTheDefaultLibDirIsWhereTheCommittedLibsAre`: why the path is asserted as
  text rather than against `LibRainDeploySnapshot.LIB_DIR` — against the
  constant the assertion cannot fail.
- `LIBS_FIXTURE_DIR`: why the fixture root is outside `src/` and `test/`.
- `testRegenerateLibsEmitsExactlyTheCommittedLibs`: the gap it closes — every
  other assertion is output-anchored and both hooks took a `revert()` as their
  first statement with the suite still green — why the committed tree is the
  oracle, why the count is asserted, why `regenerateSnapshots` has no
  counterpart, and the read-then-remove-then-assert ordering.
- `testRegenerateLibsRefusesToWriteTheCommittedLibs`: why `setUp`'s harness is
  the default one and what a call that went through would rewrite.

Left cut: `externalLibDir`'s doc block, which restates its signature.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN
@thedavidmeister

Copy link
Copy Markdown
Contributor Author

rainix / static / static is pre-existing, not this PR

Reproduced on a clean main. Tip feefc95, untouched working tree, in the shell CI pins:

$ nix develop github:rainlanguage/rainix/8657b83b68f41957ab85da91132c3f652c1f32c0#sol-shell -c forge lint -D warnings
Error: Compiler run failed:
Warning (2018): Function state mutability can be restricted to pure
   --> test/src/lib/LibRainDeploySnapshot.t.sol:667:5
Warning (2018): Function state mutability can be restricted to pure
   --> test/src/lib/LibRainDeploySnapshot.t.sol:679:5

These are solc diagnostics, not forge-lint rules. -D is a compiler flag, so
forge lint -D warnings denies solc's own warnings and aborts at compile before
reaching a single lint rule.

Why it reads as new. main's last run (35015487248, feefc95,
2026-09-15T19:45:08Z) is green and its static job has no forge lint step at
all — it ran soldeer install, slither ., forge fmt --check,
rainix-sol-single-contract. The step arrived upstream in
rainlanguage/rainix@55c8198e ("Gate every sol consumer on forge lint and the
pre-commit hook bundle") at 2026-09-15T20:59:35Z, 74 minutes after that run.
.github/workflows/rainix.yaml consumes rainix-sol.yaml@main, floating, so
every push from then on picks the gate up. main has not been pushed since, so
main is stale-green over red code.

This branch. merge-base de11ac01efc51e8e86a2910a07ce5525a358f6d6. LibRainDeploySnapshot.t.sol is not in this PR's diff at all; the two functions sit at 599 and 611 on both sides, byte-identical. Both functions came in with
a201ec8 (2026-09-14), which is on main.

Clearing the two warnings will not be enough. With both flipped to pure the
compile succeeds and forge lint reports four findings CI has never printed,
every one on code no restore branch touches:

  • missing-zero-check x2 — test/concrete/MockChainDependentOwner.sol:26, both constructor address params
  • boolean-cst — the trailing : false in the semver precedes ternary, test/src/lib/LibRainDeploySnapshot.t.sol:1601
  • block-timestamp — test/src/concrete/MigrationRegistryApplyMigration.t.sol:924

Error: aborting due to 4 linter warning(s). Behind forge lint, the same
upstream commit added a pre-commit run --all-files step that has never executed
on this repo because lint fails first, so what that step does here is unknown.

All six restore branches (#219, #226, #228, #229, #232, #234) fail identically on
code none of them touches. The fix belongs on main once, not six times.

baku-ccron and others added 3 commits September 16, 2026 01:54
This branch gives `BuildHarness` a constructor; `BuildRecordRootHarness`
arrived on main after that and declares its own, so the merge left the base
unconstructed and the compile failed. Empty is `Build`'s own lib directory,
which is what the record-root harness wants: it only regenerates snapshots,
and `externalRegenerateLibs` stays refused.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01V8ViHcKLVk2YoS2joH4HdN
@thedavidmeister
thedavidmeister merged commit fcbeabc into main Sep 16, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No argument, loop bound or call ordering inside Build's generator hooks is asserted anywhere

1 participant