Skip to content

Stop make check rewriting typescript/package-lock.json - #631

Merged
jeremy merged 24 commits into
mainfrom
fix/macos-lockfile-churn
Aug 4, 2026
Merged

jeremy merged 24 commits into
mainfrom
fix/macos-lockfile-churn

Conversation

@jeremy

@jeremy jeremy commented Aug 4, 2026 •

Copy link
Copy Markdown
Member

Closes #612.

make check left typescript/package-lock.json modified every run — 24 deletions, each a libc: [glibc|musl] array on a Linux-only optional dependency. Developers had to notice and restore it or risk committing the rewritten file, which then flips back for whoever runs it next. Pre-freeze blocker for v0.13.0, since release verification runs through make check.

Correction: this is not macOS vs Linux

The issue and the first version of this description both framed the churn as a platform split. It is npm-version dependent, and byte-identical across operating systems within a version. Same package.json, resolved four ways, sha256 of the result:

Platform npm sha256
macOS (darwin/arm64) 10.9.8 034075df11e3
Linux (x86_64) 10.9.8 034075df11e3
macOS (darwin/arm64) 11.19.0 3c96ea13247b
Linux (x86_64) 11.19.0 3c96ea13247b

npm >= 11.11.0 records libc on Linux-only optional dependencies; every npm below it drops them. Bisected rather than asserted: 11.6.0 → 0 entries, 11.10.0 → 0, 11.11.0 → 18, 11.12.0 → 18.

The platform correlation was an artifact of who runs which npm. Linux developers on the repo's own pin strip the hints just as thoroughly — measured above, not assumed. The writers genuinely straddle the threshold:

Writer npm writes libc?
.mise.toml node 22 (local) 10.9.8 no
CI test.yml node 22 10.9.x no
CI security.yml / release-typescript.yml node 24 11.5.1 no
Dependabot current, 11.19.x yes

That is the whole history of the file: dc5f17ee6 (a Dependabot bump) added eight libc arrays; #624 removed them again on a local reconcile — the "24 deletions" in the report, collateral to a one-line override change.

I have corrected #612's text to match.

The fix

conformance/runner/typescript's pretest shelled into typescript/ and ran npm install, which reconciles package-lock.json in place and writes the result. Nothing else in the check path writes it — make ts-check and both conformance targets already use npm ci.

The first cut of this PR simply swapped in npm ci there, and that was wrong. typescript/node_modules is shared with ts-check, and npm ci deletes node_modules before installing — so under make -j check the conformance runner could delete dependencies out from under a sibling TypeScript target mid-run. That trades a dirty file for an intermittent, load-dependent failure, which is worse.

Fixed in the dependency graph instead. conformance-typescript (and -live) now depend on ts-build, routing the install through the ts-install stamp that make already serialises; the runner's own npm ci touches only its private node_modules:

$ make -n conformance-typescript
cd typescript && npm ci
touch typescript/node_modules/.install-stamp
cd typescript && npm run build
cd conformance/runner/typescript && npm ci && npm test

The race, reproduced both ways

Not a theoretical concern. With the pre-fix shape restored — pretest running npm ci in the shared typescript/, and conformance-typescript not depending on ts-build — make -j8 ts-check conformance-typescript fails every time:

RACE attempt 1: MY_RACE_EXIT_IS 2
RACE attempt 2: MY_RACE_EXIT_IS 2
RACE attempt 3: MY_RACE_EXIT_IS 2
RACE attempt 4: MY_RACE_EXIT_IS 2

make: *** [ts-test] Error 127          # binary vanished mid-run
mise ERROR No version is set for shim: tsc
make: *** [ts-typecheck] Error 1
ERROR: TypeScript generation failed:
make: *** [ts-check-drift] Error 1

The same command on the fixed shape, with the install stamp deleted each time so npm ci genuinely re-runs:

parallel run 1: MY_PAR_EXIT_IS 0
parallel run 2: MY_PAR_EXIT_IS 0
parallel run 3: MY_PAR_EXIT_IS 0

Why the lockfile content stays as it is

Pinning a toolchain converges nothing, because Dependabot's npm is not ours to pin and it moves. So the question is which committed content survives contact with every npm in play. Reconciling both candidates under each — npm install --package-lock-only --ignore-scripts, then sha256:

Committed form npm 10.9.8 npm 11.5.1 npm 11.19.0 fixed point?
stripped (no libc, current) 4c69c0050607 4c69c0050607 4c69c0050607 yes, under all three
libc-bearing (#624's parent) 4c69c0050607 4c69c0050607 48decd5d00e8 only under 11.19

The committed stripped form is the only fixed point, so it stays. Restoring the eight arrays would look like repairing #624's collateral damage while actually re-arming the identical 24-line churn for every npm below 11.11.0 — including the one this repo pins. Worth revisiting when the node floor moves past that threshold; until then the rule worth keeping is that the lockfile matches what the pinned toolchain produces.

No consumer impact either way: the lockfile is not published (typescript/package.json ships dist and src/generated), and the eight entries are dev-only @oxlint/binding-linux-* optionals.

Two guards, both with their own tests

scripts/lint-npm-lockfile-writes — a diagnostic, not the gate

An allowlist of permitted invocations, not a denylist of writers. The first cut denied a list of writing subcommands and four ordinary spellings walked through it — npm accepts any unambiguous prefix, and flags may sit between the command and its subcommand:

npm in                                 npm ins
npm --prefix ../typescript install     npm --prefix=../typescript install

A denylist has to enumerate every spelling of every writer forever. So ALLOWED_SUBCOMMANDS and the option tables are fail-closed inventories of what cannot write a lockfile; anything else is reported with instructions to extend them. Getting the tables wrong costs a false alarm, never a miss.

Review drove that parser through eight more bypasses, each a spelling a person might plausibly write, each now pinned:

Bypass Resolution
npm in, npm ins allowlist, not denylist
npm --prefix <path> install flags consumed before the subcommand
npm --shell ci install unknown options fail closed — a value can impersonate a subcommand
npm ci;npm install, &&, |, $(…) operators padded before tokenising
npm --prefix \ + newline an unresolvable invocation is reported, not assumed innocent
"npm" install, n"p"m install quoting resolved by balance
a script's 2nd line NUL-framed records; newline rewritten to ;
npm audit --audit-level high fix one shared option-skipper for both call sites
sh -c 'npm install' -c argument scanned as code, not as a quoted string
npm audit ... \ + newline + fix continuations joined before tokenising
a writer in go/Makefile every tracked Makefile walked, not just the root
echo ok # npm install … false alarm — the scan stops at an unquoted #
echo "please npm install here" false alarm — multi-word quoted spans are data
np\m install escapes decoded, not just a trailing one
echo "npm" install false alarm — only words in command position count
npm exec -- npm install exec off the allowlist; it wraps an arbitrary command
CI=1 npm install assignments keep the command position
if false; then npm install; fi keywords are syntax; they keep the position

Two of these cut both ways, and the self-test pins the over-correction as well as the bypass:

  • Stripping quotes anywhere fixed n"p"m and broke grep -Eq 'npm (run )?test' — a quoted regex read as an invocation. Balance separates them: a token that closes every quote it opens is the command the shell would run; one that leaves a quote open begins a string, and strings are data.
  • Except after sh -c, where the quoted text is a program. A wrapper switches the scan into code mode; an operator ends it, so data stays data past the &&.

73 cases, and shown non-vacuous — each mutation of the gate fails exactly the cases that mechanism owns:

Mutation failing cases
allow install 13
drop the value-flag skip 2
accept an unparsed subcommand 2
drop operator padding 4
strict npm grep filter 2
whole-token-only unwrap 2
unconditional quote removal 2 (the accept side)
newline-framed records 2
audit lookahead skips names only 1
--audit-level not value-taking 4
no shell-wrapper handling 3
wrapper context never reset 2 (the accept side)

It also had to be made cheap: the first version forked a subshell per tracked line and took 234 seconds against a comment claiming 0.2. The parser now runs in-process behind a filter. 0.5s.

conformance/runner/typescript/assert-sdk-built.mjs — the runner may not test a stale SDK

The runner resolves the SDK through its package exports, which point at typescript/dist, so a bare npm test there tests whatever was last built. The old pretest made that impossible by rebuilding every run — and that rebuild is what raced ts-check, so it could not come back. A read-only freshness assertion took its place, and review found two ways it still lied:

  • A deleted source. Comparing the newest file mtime passes after a deletion: every surviving file is older than dist, while dist still carries the removed module. Directory mtimes are the only record, so they are walked too — which covers additions and renames as well.
  • A build that did not finish. tsconfig does not set noEmitOnError, so tsc can emit a partial dist and still exit non-zero, skipping postbuild. postbuild now writes dist/.build-complete after the copy, and the assertion requires that marker rather than the entry point.
  • And because tsc does not clean outDir, a successful rebuild after a deletion still served the stale module through the ./oauth export — the marker was making a stale build look verified. prebuild now removes dist first. Measured on the real build: probe source present after build, gone after delete-and-rebuild.
  • A dependency bump. typescript/package-lock.json was not among the inputs, so an update that changes the compiler — and therefore the emitted JavaScript and declarations — left dist looking current.
  • A deleted input, backwards. newestMtime() returns 0 for a missing path, so removing tsconfig.json lowered the computed source time and made the build look fresher than leaving it alone. The three file inputs must now exist, and the SDK root's own mtime — read shallowly, so dist/ and node_modules/ cannot make it look perpetually new — is folded in.

13 cases; the mutant restoring file-only mtimes fails exactly the three deletion/rename cases, and the one ignoring the marker fails exactly the partial-build case.

Two of those were the gate failing in exactly the way the bug it guards against failed: a check that silently passes. A root-only Makefile walk reported success over go/Makefile without opening it, and an unparsable continuation was waved through rather than reported. Both now fail closed.

And a third guard that does not parse at all

Review found nine spellings the parser missed, across six rounds. The pattern was
the finding. The parser is already fail-closed wherever it classifies — unknown
subcommand, unrecognised option, unresolvable subcommand are all reported — but
the misses are one step earlier, in deciding whether a word is a command at all,
and that is shell parsing. Inverting there means failing on every quoted mention
of npm, which review independently reported twice as an unacceptable false alarm
(echo ok # npm install …, echo "please npm install here"), neither with an
allowlist remedy.

So make check now also observes. scripts/assert-lockfiles-unchanged hashes
every dependency manifest before the checks and again after — npm lockfiles,
Gemfile.lock, uv.lock, and the Go checksum files — and any difference fails.
check is a thin wrapper around check-targets so the hashes bracket the run,
and --verify runs whether or not the checks passed, preserving their exit
status. It parses nothing:

  ok    written via eval of an assembled string — rejected (exit 1)
  ok    written by a command held in a variable — rejected (exit 1)
  ok    a NEW untracked lockfile created at the repo root — rejected (exit 1)
  ok    go.work.sum rewritten — rejected (exit 1)
  ok    failing checks still get the tripwire diagnostic, exit preserved

It runs in CI too, and that is where it is authoritative. --verify-clean
needs no snapshot — a CI checkout starts pristine, so the committed tree is the
baseline — and it asserts that no dependency manifest differs from HEAD and that
none has appeared. That second half matters: a script creating a lockfile where
none was tracked is invisible to anything built on git ls-files. It runs at the
end of all 20 jobs that install dependencies (11 in test.yml, 9 in
security.yml) with if: always(), so a job cannot both fail and hide what it
dirtied.

It earned its keep on the first run: two jobs where the step could not even
execute (a working-directory default, and an aggregator job with no checkout)
were found and fixed before this could claim to be passing.

24 cases. The two are complementary: the static gate fails at the commit that
introduces the writer, naming file and line, and works even where the write is a
byte-level no-op — which is the common case, since the committed lockfile is a
fixed point under every npm in play. The tripwire cannot say what went wrong, only
that something did — but nothing can talk a byte comparison out of noticing.

Every make check run now ends with Lockfiles unchanged by the checks (7 files, byte-identical).

Why the parser is a diagnostic and not the gate

Six rounds of review found eleven spellings the parser missed, and every one
was a silent hole rather than a loud false alarm — the worst failure mode a gate
can have. Separating code from data is shell parsing; this is not a shell, and
patching grammar does not converge. Review put it best in the mid-token thread:
"otherwise only the post-check tripwire notices." The tripwire already caught
all of them.

So that is now the architecture rather than an accident. The parser was renamed to
lint-npm-lockfile-writes, prints and exits 0, and holds no gating authority.
--strict still exits non-zero, which is how its own self-test observes the
rejections.

question can be out-spelled? silent when?
assert-lockfiles-unchanged — the gate did anything change? no, it reads no source the write is byte-neutral
lint-npm-lockfile-writes — diagnostic is anything able to write? yes never — it reads the source

The diagnostic is still worth running for the one thing the comparison cannot do:
name the file, the line and the reason at the commit that reintroduces a writer.
Today the comparison would not even notice that commit — the committed lockfile
is a fixed point under npm 10.9.8, 11.5.1 and 11.19.0
, so a reintroduced install
writes back identical bytes until some npm's output diverges. The diagnostic is
the early warning for exactly that window, and a spelling it misses is now a
weaker message rather than a shipped hole.

The three live findings were fixed anyway, because a diagnostic that cannot read
@npm install — the standard silent-recipe form — is worth very little:

Bypass Resolution
@npm install, +npm update make strips @, -, + before the shell sees them
np\ + newline + m install recipe lines joined on a trailing backslash before filtering
env -C . npm install command-running prefixes consume their own value-taking options

73 cases.

What these guards are not

Stated in the gate's header rather than left implied: this is a regression guard against an accidental reintroduction — the pretest in #612 was one well-meant line — and it reads text with a shell-shaped parser, not a shell. A parser that is not a shell can always be out-argued by eval on an assembled string or a variable holding the subcommand. A green run means nobody reintroduced this by accident; it is not a security boundary.

fast-uri: >=4.1.2 → >=3.1.5 <4

Loose end from #624. Both escape GHSA-7p8r-x3mc-p8w7, but @redocly/ajv — the only consumer in the graph — declares fast-uri: ^3.0.1. The >=4.1.2 floor forced a transitive major outside the range its dependent asks for; 3.1.5 is the patched release on the line it actually wants, and #625 already showed 3.1.5 viable in the conformance runner's graph.

It resolves cleanly. Measured at CI's exact threshold, npm audit --audit-level=high:

override ">=3.1.5 <4"  ->  node_modules/@redocly/ajv/node_modules/fast-uri 3.1.5  ->  exit 0, found 0 vulnerabilities
override "3.1.5"       ->  node_modules/@redocly/ajv/node_modules/fast-uri 3.1.5  ->  exit 0, found 0 vulnerabilities
override ">=4.1.2"     ->  node_modules/fast-uri 4.1.2                            ->  exit 0, found 0 vulnerabilities

The bounded range wins: staying on the patched 3.x line is less risk than a major bump for a transitive toolchain dependency, and it puts the resolution back inside ^3.0.1.

The audit is not vacuously green — the gate was shown to fire, including against the nested placement the 3.x resolution produces:

override "3.1.4"  ->  node_modules/@redocly/ajv/node_modules/fast-uri 3.1.4
MY_RED_AUDIT_EXIT_IS 1
fast-uri  3.0.0 - 3.1.4
fast-uri vulnerable to host confusion via backslash authority introducer - GHSA-7p8r-x3mc-p8w7
2 high severity vulnerabilities

The override-floor pattern — recommendation, not a rewrite

An open-ended >=x.y.z floor constrains only the low end. It cannot stop npm resolving forward into a newer major carrying its own vulnerable range, and the override never looks stale while it happens. That is precisely what bit us: >=3.1.3 satisfied none of the advisory's three patched lines and left npm free to land on 4.1.1, inside >=4.0.0 <4.1.2. brace-expansion needed the same treatment two days earlier in #616 — two data points, one shape.

Recommended policy: write a security override as a bounded range that pins the major — ^<patched>, or >=<patched> <<next-major> — so the floor lifts the package out of the known-vulnerable range without handing npm a blank cheque on everything above it. Prefer the major the dependents actually declare; jump majors only when the advisory leaves no patched release on that line, and then bound the new one.

Against that policy, the four overrides today:

Override Declared by Resolves Shape
fast-uri: ">=3.1.5 <4" @redocly/ajv ^3.0.1 3.1.5 bounded — this PR
js-yaml: "^4.3.0" @redocly/openapi-core ^4.1.0 4.3.0 already bounded, nothing to do
brace-expansion: ">=5.0.9" minimatch ^5.0.2 5.0.9 open-ended → would be ^5.0.9
minimatch: ">=10.2.1" @redocly/openapi-core ^5.0.1 10.2.4 open-ended, and five majors past its declarer

Deliberately not changed here — one advisory, one fix, and neither is currently failing. Flagged for a decision rather than folding a silent dependency-resolution change into a lockfile bugfix. minimatch is the one worth a real look: it is both open-ended and already forced far outside ^5.0.1.

Acceptance proof

Not "one clean run" — two consecutive runs leaving the tree empty, since anything that merely stabilises one writer while flipping another just relocates the churn.

git status --porcelain empty after both runs, and PRE_SHA == POST_SHA confirms both describe the same tree — a long make check is not atomic.

Exit codes were written with a distinct marker to a private file rather than grepped from the combined log, because this repo's own tooling prints a line reading REAL_EXIT=0 that a generic grep would match.

Cross-platform acceptance test — the same commit, checked out and run on both, comparing the lockfile byte for byte:

And the fix is npm-version-proof, not merely green under the pinned npm — npm ci under the npm that does write libc still leaves the file alone:

$ npx npm@11.19.0 ci --prefer-offline --silent   # Linux
MY_NPM11_CI_EXIT_IS 0
before=4c69c0050607f89656a2e444e9b3b9d7bad381549a2002ede9c8c7f7cd92ceec
after =4c69c0050607f89656a2e444e9b3b9d7bad381549a2002ede9c8c7f7cd92ceec
NPM11_CI_DID_NOT_WRITE_LOCKFILE

Neither spec/basecamp.smithy nor openapi.json is touched by this branch.

Note on CI checks. pull_request-triggered workflows stopped firing repo-wide at 01:44Z (the last was on another branch), while pull_request_target ones kept working — so the checks list on this PR lags its head. Not caused by this branch, and closing/reopening did not re-trigger them. Test and Security were therefore dispatched manually against each head; results are quoted above and are green, including Spec Gates (with both new gate steps), npm Audit (TypeScript SDK) and Secret Scanning.

Copilot AI balanced review requested due to automatic review settings August 4, 2026 00:48
@jeremy jeremy added the bug Something isn't working label Aug 4, 2026
@github-actions github-actions Bot added github-actions Pull requests that update GitHub Actions typescript Pull requests that update TypeScript code conformance Conformance test suite labels Aug 4, 2026
@github-actions

github-actions Bot commented Aug 4, 2026 •

Copy link
Copy Markdown
Contributor

Sensitive Change Detection (shadow mode)

This PR modifies control-plane files:

  • .github/workflows/security.yml
  • .github/workflows/test.yml

Shadow mode — this check is informational only. When activated, changes to these paths will require approval from a maintainer.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: e1de168ca7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread conformance/runner/typescript/package.json Outdated
Comment thread scripts/check-npm-lockfile-readonly Outdated
jeremy added a commit that referenced this pull request Aug 4, 2026
Three review findings from #631.

1. `npm ci` in the runner's pretest raced ts-check under `make -j`.

The pretest shelled into typescript/ and installed there, and typescript/
node_modules is shared with ts-check. `npm install` merely rewrote the lockfile;
`npm ci` deletes node_modules before installing, so swapping one for the other
traded a dirty file for a target that can delete dependencies out from under a
sibling mid-run — intermittent, load-dependent, and worse than the bug.

Fixed in the dependency graph rather than the command. conformance-typescript
(and -live) now depend on ts-build, which routes the install through the
ts-install stamp that make already serialises. The runner's own `npm ci` touches
only its private node_modules. pretest keeps a read-only existence check so a
bare `npm test` in the runner directory still says what to do instead of failing
on a missing import.

    $ make -n conformance-typescript
    cd typescript && npm ci
    touch typescript/node_modules/.install-stamp
    cd typescript && npm run build
    cd conformance/runner/typescript && npm ci && npm test

2. The writer denylist false-greened four ordinary spellings.

npm accepts any unambiguous prefix, so `npm in` and `npm ins` are `npm install`;
and flags may sit between the command and its subcommand, so
`npm --prefix ../x install` and `npm --prefix=../x install` never matched an
`npm install` pattern at all. All four write the lockfile. A denylist has to
enumerate every spelling of every writer forever, which is the wrong shape.

Inverted: ALLOWED_SUBCOMMANDS is a fail-closed inventory of invocations that
cannot write a lockfile, and anything else fails with instructions to extend it.
The parser skips flags, consumes value-taking flags with their arguments, and
resolves the subcommand, so abbreviations and flag-prefixed forms land in the
failing branch by construction. `npm audit` passes; `npm audit fix` does not.

scripts/test-check-npm-lockfile-readonly drives the gate from outside with
synthetic single-commit repos — 20 cases, including all four bypasses, the #612
pretest verbatim, the bump-version.sh exemption, and a control proving the same
content is rejected under any other name. Shown to be non-vacuous: three
mutations of the gate each fail it (allow `install` → 9 cases, drop the
value-flag skip → 1, drop the path form → 20). The value-flag case fails on the
message fragment rather than the exit status, which is the reason the assertions
check both.

Two false positives surfaced while wiring it and are pinned as accept-cases: a
subcommand can arrive wearing the quote that closed the string it sat in
(`npm ci'`), and "node/npm is required" is prose, not a path to npm.

3. The `libc` threshold is exactly npm 11.11.0, bisected rather than asserted.

    npm 11.6.0  fresh -> 0 libc entries
    npm 11.10.0 fresh -> 0
    npm 11.11.0 fresh -> 18
    npm 11.12.0 fresh -> 18

The PR body's macOS-vs-Linux framing is corrected separately; the mechanism is
npm-version dependent and byte-identical across operating systems within a
version.
Copilot AI review requested due to automatic review settings August 4, 2026 01:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 7 out of 8 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • typescript/package-lock.json: Generated file

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 705f917612

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check-npm-lockfile-readonly Outdated
Comment thread scripts/check-npm-lockfile-readonly Outdated
Comment thread conformance/runner/typescript/package.json Outdated
@jeremy jeremy changed the title Stop make check rewriting typescript/package-lock.json on macOS Stop make check rewriting typescript/package-lock.json Aug 4, 2026
jeremy added a commit that referenced this pull request Aug 4, 2026
…fresh

Three more review findings from #631.

1. The gate cost 234 seconds, against a Makefile comment claiming 0.2.

check_text reached scan_line through a `< <(…)` process substitution, which
forks a subshell per call, and it was called on every tracked line — 15,431
under scripts/ alone. Now scan_line writes to a REASONS global and runs
in-process, and both file walks pre-filter with `grep -n npm`, since only lines
mentioning npm can carry a violation. 234s -> 0.588s measured. The Makefile
comment now says ~0.6s and why.

2. A backslash continuation walked through the allowlist.

    npm --prefix \
      typescript install

Each half alone looks harmless: the first line has no subcommand, the second has
no npm. The parser ran off the end, left `sub` empty, and the empty case fell
through to silent acceptance — the one branch that was not fail-closed. An npm
invocation whose subcommand cannot be resolved is now reported as such, so the
rejection rests on "could not read this", not on recognising a writer.

Two cases pin it, and the mutant that restores the silent-accept fails exactly
those two and nothing else:

    accept unparsed npm  ->  exit 1, 2 failing cases
      FAIL  npm --prefix \ <newline> install (line continuation)
      FAIL  a bare npm with its subcommand off the end of the line

3. pretest accepted any dist/index.js, however stale.

The runner resolves the SDK through its package exports, which point at
typescript/dist, so an existence check let `npm test` in the runner directory
report green against code no longer in the tree. The previous pretest could not:
it rebuilt every run. That rebuild is what raced ts-check, so it cannot come
back.

conformance/runner/typescript/assert-sdk-built.mjs replaces the existence check
with a read-only freshness assertion — newest mtime under typescript/src, plus
tsconfig.json and package.json, against dist/index.js. It writes nothing and
touches no shared state, and `make ts-build` satisfies it by construction:

    freshly built           -> exit 0
    source touched after    -> exit 1, "SDK build is stale"
    dist absent             -> exit 1, "SDK is not built"
    rebuilt                 -> exit 0

Self-test is now 22 cases. Gate, self-test, actions lint, runner-test
reachability and its self-test, and conformance-typescript (214 passed,
2 skipped) all green.
Copilot AI review requested due to automatic review settings August 4, 2026 02:03

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: aa691b0568

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check-npm-lockfile-readonly Outdated
Comment thread conformance/runner/typescript/assert-sdk-built.mjs Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 9 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • typescript/package-lock.json: Generated file

@jeremy jeremy closed this Aug 4, 2026
@jeremy jeremy reopened this Aug 4, 2026
jeremy added a commit that referenced this pull request Aug 4, 2026
…s check

Two more review findings from #631, both bypasses of the guards added earlier.

1. `npm ci;npm install` walked through the allowlist.

Shell operators need no surrounding whitespace, so `read -a` left `ci;npm` as a
single token. Its `ci` prefix was allowlisted and the outer scan never saw the
embedded second `npm` as a token at all. Same for `&&`, `|`, and an install
inside `$(…)`.

Every operator is now padded before tokenising, so each becomes its own word and
any npm following one starts a fresh command. Four cases pin it, and the mutant
that removes the padding fails exactly those four:

    drop operator padding  ->  exit 1, 4 failing cases
      FAIL  npm ci;npm install (no space around ;)
      FAIL  npm ci&&npm install (no space around &&)
      FAIL  npm ci|npm install (no space around |)
      FAIL  an install inside a command substitution

2. The freshness assertion still passed after a source was DELETED.

It compared the newest *file* mtime under src against dist. A deletion leaves
every surviving file older than dist while dist still carries the removed
module, so nothing looked stale. Directory mtimes are the only record of a
deletion, and they are now walked too — which also covers additions and renames.

scripts/test-assert-sdk-built drives the assertion at synthetic SDK trees
through a new CONFORMANCE_SDK_ROOT override; the deleted-source case cannot be
staged in the real checkout without destroying it. 9 cases. The mutant that
restores file-only mtimes fails exactly the three that need directories:

    file mtimes only  ->  exit 1, 3 failing cases
      FAIL  a source DELETED after the build
      FAIL  a source RENAMED after the build
      FAIL  a whole source DIRECTORY removed

3. A stale CI comment claimed a guard that does not exist.

test.yml justified using `npx vitest run` over `npm test` by saying "the
runner's dist-freshness globalSetup still fails loudly if dist/ were stale."
There is no such globalSetup anywhere in the runner, and never was — which is
precisely why a stale dist/ could have been tested silently. The comment now
says what is actually true, and names assert-sdk-built.mjs as the guard that was
missing.

Self-tests: npm gate 26 cases, freshness 9. Gate runs in 0.457s. Local gates,
actions lint, runner-test reachability + self-test, and conformance-typescript
(214 passed, 2 skipped) all green.
Copilot AI review requested due to automatic review settings August 4, 2026 02:48

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 438ad25594

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check-npm-lockfile-readonly Outdated
jeremy added a commit that referenced this pull request Aug 4, 2026
Sixth review finding on #631, and the same shape as the last three: npm accepts
any config key as `--key value`, so an option this parser does not recognise
cannot be assumed valueless.

    npm --shell ci install --package-lock-only

`--shell` takes a value, so `ci` is that value and the real subcommand is
`install` — but the parser read `ci`, found it allowlisted, and passed the line.
The invocation writes package-lock.json.

Options are now allowlisted the same way subcommands are, in three cases:

    --anything=value    one token, unambiguous, skipped whatever it is
    VALUE_FLAGS         consumes the next token too
    VALUELESS_FLAGS     consumes nothing

Anything else is reported as unclassifiable rather than guessed at. `--shell` is
also added to VALUE_FLAGS, but that is not what protects the case — mutating it
back out leaves the self-test green, because the fail-closed rule catches it as
an unrecognised option. Only the true pre-fix state fails:

    drop --shell, keep fail-closed         ->  self-test green (general rule holds)
    drop --shell AND assume valueless      ->  exit 1, 2 failing cases
      FAIL  npm --shell ci install (value swallows a fake subcommand)
      FAIL  an option this parser cannot classify

Four cases added, two rejecting and two guarding against over-rejection: known
valueless options before the subcommand, and an unknown option in `--key=value`
form, both still accepted. Self-test is 30 cases; the gate still runs in under a
second.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 10 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • typescript/package-lock.json: Generated file

Copilot AI review requested due to automatic review settings August 4, 2026 02:57
Two review findings against the tripwire added in a7e1fff — both holes in the
thing meant to have no holes, which is the right place to be strict.

1. A lockfile created where none existed was in neither snapshot.

The walk used `git ls-files`, which lists what is already tracked. A check that
created a root package-lock.json therefore appeared on neither side of the
comparison and verification passed over it, leaving the tree dirty. It now walks
the filesystem for lockfile names, pruning node_modules/.git/dist/build, so a
file that did not exist before and does now is exactly what it notices.

2. Verification never ran when the checks failed.

`@$(MAKE) check-targets` aborting the recipe meant a target that wrote a lockfile
and THEN failed produced only its own error — the dirty tree went unexplained,
which is precisely the case where the diagnostic is worth most. `--verify` now
runs regardless, and the sub-make's exit status is preserved rather than
overwritten.

Both reproduced here. For (1), the self-test driven at an index-based copy:

    MY_REDPROOF_11_EXIT_IS 1
      FAIL  a NEW untracked lockfile created at the repo root — exited 0
      FAIL  a new lockfile in a directory that had none — exited 0

For (2), the two recipe shapes run against a synthetic Makefile whose
`check-targets` writes a lockfile and exits 7:

    PRE-FIX  (abort on failure)   exit 2, tripwire diagnostic shown: False
    FIXED    (verify regardless)  exit 2, tripwire diagnostic shown: True

and the self-test's wrapper case pins the status contract itself —
WRAPPER_EXIT=7, not 1, so a lockfile failure cannot mask what actually broke.

Self-test is 12 cases. Local sweep green: three gates and their self-tests,
actions lint, runner-test reachability and its self-test, ts-check (1393 passed),
conformance-typescript (221 passed, 2 skipped).
Copilot AI review requested due to automatic review settings August 4, 2026 05:27

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5f34e2bd43

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/assert-lockfiles-unchanged Outdated
`go` rewrites go.sum as readily as `npm` rewrites a lockfile, and `make check`
runs Go targets — so `go/go.sum`, `conformance/runner/go/go.sum` and
`go.work.sum` belonged in the snapshot from the start. Nothing writes them today;
the guard is the same invariant and costs one line.

Red proof against the un-fixed tripwire, with the Go names dropped from the walk:

    MY_REDPROOF_13_EXIT_IS 1
      FAIL  go/go.sum rewritten — exited 0; it must reject this
      FAIL  the conformance runner's go.sum rewritten — exited 0; it must reject this
      FAIL  go.work.sum rewritten — exited 0; it must reject this

Self-test is 15 cases.
Copilot AI review requested due to automatic review settings August 4, 2026 05:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a13e594ac9

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check-npm-lockfile-readonly Outdated
Comment thread scripts/assert-lockfiles-unchanged Outdated
Two more review findings, and the second is a live hazard in this repo rather
than a hypothetical.

1. Shell keywords consumed the command position.

    if false; then npm install --package-lock-only; fi

`then` was treated as the command, clearing the position, so the `npm` after it
was skipped. Same for `while`/`do`, `else`, and `!`. Keywords are syntax, not
commands, and the position belongs to the word after them.

This one matters beyond the parser: a conditional that is false in CI but true on
a developer's machine is exactly the case where the static gate has to catch what
the runtime tripwire would only notice after that developer's lockfile had
already been rewritten.

2. Concurrent runs shared one snapshot path.

The tripwire keyed its baseline on the user, so two `make check` runs — this repo
is worked in from ~30 worktrees at once, and a sibling worktree was running one
while this was being written — overwrote each other's baseline. `check` now
allocates the path with mktemp per invocation and threads it through both calls,
and the bare-run fallback is keyed on the repo root.

Both shown failing against the un-fixed artefact:

    14  keywords consume position   MY_REDPROOF_14_EXIT_IS 1
          FAIL  if false; then npm install; fi — gate exited 0
          FAIL  while ...; do npm install ...; done — gate exited 0
          FAIL  ! npm install (negated) — gate exited 0
          FAIL  a writer inside an else branch — gate exited 0
    15  one snapshot path per user  MY_REDPROOF_15_EXIT_IS 1
          FAIL  default-path baselines crossed — clean=1 dirty=1

Getting 15 to bite took fixing the test first: the two fixture repos were
byte-identical, so a crossed baseline was undetectable. They now differ by
content, which is what makes the case real.

Self-tests: npm gate 68 cases, tripwire 17, freshness 13.
Copilot AI review requested due to automatic review settings August 4, 2026 05:44

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a673d3e277

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/check-npm-lockfile-readonly Outdated
The static parser has now been patched five rounds running — continuations,
comments, delegated Makefiles, `exec` delegation, assignment prefixes, control
keywords. Writing a shell parser by increments is not a winnable game, and the
right response is to stop relying on grammar for the guarantee.

`--verify-clean` is the CI form of the tripwire. A CI checkout starts pristine,
so the committed tree is already the baseline and no snapshot is needed: it
asserts that no dependency manifest differs from HEAD and that none has appeared.
It runs at the end of all 21 jobs that install dependencies — 12 in test.yml,
9 in security.yml — with `if: always()`, so a job cannot both fail and hide what
it dirtied. It parses nothing, so no spelling reaches it.

It covers untracked APPEARANCE, not only tracked mutation. That was the gap
worth closing: a script creating a lockfile where none was tracked is invisible
to anything built on `git ls-files`. `git status --untracked-files=all` sees it.

    MY_REDPROOF_16_EXIT_IS 1        (--verify-clean restricted to tracked files)
      FAIL  a NEW untracked lockfile created — exited 0; it must reject this
      FAIL  a new lockfile in a new directory — exited 0; it must reject this

On demoting the static gate: not done, deliberately, and the reason is measured
rather than a preference. The committed lockfile is a fixed point under npm
10.9.8, 11.5.1 AND 11.19.0 — every npm in play — which is exactly why it was
chosen. So a reintroduced `npm install` running in CI writes back the SAME bytes,
and the dynamic check is silent for it. The dynamic check is ground truth for
"did this run dirty the tree", which is the release-blocking property; it is not
a detector of a reintroduced writer until some npm's output diverges from the
committed file. The static gate is the only thing that catches that beforehand.

So the division of labour, now stated in both scripts:

  * dynamic, authoritative, unevadable — did anything change? Cannot be
    out-spelled, but is silent while writes are byte-neutral.
  * static, best-effort, specific — is anything ABLE to write? Catches the
    reintroduction at the commit, with a file and line, before the bytes ever
    diverge; and every miss review has found was a false negative in that
    early-warning role, never a hole in the clean-tree guarantee.

Demoting the static gate to non-gating would leave the reintroduction case
covered by nobody until an npm release moves the fixed point. Keeping both
costs 0.5s.

Self-tests: tripwire 24 cases, npm gate 68, freshness 13.
Copilot AI review requested due to automatic review settings August 4, 2026 05:50

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Jobs with a `working-directory` default resolved `./scripts/...` against that
subdirectory, so the step exited 127 rather than running. `working-directory: .`
on the step; the script cds to the repo root itself, which is what the git
pathspecs need.
Copilot AI review requested due to automatic review settings August 4, 2026 05:55

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 918dc3b517

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread .github/workflows/test.yml
Comment thread .github/workflows/security.yml
Comment thread .github/workflows/test.yml Outdated
Comment thread scripts/check-npm-lockfile-readonly Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b6f0daf02c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread scripts/lint-npm-lockfile-writes
The `conformance` job has no checkout — it only reads the other jobs' results —
so the step exited 127 there. Every job that actually installs dependencies keeps
it: 11 in test.yml, 9 in security.yml.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

…gate

Six rounds of review found eleven spellings the parser missed: unambiguous
prefixes, flags before the subcommand, unknown value-taking options, unpadded
operators, line continuations, multiline script values, quoting that splits the
command name, `sh -c`, `npm exec` delegation, assignment prefixes, control
keywords — and this round, a recipe split mid-token, `env -C`, and make's own
`@` prefix. Every one was a SILENT hole, which is the worst failure mode a gate
can have. Separating code from data is shell parsing; this is not a shell, and
patching grammar does not converge.

Codex said it plainly in the mid-token thread: "otherwise only the post-check
tripwire notices." The tripwire already catches all of them. So make that the
architecture rather than leaving it implicit.

  scripts/check-npm-lockfile-readonly  ->  scripts/lint-npm-lockfile-writes

It prints and exits 0. `--strict` still exits non-zero, which is how its own
self-test observes the rejections. A spelling it misses is now a weaker message,
never a shipped hole. The Makefile target and CI step are renamed to match, and
both scripts state the division of labour in their headers:

  * assert-lockfiles-unchanged — the guarantee. Parses nothing. Compares the
    bytes of every dependency manifest before and after the checks, and in CI
    against the committed tree. Cannot be out-spelled. Silent only while a write
    is byte-neutral.
  * lint-npm-lockfile-writes — best-effort. Reads source, so it can be fooled.
    Worth keeping for the one thing the comparison cannot do: name the file, the
    line and the reason at the commit that reintroduces a writer — which today it
    would not even notice, since the committed lockfile is a fixed point under
    npm 10.9.8, 11.5.1 and 11.19.0.

The three live findings are fixed anyway, because a diagnostic that misses the
standard silent-recipe form is not worth much:

  * `@npm install` / `+npm update` — make strips `@`, `-` and `+` before the
    shell sees them.
  * `np\` + newline + `m install` — Makefile recipe lines are joined on a
    trailing backslash before filtering, so a token split across the break is
    seen whole. This also lets the earlier `npm --prefix \` case resolve all the
    way to `npm install` rather than merely being unreadable.
  * `env -C . npm install` — command-running prefixes now consume their own
    value-taking options, so `.` is not mistaken for the command.

Self-test is 73 cases.

Two items from earlier are already in: the snapshot path is allocated per
invocation with mktemp and keyed on the repo root for bare runs (a673d3e), and
the untracked-appearance case is covered in both modes, with `find` locally and
`git status --untracked-files=all` in CI (5f34e2b, 918dc3b).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 12 out of 13 changed files in this pull request and generated no new comments.

Files not reviewed (1)
  • typescript/package-lock.json: Generated file
Suppressed comments (3)

scripts/lint-npm-lockfile-writes:268

  • The comment says GNU make strips @, - and + from the front of a recipe, but the character class only matches @ and +, so the - (ignore-errors) prefix is not stripped. A recipe written as -npm install therefore never resolves to npm in command position and slips past this diagnostic (there are mk_at/mk_plus self-test cases but no -npm case). This is only the advisory gate, so the impact is a missed warning rather than a shipped hole, but the code doesn't match its own comment. Including - in the class fixes it without introducing false positives, since a --leading token in command position is otherwise not a valid command.
    ((at_cmd)) && [[ $w == [@+]* ]] && w=${w#[@+]}

conformance/runner/typescript/assert-sdk-built.mjs:63

  • The helper name newestMtime2 is ambiguous — the 2 suffix reads like a second/duplicate version of newestMtime, whereas its actual purpose is to return a single path's mtime without recursing into it (used for the SDK root so the walk doesn't sweep dist/ and node_modules/). A descriptive name such as shallowMtime or mtimeOf would make the distinction from the recursive newestMtime clear at the call site (line 99). (Renaming also requires updating the call on line 99.)
function newestMtime2(path) {
  try {
    return statSync(path).mtimeMs;
  } catch {
    return 0;
  }
}

scripts/assert-lockfiles-unchanged:18

  • The PR description appears stale relative to the shipped design. It describes a single gate named scripts/check-npm-lockfile-readonly (with a test-check-npm-lockfile-readonly, "22 cases") that "enforces the invariant this PR claims." The code instead ships a two-part design: scripts/lint-npm-lockfile-writes (explicitly an advisory diagnostic that "is NOT the guarantee") and scripts/assert-lockfiles-unchanged (the byte-comparison guarantee, which the description never mentions by name). Consider updating the description to match the final file names and the advisory-plus-guarantee split, so reviewers/future readers aren't looking for a check-npm-lockfile-readonly script that doesn't exist. The code is the correct reference here.
# ---------------------------------------------------------------------------
# This is the guarantee. scripts/lint-npm-lockfile-writes is not.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working conformance Conformance test suite github-actions Pull requests that update GitHub Actions typescript Pull requests that update TypeScript code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

make check dirties typescript/package-lock.json on macOS

2 participants