Skip to content

fix(cache): restore symlinks in task outputs as links - #885

Merged
wan9chi merged 9 commits into
voidzero-dev:mainfrom
jasonkuhrt:fix/cache-output-symlinks
Oct 30, 2026
Merged

fix(cache): restore symlinks in task outputs as links#885
wan9chi merged 9 commits into
voidzero-dev:mainfrom
jasonkuhrt:fix/cache-output-symlinks

Conversation

@jasonkuhrt

@jasonkuhrt jasonkuhrt commented Oct 24, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Addresses causes 1 and 2 of #884: links an output glob matches now survive a cache hit.

Motivation

A cache hit leaves out the symbolic links in a task's outputs, so a replayed build differs from the one the task produced. Nitro's server build is one case: its dependency tracer links server/node_modules/tslib to .nf3/tslib@2.8.1, and after a hit the replayed server resolves tslib outside its own output. #884 has a minimal repro.

Changes

  • create_output_archive reads outputs with symlink_metadata and stores a link as a tar symlink entry with its target, never followed. A directory link, a file link and a dangling link round-trip as the task left them.
  • walk_glob_files passes links through to output collection; compute_globbed_inputs still takes regular files only, so input hashing is unchanged.
  • On Windows, tar restores every link with symlink_file. Extraction there unpacks as Archive::unpack does, then turns a restored link whose target is a directory into a directory link.
  • CACHE_SCHEMA_VERSION goes to 19, so entries archived without their links aren't replayed.

Restoring still refuses to write through a link that points outside the workspace (tar's unpack_in check); a test covers it.

Not covered: automatic output tracking still doesn't see a link being created (fspy has no symlink/symlinkat or CreateSymbolicLinkW interceptor), so a task needs an output glob that matches its links. #884 keeps that part open.

@jasonkuhrt
fix(cache): restore symlinks in task outputs as links
18504e7 
Output archives followed symbolic links and kept regular files only, and
output globs listed regular files only, so a cache hit dropped every link a
task's output held. Archive links as links, let output globs list them, and
restore directory links as directory links on Windows. Bump the cache schema
so entries archived without their links are not replayed.
@jasonkuhrt
docs(changelog): link voidzero-dev#885
dfa2fea
@jasonkuhrt jasonkuhrt mentioned this pull request Oct 28, 2026
Open
wan9chi and others added 4 commits October 29, 2026 23:35
@wan9chi @claude
fix(cache): reject symlinks in output archives on Windows
5a62d57 
`tar` restores every link on Windows with `symlink_file`, which can't link
a directory, and Windows doesn't resolve a relative link target with the
`/` separators `tar` stores. Fail on a symbolic link when creating or
restoring an output archive on Windows instead of restoring a broken link,
and remove a partly written archive when creating it fails.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi @claude
test(cache): cover symlinks restored on a cache hit end to end
c9f75e2 
Add `vtt symlink` and an output_cache_test case that restores a directory
link after deleting dist/, then again over the link left in place. The
case is Unix-only because Windows rejects links in output archives.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi @claude
test(cache): cover the links in a real Nitro build's traced dependencies
2a63ff9 
Add a nitro_build_cache e2e fixture that runs a real Nitro server build.
Its route imports two packages that depend on different versions of
`tslib` (small stand-ins committed in the fixture's node_modules), so nf3
stores each version under `.output/server/node_modules/.nf3` and links to
it. After a cache hit, each package must still resolve its own version
from the restored output. The case needs Node.js and Nitro, so it is
ignored by default, and it is Unix-only because Windows rejects links in
output archives. Add nitro to packages/tools for it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi @claude
Merge branch 'main' into fix/cache-output-symlinks
3b784ca 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@socket-security

socket-security Bot commented Oct 29, 2026 •
edited
Loading

Copy link
Copy Markdown

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff Package Supply Chain
Security		 Vulnerability Quality Maintenance License
Addednpm/​nitro@​3.0.260903-beta981008286100

View full report

@wan9chi wan9chi added the ci: full Run the slow CI jobs (macOS x64, musl) on this pull request label Oct 29, 2026
wan9chi and others added 3 commits October 30, 2026 10:12
@wan9chi @claude
Revert "fix(cache): reject symlinks in output archives on Windows"
8522e39 
This reverts commit 5a62d57, to check on
Windows CI whether the PR's original directory-link fix-up restores the
links in a real Nitro build.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi @claude
test(cache): run the Nitro case on Windows, keep the link round trip …
b1ae318 
…Unix-only

With links restored on Windows again, run the real Nitro build case there
too: nf3 creates junctions on Windows, whose targets are absolute paths.
Keep `symlinks_round_trip_as_links` and the `vtt` case Unix-only, since
Windows doesn't resolve a relative link target with the `/` separators
`tar` stores, and the restore can't replace an existing directory link.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi @claude
test(cache): use explicit globs in the Nitro case
5fb0403 
On Windows, automatic input tracking records the `.output` directory
itself, which `!.output/**` doesn't exclude, so deleting `.output` turned
the second run into a cache miss. List the inputs and outputs explicitly.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi merged commit e1ae00a into voidzero-dev:main Oct 30, 2026
21 checks passed
@wan9chi

wan9chi commented Oct 30, 2026

Copy link
Copy Markdown
Member

Thanks for working on this!

@wan9chi wan9chi mentioned this pull request Oct 30, 2026
Merged
wan9chi added a commit that referenced this pull request Oct 30, 2026
@wan9chi @claude
test(cache): check that cache hits restore output symlinks as links (#…
cb0bddc 
…911)

## Motivation

The symlink e2e cases added in #885 only check that files are reachable
through the restored paths: `vtt print-file dist/pkg/index.js` in the
`vtt` case, and `require` in the Nitro case. A restore that copied each
link's target instead of recreating the link would pass them too.
Replacing each restored link with a copy of its target leaves both cases
passing, so they don't pin down what #885 fixed.

## Changes

- `vtt read-link <path>` prints a link's target, and fails if the path
isn't a symbolic link.
- `output_globs___symlinks_restored_on_cache_hit`: after each cache hit,
`vtt read-link dist/pkg` must print `store/pkg`.
- `nitro_build_restores_traced_dependency_links`: `check.cjs` checks
that `node_modules/tslib` and each package's `node_modules/tslib` are
symbolic links into `.nf3`. It resolves both ends with `realpath`, so
the output is the same with nf3's relative targets on Unix and its
absolute junction targets on Windows.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi wan9chi mentioned this pull request Oct 30, 2026
Open
wan9chi added a commit that referenced this pull request Oct 30, 2026
@wan9chi
feat(cache): restore symlinks a task creates
6f295b5 
A task that creates a symbolic link, like Nitro's dependency tracer
linking `.output/server/node_modules/tslib` into `.nf3`, didn't have the
link among its outputs unless an `output` glob matched it: fspy recorded
nothing for `symlink`, so a hit left the link out, and when the task
checked the path first, as nf3 does, the link was a plain input, so
removing it, as on a clean checkout, was a cache miss.

fspy now records the path of a new link as a write, before the call, so a
failed call is recorded too. The target is stored as text, not accessed,
and needs no record.

- Unix preload: `symlink` and `symlinkat`.
- Linux seccomp supervisor, for static binaries: `symlinkat`, plus
  `symlink` on x86_64.
- Windows: `NtFsControlFile` with `FSCTL_SET_REPARSE_POINT(_EX)`, for the
  file the reparse point is set on. `CreateSymbolicLinkW` creates the link
  with an open that lacks write access, recorded as a read, and then sets
  its reparse point. A junction, which libuv makes for Node's `fs.symlink`
  with `'junction'` and nf3 uses on Windows, is a folder with a mount
  point set on it; libuv opens the folder with write access first, so
  those junctions were recorded as written already.

vt needs no change: `Rebuildable` treats a written symlink as a file, and
the archive stores links as links (#885). std reports a junction as a
symlink, not a folder, so the mkdir recorded for it doesn't make it an
empty folder, and the write makes it rebuildable. Both e2e cases now run
on Windows too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
wan9chi added a commit that referenced this pull request Oct 30, 2026
@wan9chi @claude
feat(cache): restore symlinks a task creates
50eb6eb 
A task that creates a symbolic link, like Nitro's dependency tracer
linking `.output/server/node_modules/tslib` into `.nf3`, didn't have the
link among its outputs unless an `output` glob matched it: fspy recorded
nothing for `symlink`, so a hit left the link out, and when the task
checked the path first, as nf3 does, the link was a plain input, so
removing it, as on a clean checkout, was a cache miss.

fspy now records the path of a new link as a write, before the call, so a
failed call is recorded too. The target is stored as text, not accessed,
and needs no record.

- Unix preload: `symlink` and `symlinkat`.
- Linux seccomp supervisor, for static binaries: `symlinkat`, plus
  `symlink` on x86_64.
- Windows: `NtFsControlFile` with `FSCTL_SET_REPARSE_POINT(_EX)`, for the
  file the reparse point is set on. `CreateSymbolicLinkW` creates the link
  with an open that lacks write access, recorded as a read, and then sets
  its reparse point. A junction, which libuv makes for Node's `fs.symlink`
  with `'junction'` and nf3 uses on Windows, is a folder with a mount
  point set on it; libuv opens the folder with write access first, so
  those junctions were recorded as written already.

vt needs no change: `Rebuildable` treats a written symlink as a file, and
the archive stores links as links (#885). std reports a junction as a
symlink, not a folder, so the mkdir recorded for it doesn't make it an
empty folder, and the write makes it rebuildable. Both e2e cases now run
on Windows too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@jasonkuhrt
jasonkuhrt deleted the fix/cache-output-symlinks branch October 30, 2026 04:47
wan9chi added a commit that referenced this pull request Oct 30, 2026
@wan9chi
feat(cache): restore symlinks a task creates
25a585d 
A task that creates a symbolic link, like Nitro's dependency tracer
linking `.output/server/node_modules/tslib` into `.nf3`, didn't have the
link among its outputs unless an `output` glob matched it: fspy recorded
nothing for `symlink`, so a hit left the link out, and when the task
checked the path first, as nf3 does, the link was a plain input, so
removing it, as on a clean checkout, was a cache miss.

fspy now records the path of a new link as a write, before the call, so a
failed call is recorded too. The target is stored as text, not accessed,
and needs no record.

- Unix preload: `symlink` and `symlinkat`.
- Linux seccomp supervisor, for static binaries: `symlinkat`, plus
  `symlink` on x86_64.
- Windows: `NtFsControlFile` with `FSCTL_SET_REPARSE_POINT(_EX)`, for the
  file the reparse point is set on. `CreateSymbolicLinkW` creates the link
  with an open that lacks write access, recorded as a read, and then sets
  its reparse point. A junction, which libuv makes for Node's `fs.symlink`
  with `'junction'` and nf3 uses on Windows, is a folder with a mount
  point set on it; libuv opens the folder with write access first, so
  those junctions were recorded as written already.

vt needs no change: `Rebuildable` treats a written symlink as a file, and
the archive stores links as links (#885). std reports a junction as a
symlink, not a folder, so the mkdir recorded for it doesn't make it an
empty folder, and the write makes it rebuildable. Both e2e cases now run
on Windows too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

ci: full Run the slow CI jobs (macOS x64, musl) on this pull request

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@jasonkuhrt @wan9chi