Skip to content

feat(cache): restore symlinks a task creates - #918

Open
wan9chi wants to merge 2 commits into
feat/cache-empty-foldersfrom
feat/cache-created-symlinks
Open

wan9chi wants to merge 2 commits into
feat/cache-empty-foldersfrom
feat/cache-created-symlinks

Conversation

@wan9chi

@wan9chi wan9chi commented Oct 30, 2026 •

Copy link
Copy Markdown
Member

Motivation

A task that creates a symbolic link doesn't have it among its outputs unless an output glob matches it. Nitro's dependency tracer, for example, links .output/server/node_modules/tslib into .nf3. fspy records nothing for symlink, so a hit leaves the link out. When the task checks the path first, as nf3 does, the link is a plain input and a clean checkout misses.

Changes

fspy records the path of a new link as a write, before the call:

  • Unix preload: symlink and symlinkat.
  • Linux seccomp supervisor: the same calls for static binaries.
  • Windows: NtFsControlFile with FSCTL_SET_REPARSE_POINT(_EX), which covers CreateSymbolicLinkW and junctions.

vt needs no change: a written symlink counts as a file, and the archive stores links as links (#885).

E2E cases: symlink_in_output (synthetic, vtt symlink) and nitro_auto_outputs_restore_links (the nitro_build_cache Nitro build with automatic tracking only). Both run only on Unix in the first commit, because Windows already behaves differently there, and on every platform after.

Part of a stack that lets automatic tracking cache tasks which read their own outputs and list the folders holding them, so they also hit on a clean checkout.

🤖 Generated with Claude Code


Note

Medium Risk
Changes cross-platform file-access tracing used for cache fingerprints; incorrect recording could cause wrong hits or misses, but scope is limited to symlink/junction creation paths.

Overview
Automatic caching now treats symlinks a task creates as restorable outputs, even when no output glob matches them (e.g. Nitro nf3 links under .output/server/node_modules). Cache hits replay those paths as links, and a later run still hits if the links are missing, like on a fresh clone.

fspy records the new link path as a write before the syscall returns: on Unix via symlink / symlinkat (preload and seccomp), and on Windows via an NtFsControlFile hook when FSCTL_SET_REPARSE_POINT(_EX) runs (symbolic links and junctions). Existing archive/restore behavior for matched outputs is unchanged; this closes the gap where creation was invisible to tracking.

Docs note that Windows junctions count as written symlinks in rebuildable logic. vtt symlink works on Windows for e2e. New fspy and e2e tests cover symlink tracking and Nitro-style auto-tracked link restore.

Reviewed by Cursor Bugbot for commit 25a585d. Bugbot is set up for automated code reviews on this repo. Configure here.

@wan9chi
wan9chi added this pull request to stack #920 October 30, 2026 03:17
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from ce2ff95 to 6f295b5 Compare October 30, 2026 03:19
@github-actions

github-actions Bot commented Oct 30, 2026 •

Copy link
Copy Markdown

fspy benchmark

linux

dynamic/launch             change  +0.39%  [ -2.44% ..  +3.21%]  overhead   +55.64%
dynamic/access             change  -0.46%  [ -2.55% ..  +1.39%]  overhead   +12.51%
dynamic/access-relative    change  +0.67%  [ -0.67% ..  +1.85%]  overhead   +61.16%
dynamic/access-contended   change  -0.41%  [ -2.43% ..  +1.80%]  overhead   +15.00%
static/launch              change  +0.90%  [ -1.86% ..  +3.21%]  overhead  +189.72%
static/access              change  +0.43%  [ -0.69% ..  +1.59%]  overhead  +800.47%
static/access-relative     change  +0.35%  [ -0.88% ..  +1.27%]  overhead +1378.84%
static/access-contended    change  +0.46%  [ -0.05% ..  +1.20%]  overhead +3221.20%

macos

dynamic/launch             change  +0.07%  [ -4.15% ..  +5.41%]  overhead  +237.35%
dynamic/access             change  +0.35%  [ -4.63% ..  +4.30%]  overhead    +5.45%
dynamic/access-relative    change  -2.20%  [ -6.17% ..  +4.62%]  overhead  +247.19%
dynamic/access-contended   change  +0.00%  [ -8.11% ..  +5.24%]  overhead    +2.33%

windows

dynamic/launch             change  -0.19%  [ -2.45% ..  +2.02%]  overhead   +24.87%
dynamic/access             change  +0.00%  [ -0.56% ..  +0.56%]  overhead    +1.14%
dynamic/access-relative    change  +0.00%  [ -0.72% ..  +0.91%]  overhead    +1.28%
dynamic/access-contended   change  +0.00%  [ -0.54% ..  +0.36%]  overhead    +1.45%

@wan9chi wan9chi added the ci: full Run the slow CI jobs (macOS x64, musl) on this pull request label Oct 30, 2026
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 6f295b5 to 50eb6eb Compare October 30, 2026 04:07
@wan9chi wan9chi added ci: full Run the slow CI jobs (macOS x64, musl) on this pull request and removed ci: full Run the slow CI jobs (macOS x64, musl) on this pull request labels Oct 30, 2026
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 50eb6eb to 3265412 Compare October 30, 2026 06:31
Two cases for a task that creates symbolic links. In `symlink_in_output`,
the task links `out/link` to `out/target.txt` with `vtt symlink`, which now
works on Windows too, making a directory link when the target is a
directory. In `nitro_auto_outputs_restore_links`, the Nitro build of
`nitro_build_cache` runs with automatic tracking only, without `input` or
`output` globs; nf3 links `tslib` and each package's `tslib` into `.nf3`.

fspy records nothing for `symlink` on Unix, so a link a task creates isn't
an output. Removing `out/link` is a cache hit that leaves it missing. nf3
checks each path before linking it, so Nitro's links are plain inputs,
and removing `.output`, as on a clean checkout, is a cache miss. Several
inputs are then missing, and vt names whichever its parallel validation
finds first, so the step redacts its miss reason.

Both cases are Unix only for now. On Windows, `CreateSymbolicLinkW` opens
the new link without write access, so fspy records it as a read, and
removing `out/link` is a cache miss. nf3 makes junctions there, which
libuv opens with write access before setting the reparse point, so
they're recorded as written already.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A task that creates a symbolic link, like Nitro's dependency tracer
linking `.output/server/node_modules/tslib` into `.nf3`, didn't have the
link among its outputs unless an `output` glob matched it: fspy recorded
nothing for `symlink`, so a hit left the link out, and when the task
checked the path first, as nf3 does, the link was a plain input, so
removing it, as on a clean checkout, was a cache miss.

fspy now records the path of a new link as a write, before the call, so a
failed call is recorded too. The target is stored as text, not accessed,
and needs no record.

- Unix preload: `symlink` and `symlinkat`.
- Linux seccomp supervisor, for static binaries: `symlinkat`, plus
  `symlink` on x86_64.
- Windows: `NtFsControlFile` with `FSCTL_SET_REPARSE_POINT(_EX)`, for the
  file the reparse point is set on. `CreateSymbolicLinkW` creates the link
  with an open that lacks write access, recorded as a read, and then sets
  its reparse point. A junction, which libuv makes for Node's `fs.symlink`
  with `'junction'` and nf3 uses on Windows, is a folder with a mount
  point set on it; libuv opens the folder with write access first, so
  those junctions were recorded as written already.

vt needs no change: `Rebuildable` treats a written symlink as a file, and
the archive stores links as links (#885). std reports a junction as a
symlink, not a folder, so the mkdir recorded for it doesn't make it an
empty folder, and the write makes it rebuildable. Both e2e cases now run
on Windows too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@wan9chi
wan9chi force-pushed the feat/cache-created-symlinks branch from 3265412 to 25a585d Compare October 30, 2026 07:08

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci: full Run the slow CI jobs (macOS x64, musl) on this pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant