[avallete]

TL;DR

On Docker Desktop for macOS, every upload to the CLI stack's Storage returned 500 ENOTSUP. Storage's file backend keeps object metadata in extended attributes, and Docker Desktop's bind mounts reject them. The slim Storage build now wraps fs-xattr: it uses native xattrs when they work, and otherwise falls back to JSON sidecar files, which a throttled background sweep keeps from piling up. The CLI can keep its bind mount, with no engine-volume workaround.

Before

flowchart LR
  A["Upload into bind mount<br/>(Docker Desktop macOS)"] --> B["fs-xattr setAttributeSync"]
  B -->|ENOTSUP| C["500 on every upload"]

Loading

After

flowchart LR
  A["Upload into bind mount<br/>(Docker Desktop macOS)"] --> B["fs-xattr wrapper"]
  B -->|native xattrs work| C["native attributes"]
  B -->|ENOTSUP| D[".slim-xattr/&lt;2 hex&gt;/&lt;sha256&gt;.json"]
  E["Background sweep<br/>30 s after start, then hourly"] -.->|removes entries whose file is gone| D

Loading

Why

Docker Desktop mounts host directories over virtiofs with no_xattr. The stack bind-mounts supabase/.temp/stack-uploads/<stack-id> to /mnt, and Storage's setAttributeSync fails:

"code":"ENOTSUP","errno":95,"message":"The file system does not support extended attributes or has the feature disabled."
at Object.setAttributeSync (file:///slim-runtime/app/node_modules/fs-xattr/index.js:66:16)

The CLI-side alternative, supabase/cli#6908, moves uploads into engine volumes. It needs a probe container, volume cleanup on destroy and new prompt wording, and uploads disappear when switching Docker contexts. This PR fixes it where the failure happens instead.

What changed

• fs-xattr wrapper (services/storage/overlay/fs-xattr/index.js): prepare-bundle-dist.mjs moves the real package to node_modules/fs-xattr/native and installs the wrapper in its place.
  • Native attributes stay authoritative.
  • On ENOTSUP, values go to <storage root>/.slim-xattr/<2 hex>/<sha256(relative path)>.json, which also records the relative path.
  • A native miss also reads the sidecar, so a directory written through Docker Desktop keeps its metadata when served by an engine with xattrs.
  • The build fails if fs-xattr changes its entry point or adds an export the wrapper doesn't cover.
• Background orphan sweep: deletes, new versions and multipart parts never go through fs-xattr, so their sidecars would stay forever.
  • Removes entries whose file is gone, plus stale .tmp files.
  • Never competes with requests:
    • it starts 30 s after the process loads (each wake from idle sleep is a fresh container) and repeats hourly;
    • it uses async I/O one entry at a time and pauses after every 50 entries, longer when the filesystem is slow.
  • A .sweep-cursor lets a pass cut short by idle sleep resume on the next start.
  • A .sweep-lock keeps one sweeper per directory; it's refreshed per batch and expires after 2 minutes.
• Smoke (services/storage/smoke.sh):
  • SLIM_STORAGE_XATTR_SIDECAR=force makes a container act as if xattrs were rejected. It uploads with a content type and cache control, checks the headers and the sidecar, then deletes an object.
  • A normal-mode container on the same volume must still serve the metadata and sweep the deleted object's sidecar (SLIM_STORAGE_XATTR_SWEEP=immediate).
• Unit tests: services/storage/fs-xattr-fallback.test.ts (bun, fake native module), added to repository checks.
• Decision record: services/storage/REPORT.md.

Reviewer notes

• Validation so far:
  • The first commit passed validation_only release runs for v1.79.27 (unpublished) and v1.79.21 (floor) on linux/amd64, linux/arm64 and darwin/arm64.
  • The sweep commit passed the same runs, including the delete-then-sweep smoke step on both Linux targets: v1.79.27, v1.79.21.
• Docker Desktop dogfood (29.8.1, Apple silicon), on the first commit:
  • The CLI's current pin (v1.73.0) failed with ENOTSUP; the candidate passed.
  • The stack e2e (whole-stack.docker.lifecycle.e2e.test.ts) passed.
  • A 17-scenario differential stress run gave identical status codes and headers with native xattrs and with the sidecar.
  • The sidecar's own cost is about 6% on small uploads and 8–14% on large ones.
  • Report and harness: .handoff/storage-xattr/ on supabase/cli branch avallete/stack-storage-xattr-sidecar-test.
• Windows Docker Desktop keeps xattrs natively (WSL2 9p drvfs with metadata), so the fallback never triggers there.
• Known limitations:
  • An object uploaded where xattrs work, then read under Docker Desktop, serves application/octet-stream.
  • The upstream supabase/storage-api image reading a fallback directory serves application/octet-stream / no-cache.
  • The fallback needs an absolute storage root.
• Release follow-up:
  • The CLI stack pins Storage v1.73.0, below this repo's floor v1.79.21, so the CLI needs a pin bump to a fixed version to benefit.
  • After merge, dispatch hotfix=true for any published v1.79.21–26 that should carry the fallback.

Part of supabase/cli#6900

🤖 Generated with Claude Code

[jgoux]

A few non-blocking notes; none of them block merge.

[jgoux]

Non-blocking: when the native set succeeds, any existing sidecar copy of attr stays as it was, whereas removeAttributeSync clears it (line 138). Regular uploads, copies and TUS always write fresh versioned paths, so object metadata is unaffected. Multipart part-N, though, is rewritten in place:

1. Upload part N where xattrs are rejected (sidecar ETag A).
2. Re-upload it on an engine with xattrs (native ETag B).
3. Complete back on the no-xattr engine: the sidecar answers A, and completion fails with InvalidChecksum.

That takes two engine switches during one upload, so it's contrived. Still, dropping or updating the sidecar entry after a successful native set would match the remove path.

[jgoux]

Non-blocking: the cursor only advances once a whole shard is done. A pass that keeps getting cut short partway through one shard restarts that shard on every wake and never reaches the later shards. That only happens if one shard (about 1/256 of all sidecars) takes longer than the time the container stays awake, so only for very large projects. If it ever comes up, saving the last processed entry name within the shard (over a sorted readdir) would fix it.

[jgoux]

Non-blocking: two processes that both see a stale lock can both rm it and continue. The second rm can delete the lock the first one just created, and whichever finishes first then deletes the other's lock in the finally at line 260. Removals are idempotent, so this only duplicates work, which fits the best-effort note above. Just flagging it in case the lock ever needs to guarantee more than that.

[jgoux]

Nit: not every path is fresh. The Iceberg S3 PutObject route calls backend.uploadObject(icebergBucket, key, undefined, …) (src/http/routes/s3/commands/put-object.ts), so those paths have no version and can be rewritten. Each upload sets both attributes again, so the only exposure is a narrow race: a delete and a re-upload of the same path land between the sweep's stat and its rm, and the sweep removes the new sidecar. It may be worth softening this comment, and the matching "never reused" line in REPORT.md.

[jgoux]

Nit: the wrapper doesn't reject a relative root. storageRoot() resolves it against process.cwd(), while FileBackend resolves it against __dirname/../../... Those usually differ, so the file falls outside the wrapper's root and the original ENOTSUP is rethrown. Wording like "a relative root is resolved against the working directory, which usually differs from Storage's, so the original ENOTSUP remains" would be more accurate.