[61465]

What

Adds magic-byte verification to the uploader so that bucket-level
allowed_mime_types restrictions catch files whose declared MIME does not
match the actual signature.

Fixes #639 — the reporter's
case was uploading a renamed .gif with Content-Type: image/jpeg to a
bucket restricted to image/jpeg,image/png; today's validateMimeType
accepts it because only the client-declared value is checked.

How

Pre-existing path (unchanged):
validateMimeType(mimeType, allowedMimeTypes) — compares the declared
Content-Type / multipart field against the bucket's allowlist.

Added path (src/storage/validators/file-signature.ts):

1. detectMimeFromBuffer(buf) — 23 signatures covering images, documents,
   archives, audio/video, and executables. Refine functions distinguish
   RIFF/WEBP from RIFF/WAVE, and ISO BMFF ftyp sub-brands
   (mp4 / heic / avif / mov / 3gp / m4a / webm). Returns undefined on
   unknown signatures so novel formats continue to work.
2. mimeFamiliesMatch(detected, declared) — intentionally loose:
   image/jpg == image/jpeg, application/zip accepted for OOXML office
   MIMEs, application/x-cfb accepted for legacy OLE2 formats.
3. wrapWithSignatureDetection(source) — returns a PassThrough and a
   promise that resolves with the detected MIME once the first 4 KiB has
   flowed. No bytes are swallowed; the backend sees the stream unmodified.

Wiring in uploader.ts (after the existing declared-MIME check):

if (allowedMimeTypes?.length) {
  const { stream, detected } = wrapWithSignatureDetection(body)
  body = stream
  detected.then((actualMime) => {
    if (actualMime && !mimeFamiliesMatch(actualMime, mimeType)) {
      stream.destroy(ERRORS.InvalidMimeType(`${actualMime} (declared as ${mimeType})`))
    }
  })
}

Buckets with no mimetype restriction stay on the fast path — no wrapping,
no extra CPU, zero behavioral change.

Why this design

• Pure helper, zero new deps. Avoids pulling in file-type (ESM-only in
  current versions) and keeps the supply-chain surface flat.
• Pure helper, no logger/db/config imports. Follows the pattern that
  worked on JWK not used in signObjectURL #629 — anything circular is caught at compile time.
• Stream-safe. Peek-and-flush PassThrough; the backend never sees a
  truncated or delayed body. Max 4 KiB extra buffered.
• Fail-closed, 415-compatible. Mismatch raises ERRORS.InvalidMimeType,
  same error code and HTTP status (415) as a header-level rejection, so
  existing clients already handle it.
• Loose family matching. image/jpg → image/jpeg, zip → OOXML office,
  x-cfb → legacy office. Prevents false positives on legitimate uploads
  while still catching the attack cases from the report.

Tests

41 vitest cases in src/storage/validators/file-signature.test.ts:

• Detection — real-signature match for each supported format (23 cases).
• Edge cases — empty buffer, 1-byte buffer, unknown prefix.
• False-positive defense — RIFF with an unknown form chunk, ftyp with
  an unknown brand (both must return undefined, not fall back to the
  entry's default MIME).
• Family matching — alias cases (jpg/jpeg, OOXML/zip, OLE2/x-cfb) and
  the eight security-relevant mismatch scenarios the fix is meant to catch:
  • renamed .gif → image/jpeg (the exact mime-type does not check uploaded files, only the filename #639 repro)
  • polyglot JPG claiming to be text/html
  • PE executable declared as application/pdf
  • ELF executable declared as image/jpeg
  • zip claiming to be image/png
  • gzip claiming to be image/jpeg
  • mp4 claiming to be image/jpeg

Also verified mime-type.test.ts still passes (37/37) — no behavior change
to the existing declared-MIME check.

Run:

npx vitest run --config vitest.unit.config.ts src/storage/validators/file-signature.test.ts
# Test Files  1 passed (1)
#      Tests  41 passed (41)

Impact

• Buckets with no allowed_mime_types → zero code-path change.
• Buckets with restrictions → one extra PassThrough + 4 KiB peek per
  upload. Negligible overhead vs. the network / storage write cost.
• Detection failure (unknown signature) is non-fatal — the declared MIME
  stands. We only refuse when we're certain the actual family is wrong.

Security notes

This closes a real file-upload hardening gap. Common attack cases this now
catches:

1. Payload smuggling — upload an EXE/ELF declared as image/png to a
   bucket that only allows images, then trigger it from a victim client.
2. Polyglot files — crafted files that are valid JPG and valid HTML.
3. Backup exfil via image buckets — upload a zip (contains db dump)
   declared as image/jpeg.

Report: #639 by @flogesell,
open 853 days.

---

Prior work in this repo for context:

• fix: auto-select url-signing JWK from JWT_JWKS (closes #629) #1488 (JWK auto-selection, closes JWK not used in signObjectURL #629) — same pure-helper pattern used here.

[61465]

Force-pushed to isolate this PR to only the MIME magic-byte validation changes.

The branch had accidentally been cut from the fix/jwk-signing-key-629 branch (PR #1488) during local work, so the diff was showing 5 unrelated JWK commits alongside the MIME changes. The branch now rebases cleanly on master with a single commit:

8e2fee2d feat(uploader): verify uploaded MIME against magic bytes (closes #639)

Diff: 3 files, +507/-0 — all under src/storage/:

• src/storage/uploader.ts — call the validator before persisting
• src/storage/validators/file-signature.ts — the magic-byte dictionary + checker
• src/storage/validators/file-signature.test.ts — unit tests

No change to the PR's semantics or scope. Sorry for the noisy history earlier.

[61465]

Thanks @depthfirst-app — fixed in dd83df46.

You were right: application/vnd.ms-excel, application/vnd.ms-powerpoint, application/msword, and the generic application/vnd.ms-* prefix are OLE2/CFB containers, not ZIP. Pairing them with application/zip in mimeFamiliesMatch let a caller declare a legacy Office MIME and smuggle an arbitrary archive through the magic-byte check — exactly the bypass you called out.

The fix narrows the zip branch to just the OOXML prefix (application/vnd.openxmlformats-officedocument.*), which actually is a ZIP container. Legacy types continue to pair with application/x-cfb as before.

Added 4 regression cases in file-signature.test.ts so this stays caught:

• rejects zip signature declared as legacy OLE2 MIME application/vnd.ms-excel
• rejects zip signature declared as legacy OLE2 MIME application/vnd.ms-powerpoint
• rejects zip signature declared as legacy OLE2 MIME application/msword
• rejects zip signature declared as legacy OLE2 MIME application/vnd.ms-outlook (generic prefix check)

Local run: 45/45 pass on vitest run src/storage/validators/file-signature.test.ts.

[61465]

Thanks @depthfirst-app — second catch fixed in 30134a5d.

The broad startsWith('application/vnd.ms-') prefix pulled in unrelated subtypes (vnd.ms-fontobject, vnd.ms-cab-compressed, vnd.ms-xpsdocument, vnd.ms-htmlhelp) that are NOT OLE2/CFB. A caller could declare an allowed non-OLE2 vnd.ms-* type and smuggle OLE2 bytes through the magic-byte check.

Replaced the prefix match with an explicit OLE2_CFB_MIME_TYPES set:

• application/msword
• application/vnd.ms-excel
• application/vnd.ms-powerpoint
• application/vnd.ms-project
• application/vnd.ms-outlook
• application/vnd.ms-works
• application/visio
• application/vnd.visio
• application/x-ole-storage

Added 4 regression cases covering vnd.ms-fontobject, vnd.ms-cab-compressed, vnd.ms-xpsdocument, and vnd.ms-htmlhelp — all now rejected on the x-cfb path.

49/49 unit tests pass locally.

[coveralls]

Coverage Report for CI Build 41229880829

Coverage decreased (-0.1%) to 86.84%

Details

• Coverage decreased (-0.1%) from the base build.
• Patch coverage: 28 uncovered changes across 3 files (112 of 140 lines covered, 80.0%).
• No coverage regressions found.

Uncovered Changes

File	Changed	Covered	%
src/http/routes/tus/lifecycle.ts	25	9	36.0%
src/storage/validators/file-signature.ts	109	99	90.83%
src/storage/uploader.ts	6	4	66.67%

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	15732
Covered Lines:	14107
Line Coverage:	89.67%
Relevant Branches:	10097
Covered Branches:	8323
Branch Coverage:	82.43%
Branches in Coverage %:	Yes
Coverage Strength:	1214.7 hits per line

---

💛 - Coveralls

[61465]

Thanks @coveralls — rebased on master + added a515ebd2 to cover the uncovered file-signature.ts lines.

The rebase picks up #1500 (TUS X-Supabase-Object-Id, merged earlier today) which drops the 11-line regression warning from the drift. New commits added coverage for:

• ISO BMFF ftyp brand switch — 10 cases covering every arm (mp42, isom, avc1, M4V, M4A, qt, 3gp4, heic, hevm, avif) plus the default (unknown brand returns undefined).
• WAV "WAVE" tag refinement — proves a RIFF container missing the WAVE tag at bytes 8-12 does not get labelled as audio/wav (so an AVI RIFF declared as audio/wav is caught).

The remaining uncovered area in uploader.ts:701 (3 lines inside void detected.then(...)) is reachable only from a full integration test that boots the fastify + file-backend stack, which lives in src/test/uploader.test.ts and requires the Postgres test harness. The logic is covered by the 61 unit tests on mimeFamiliesMatch + detectMimeFromBuffer — happy to port an integration coverage test if you'd prefer the number to land under the uploader itself.

61/61 unit tests + biome + prettier all green locally.

[depthfirst-app]

⚪ Severity: LOW

The four-byte EBML header identifies Matroska, not specifically WebM; the subtype appears later in the EBML header. An attacker can upload a non-WebM MKV while declaring video/webm; it is classified as WebM and passes a bucket restricted to that MIME, bypassing the format policy.
Helpful? Add 👍 / 👎

💡 Fix Suggestion

Suggestion: Add a refine function to the EBML entry so it reads the DocType element (ID 0x42 0x82) in the EBML header. Return 'video/webm' only when the DocType is 'webm'. Return 'video/x-matroska' when it is 'matroska'. Returning 'video/x-matroska' matters because detectMimeFromBuffer treats an undefined refine result as 'unknown'. Unknown types are not rejected, so an MKV declared as video/webm would still pass. With a distinct MIME, mimeFamiliesMatch flags the mismatch against a WebM-only bucket. Also add a test that uses a Matroska header.

⚠️ Experimental Feature: This code suggestion is automatically generated. Please review carefully.

Suggested change

	{ mime: 'video/webm', signatures: [[0x1a, 0x45, 0xdf, 0xa3]] },
	{
	mime: 'video/webm',
	signatures: [[0x1a, 0x45, 0xdf, 0xa3]],
	refine: (buf) => {
	// EBML header: find the DocType element (ID 0x42 0x82) and read its value.
	const head = buf.subarray(0, Math.min(buf.length, 64))
	for (let i = 4; i < head.length - 2; i++) {
	if (head[i] !== 0x42 || head[i + 1] !== 0x82) continue
	const len = head[i + 2] & 0x7f
	const docType = head.subarray(i + 3, i + 3 + len).toString('ascii')
	if (docType === 'webm') return 'video/webm'
	if (docType === 'matroska') return 'video/x-matroska'
	return undefined
	}
	return undefined
	},
	},

[61465]

Thanks @depthfirst-app — third catch fixed in de71f32.

You're right: 1a45dfa3 is the EBML envelope magic, which Matroska and WebM share. A client could declare video/webm on a WebM-only bucket and upload a .mkv (or any EBML container) — the magic-byte path labelled them all video/webm.

Refine now scans a 128-byte window at the head of the file for the DocType element 0x4282 0x84 "webm":

• 0x4282 — DocType element id (EBML spec)
• 0x84 — VInt length = 4
• "webm" — DocType value, exactly 4 bytes

Only an exact "webm" DocType returns video/webm. Matroska DocType (longer value) and EBML containers without any DocType return undefined, so detection falls back to generic handling at the caller rather than mislabelling.

Tests: 3 new (webm with DocType / matroska / bare EBML header), 62/62 unit still green.

[61465]

Thanks @depthfirst-app — fifth catch is a real one, confirmed and fixed in 73a0e66.

Verified the gap by grep:

• wrapWithSignatureDetection was only called from uploader.ts:703 (standard upload leg)
• onUploadFinish in tus/lifecycle.ts called headObject + completeUpload without any signature check
• onCreate validated the declared contentType string against bucket.allowed_mime_types but had no visibility into the bytes

So the attack chain you described is exact: declare image/jpeg on a JPEG-restricted bucket, stream GIF bytes through TUS, validateMimeType passes the string, onUploadFinish commits.

Fix shape, as you suggested — a shared helper so the two legs cannot diverge:

• New export detectMimeFromStoredObject(getRange) in file-signature.ts. Takes a closure so the TUS caller binds backend / bucket / key / version without pulling the backend-adapter type into the validator module. Reads via bytes=0-${SIGNATURE_PEEK_BYTES - 1} so the cost is one 4 KiB ranged read regardless of final object size.
• onUploadFinish now:
  1. Looks up allowed_mime_types on the bucket.
  2. If the bucket has an allowlist AND the upload metadata declared a contentType, calls detectMimeFromStoredObject with a getObject-bound range closure.
  3. On mimeFamiliesMatch(detected, declared) === false, best-effort deletes the stored bytes (so the policy-violating object does not persist) and throws ERRORS.InvalidMimeType.

Buckets without allowed_mime_types stay on the fast path — no ranged read, no extra latency. Buckets with an allowlist pay one extra 4 KiB read per TUS completion, which is the same cost the standard leg already pays on the first chunk.

8 new tests on the helper directly (70/70 on file-signature.test.ts):

• bytes=0-N range request shape
• PNG / GIF (the mime-type does not check uploaded files, only the filename #639 case) / WAV (string-chunk path) / Uint8Array chunks (S3-adapter path)
• undefined on empty + unknown bodies
• bounded reads under a multi-chunk body

Biome + prettier clean on the touched files. CI should pick up the push shortly.

[61465]

Pushed 0ebee38 closing two more gaps — one from the follow-up bot comment on d4ddb91, one from a NEXUS adversarial pre-push review on this fork before the next bot pass.

1. Missing-contentType bypass (your 0ebee38 finding at 15:13 UTC)

Confirmed by re-reading the two legs together: onCreate only runs validateMimeType when metadata?.contentType && bucket.allowed_mime_types?.length, and the previous onUploadFinish guard only entered the signature block when declaredContentType was present. A TUS client that omitted contentType passed both.

Fix: onUploadFinish now always loads the bucket, and runs the signature check whenever allowed_mime_types is non-empty — independent of whether the client sent a contentType. The two branches are:

• contentType declared: previous mimeFamiliesMatch(detected, declaredContentType) comparison (unchanged).
• contentType missing (new): fail closed. Undetectable signature rejects (unknown (no contentType declared)). Detected MIME outside allowed_mime_types rejects. Only an in-allowlist detected MIME passes.

2. Verification-path exception leaking an unverified object

Before this patch, if findBucket or detectMimeFromStoredObject threw, the outer try/catch in onUploadFinish re-threw without cleaning up the stored bytes. The client saw an error, but the backend retained a policy-violating object on an MIME-restricted bucket — a smaller leak than a bypass, but worth closing.

Fix: wrapped the verification block in its own try/catch that best-effort deletes the stored bytes on any error (including the explicit InvalidMimeType rejections already produced inside) before re-throwing. The extra delete on the rejection path is a no-op because that branch already deleted; the net effect is one cleanup regardless of which failure mode triggered it.

70/70 unit tests still green on file-signature.test.ts. Lint clean on the touched files.

Side-note on the design choices intentionally kept:

• detected === undefined && declaredContentType present → pass. This mirrors the standard upload leg (uploader.ts:720-721), which also treats "signature unknown" as "not enough evidence to reject". Only the no-contentType branch rejects on unknown, because there is no declared type to fall back on.
• The explicit allowed_mime_types.some((m) => mimeFamiliesMatch(detected, m)) check in the no-contentType branch uses the same family-match helper the standard leg uses, so a bucket that allows image/jpeg and receives a detected image/jpeg passes, but one that receives image/png rejects. The helper is strict about families — image/jpeg and image/png are not treated as equivalent.

[61465]

Pushed two follow-ups covering the last of the open review findings:

eabc98c — scope the TUS signature-verify try/catch to the policy-enforcing path

CI's Test / Unit on 45d9662 surfaced a regression in the two "tus upload completion" cases: the outer try/catch introduced in 0ebee38 wrapped the entire verification block including findBucket, so any error inside hit a backend.deleteObject call. The existing lifecycle test mocks are minimal and don't stub deleteObject because the happy-path tests exercise buckets without allowed_mime_types — a path that never needs cleanup.

Moved findBucket out of the try/catch and only entered it when bucket.allowed_mime_types?.length is truthy. The cleanup contract on MIME-restricted buckets is unchanged (any error inside still deletes before re-throwing); buckets without a policy take the fast path with no cleanup risk.

00ce86c — close the small-file race in wrapWithSignatureDetection (your 5:18 UTC finding)

Confirmed by re-reading the wrapper: for files under SIGNATURE_PEEK_BYTES the detected promise resolves only inside the source end handler, at which point out.end() has already fired to the backend. The void detected.then(...) destroy pattern in uploader.ts could not win that race.

Fix shape, following your suggestion:

• Added an optional options.validate(mime) closure to wrapWithSignatureDetection. It runs synchronously the moment the MIME is known.
• On a returned Error, the wrapper sets a terminalError flag that stops forwarding chunks, destroys the output stream in place of end(), and explicitly destroys the source so no further data events fire.
• uploader.ts migrated to wrapWithSignatureDetection(body, { validate: ... }). The async detected.then destroy call is gone — the sync validate closure now carries the enforcement. Behaviour on files larger than SIGNATURE_PEEK_BYTES is unchanged (sync validation fires mid-stream, same point the async callback would have reacted).

3 new unit tests cover the race:

• destroys output stream synchronously on validate() returning an error (small-file case)
• passes bytes through unchanged on validate() returning undefined
• backwards-compat: no options still works, detected promise still fulfills

73/73 unit tests on file-signature.test.ts green. Biome clean on all three touched files.

---

Thanks for the five catches on this PR. The sweep covered OOXML, OLE2, EBML DocType, TUS leg bypass + missing-contentType, and now the small-file race. The current branch should hold against the same adversarial read.