Skip to content

feat: export a generated 3D model as USDZ so it opens in AR on your phone - #5989

Merged
atomantic merged 1 commit into
mainfrom
claim/issue-5756
Sep 3, 2026
Merged

feat: export a generated 3D model as USDZ so it opens in AR on your phone#5989
atomantic merged 1 commit into
mainfrom
claim/issue-5756

Conversation

@atomantic

@atomantic atomantic commented Sep 3, 2026

Copy link
Copy Markdown
Owner

Summary

The 3D page could only ever spin a generated mesh inside a browser canvas — "Download .glb" and "Download full-resolution mesh (.obj)" were the only terminal steps. It now offers Export for AR (.usdz), the one format that turns a generated model into something you can place in your actual room at real scale, with no app install and nothing uploaded to a third party. On a desktop the same panel renders a QR code pointing at the install's own URL, which is the bridge from "I generated this at my desk" to "it is on my floor".

The conversion runs in the browser, via three's USDZExporter — already present in the installed three@0.185.1, so no new dependency and no USD toolchain. By the time the user can ask, the viewer has already parsed the GLB and decoded its textures; a server-side converter would have to redo all of that in Node to produce a file the page already holds in memory.

What shipped

  • POST / GET /api/image-to-3d/models/:id/usdz — store and re-serve the artifact as data/image-to-3d/<id>/model.usdz, a sibling of the GLB. Persisted rather than handed back as a blob URL: AR Quick Look does not reliably open a blob and a blob does not survive a reload, so revisiting a record reuses the stored export instead of re-converting.
  • The header contract is the feature. The GET is served inline as model/vnd.usdz+zip — Safari will not engage AR Quick Look on an attachment response. streamAttachment gains an opt-in disposition for this and keeps its existing disconnect/teardown behavior.
  • Uploaded bytes are treated as untrusted input (they come from the client): ready record, non-empty, under a 64 MB cap, and actually a zip archive (PK\x03\x04 — USDZ is a stored zip). The cap is enforced twice: at the express.raw parser, before the body is buffered, and in the service as the invariant for any other caller.
  • Stale exports are invalidated. A successful re-render clears both record.usdzPath and the file — it describes geometry the render just replaced. A failed render deliberately leaves it alone, since model.glb is untouched and the export still matches what the viewer shows.
  • The AR affordance is feature-detected, never user-agent sniffed. relList.supports('ar') decides; only Safari on iOS/iPadOS gets the <a rel="ar"> handoff (whose only child is an <img>, as Safari requires). Every other browser — including Chrome on the same iPhone — gets a plain download plus the QR bridge, so nobody is shown a "View in AR" button that cannot do anything. The probe is wrapped because DOMTokenList.supports() is specified to throw for an attribute with no supported-tokens list, and it runs during render.
  • GlbViewer gains onSceneLoaded, handing the displayed graph (post force-opaque clone) up to the page and retracting it on unload — the force-opaque cleanup disposes that clone's materials, so a retained handle would export a scene whose textures are gone.
  • client/src/lib/usdzExport.js (barrel + README registered) holds the pure half: lazy-loaded exporter, countSceneTriangles, supportsArQuickLook, and the budgets.

Decisions recorded

  • Size budget: warn, don't decimate. The issue floated "export a decimated variant". The GLB the exporter reads is already the decimated mesh (the million-face model.obj sidecar is never loaded here), and in-browser simplification would mean a new dependency or a hand-rolled mesh simplifier — out of proportion to the problem. Instead the export bounds textures at 1024px (USDZ stores every texel raw, with no Draco/meshopt equivalent — textures, not geometry, are what make a file unopenable on a phone) and, above AR_TRIANGLE_BUDGET, names the triangle count and points at a lower Quality tier rather than silently shipping a file that opens to an empty room. Nothing fails silently, which was the requirement.
  • Backup tier: keep it. No DEFAULT_EXCLUDES entry added. The artifact is a few megabytes, and re-deriving it needs a browser session with the model open — cheaper to keep than to reproduce, exactly the rationale already recorded for the published rig/ pair. (The gigabyte model.obj sidecar stays excluded.)
  • Delete lifecycle. The USDZ lives inside recordDir(id), so it inherits the GLB's lifecycle exactly: removed wherever cleanupRenderDir runs, retained wherever the GLB is. No new, divergent cleanup path was invented.

Test plan

  • cd server && npm test — 1888 files / 38080 tests pass. New coverage: the route accepts a raw body past the app-wide JSON parser and serves inline + model/vnd.usdz+zip; the service refuses a non-zip, empty, oversized, and not-yet-rendered payload, 404s a never-exported record distinctly from a missing one, and clears a stale export when a re-render succeeds.
  • cd client && npm test — 826 files / 10107 tests pass. New coverage: the panel is hidden until a mesh exists and disabled until the scene loads; export → upload → record update; the budget warning names the count; a conversion failure persists nothing; a stored export is reused rather than re-run; and the AR anchor renders only where Quick Look exists, with the QR + download path everywhere else. GlbViewer pins the onSceneLoaded hand-off and its retraction; countSceneTriangles is covered across indexed, non-indexed, and non-mesh children.
  • npm run build passes; server/lib/apiRouteCatalog.generated.json regenerated for the two new operations.
  • Not exercised here: the AR Quick Look handoff itself needs a physical iPhone/iPad, so the header pair and anchor shape are pinned by assertion rather than by a device run.

Closes #5756

https://claude.ai/code/session_01Uz59AsJawa1Djqm2T9Fb8t

…hone (#5756)

The 3D page could only ever spin a generated mesh inside a browser canvas.
It now offers "Export for AR (.usdz)", which converts the model you are
looking at into the one format that drops it into your actual room at real
scale — tap and it is on your floor, no app install and no upload to anyone
else. On a desktop the same panel shows a QR code pointing at the install's
own URL, so a model generated at your desk reaches your phone by scanning it.

The conversion runs in the browser via three's USDZExporter, which is already
in the installed three.js — no new dependency, no USD toolchain, and no
re-decoding a GLB the viewer has already parsed. The bytes are persisted as a
sibling artifact of the record rather than handed back as a blob URL (AR Quick
Look does not reliably open a blob, and a blob does not survive a reload), so
revisiting a model reuses the stored export instead of re-converting it.

- POST/GET /api/image-to-3d/models/:id/usdz store and re-serve the artifact.
  The GET is served `inline` as `model/vnd.usdz+zip` — Safari will not engage
  AR Quick Look without that exact header pair — via streamAttachment, which
  gains an opt-in `disposition` for it.
- Uploaded bytes are validated as untrusted input: ready record, non-empty,
  under a 64 MB cap, and actually a zip archive.
- A successful re-render clears the stored export and its record pointer: the
  file would describe geometry the render just replaced. A FAILED render
  leaves it alone, since model.glb is untouched and the export still matches.
- The AR affordance is feature-detected, never user-agent sniffed. Only Safari
  on iOS/iPadOS gets the `<a rel="ar">` handoff; everywhere else the panel
  offers a plain download plus the QR bridge, so no browser is shown a "View
  in AR" button that cannot do anything.
- Meshes above the AR triangle budget still export, but say so and name the
  count, pointing at a lower Quality tier rather than shipping a file that
  opens to an empty room.

Claude-Session: https://claude.ai/code/session_01Uz59AsJawa1Djqm2T9Fb8t
@atomantic
atomantic merged commit 788817e into main Sep 3, 2026
7 checks passed
@atomantic
atomantic deleted the claim/issue-5756 branch September 3, 2026 04:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[repo-study-remesher-usdz-ar-export] Export a generated 3D model as USDZ so it opens in AR on the user's phone

1 participant