feat: export a generated 3D model as USDZ so it opens in AR on your phone - #5989
Merged
Conversation
…hone (#5756) The 3D page could only ever spin a generated mesh inside a browser canvas. It now offers "Export for AR (.usdz)", which converts the model you are looking at into the one format that drops it into your actual room at real scale — tap and it is on your floor, no app install and no upload to anyone else. On a desktop the same panel shows a QR code pointing at the install's own URL, so a model generated at your desk reaches your phone by scanning it. The conversion runs in the browser via three's USDZExporter, which is already in the installed three.js — no new dependency, no USD toolchain, and no re-decoding a GLB the viewer has already parsed. The bytes are persisted as a sibling artifact of the record rather than handed back as a blob URL (AR Quick Look does not reliably open a blob, and a blob does not survive a reload), so revisiting a model reuses the stored export instead of re-converting it. - POST/GET /api/image-to-3d/models/:id/usdz store and re-serve the artifact. The GET is served `inline` as `model/vnd.usdz+zip` — Safari will not engage AR Quick Look without that exact header pair — via streamAttachment, which gains an opt-in `disposition` for it. - Uploaded bytes are validated as untrusted input: ready record, non-empty, under a 64 MB cap, and actually a zip archive. - A successful re-render clears the stored export and its record pointer: the file would describe geometry the render just replaced. A FAILED render leaves it alone, since model.glb is untouched and the export still matches. - The AR affordance is feature-detected, never user-agent sniffed. Only Safari on iOS/iPadOS gets the `<a rel="ar">` handoff; everywhere else the panel offers a plain download plus the QR bridge, so no browser is shown a "View in AR" button that cannot do anything. - Meshes above the AR triangle budget still export, but say so and name the count, pointing at a lower Quality tier rather than shipping a file that opens to an empty room. Claude-Session: https://claude.ai/code/session_01Uz59AsJawa1Djqm2T9Fb8t
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The 3D page could only ever spin a generated mesh inside a browser canvas — "Download .glb" and "Download full-resolution mesh (.obj)" were the only terminal steps. It now offers Export for AR (.usdz), the one format that turns a generated model into something you can place in your actual room at real scale, with no app install and nothing uploaded to a third party. On a desktop the same panel renders a QR code pointing at the install's own URL, which is the bridge from "I generated this at my desk" to "it is on my floor".
The conversion runs in the browser, via three's
USDZExporter— already present in the installedthree@0.185.1, so no new dependency and no USD toolchain. By the time the user can ask, the viewer has already parsed the GLB and decoded its textures; a server-side converter would have to redo all of that in Node to produce a file the page already holds in memory.What shipped
POST/GET /api/image-to-3d/models/:id/usdz— store and re-serve the artifact asdata/image-to-3d/<id>/model.usdz, a sibling of the GLB. Persisted rather than handed back as a blob URL: AR Quick Look does not reliably open a blob and a blob does not survive a reload, so revisiting a record reuses the stored export instead of re-converting.GETis servedinlineasmodel/vnd.usdz+zip— Safari will not engage AR Quick Look on anattachmentresponse.streamAttachmentgains an opt-indispositionfor this and keeps its existing disconnect/teardown behavior.PK\x03\x04— USDZ is a stored zip). The cap is enforced twice: at theexpress.rawparser, before the body is buffered, and in the service as the invariant for any other caller.record.usdzPathand the file — it describes geometry the render just replaced. A failed render deliberately leaves it alone, sincemodel.glbis untouched and the export still matches what the viewer shows.relList.supports('ar')decides; only Safari on iOS/iPadOS gets the<a rel="ar">handoff (whose only child is an<img>, as Safari requires). Every other browser — including Chrome on the same iPhone — gets a plain download plus the QR bridge, so nobody is shown a "View in AR" button that cannot do anything. The probe is wrapped becauseDOMTokenList.supports()is specified to throw for an attribute with no supported-tokens list, and it runs during render.GlbViewergainsonSceneLoaded, handing the displayed graph (post force-opaque clone) up to the page and retracting it on unload — the force-opaque cleanup disposes that clone's materials, so a retained handle would export a scene whose textures are gone.client/src/lib/usdzExport.js(barrel + README registered) holds the pure half: lazy-loaded exporter,countSceneTriangles,supportsArQuickLook, and the budgets.Decisions recorded
model.objsidecar is never loaded here), and in-browser simplification would mean a new dependency or a hand-rolled mesh simplifier — out of proportion to the problem. Instead the export bounds textures at 1024px (USDZ stores every texel raw, with no Draco/meshopt equivalent — textures, not geometry, are what make a file unopenable on a phone) and, aboveAR_TRIANGLE_BUDGET, names the triangle count and points at a lower Quality tier rather than silently shipping a file that opens to an empty room. Nothing fails silently, which was the requirement.DEFAULT_EXCLUDESentry added. The artifact is a few megabytes, and re-deriving it needs a browser session with the model open — cheaper to keep than to reproduce, exactly the rationale already recorded for the publishedrig/pair. (The gigabytemodel.objsidecar stays excluded.)recordDir(id), so it inherits the GLB's lifecycle exactly: removed wherevercleanupRenderDirruns, retained wherever the GLB is. No new, divergent cleanup path was invented.Test plan
cd server && npm test— 1888 files / 38080 tests pass. New coverage: the route accepts a raw body past the app-wide JSON parser and servesinline+model/vnd.usdz+zip; the service refuses a non-zip, empty, oversized, and not-yet-rendered payload, 404s a never-exported record distinctly from a missing one, and clears a stale export when a re-render succeeds.cd client && npm test— 826 files / 10107 tests pass. New coverage: the panel is hidden until a mesh exists and disabled until the scene loads; export → upload → record update; the budget warning names the count; a conversion failure persists nothing; a stored export is reused rather than re-run; and the AR anchor renders only where Quick Look exists, with the QR + download path everywhere else.GlbViewerpins theonSceneLoadedhand-off and its retraction;countSceneTrianglesis covered across indexed, non-indexed, and non-mesh children.npm run buildpasses;server/lib/apiRouteCatalog.generated.jsonregenerated for the two new operations.Closes #5756
https://claude.ai/code/session_01Uz59AsJawa1Djqm2T9Fb8t