trellis2: add trellis.cpp as a second runtime flavor (Windows/Linux, Vulkan/CUDA/ROCm — no Python)

[fernandotonon]

Follow-up to the TRELLIS.2 backend integration (Python sidecar, ai/trellis2/, audit in docs/trellis2-dependencies.md).

Why

The Python sidecar runs upstream-exact TRELLIS.2 but requires Linux + CUDA + a Python env. pwilkin/trellis.cpp (MIT, C++/GGML) runs the same pipeline on CUDA + Vulkan + ROCm with a CPU fallback, on Windows and Linux, with quantized GGUF weights and no Python — a much better multiplatform story for QtMeshEditor. (Wimacs/trellis2.c is the MIT C alternative; watching it, but trellis.cpp has more traction + per-stage numerical validation vs PyTorch.)

Plan

• Trellis2Predictor: second runtime flavor — detect a trellis-cli binary (env QTMESH_TRELLIS2_CLI / QSettings ai/trellis2Cli / PATH) alongside the existing Python env; detection order: trellis.cpp → Python sidecar.
• Upstream PR (the key enabler): an option for trellis.cpp to dump the RAW generation — vertices/faces + the sparse PBR attribute volume — instead of (or in addition to) its own baked GLB, so QtMeshEditor keeps ownership of game-ready processing, UVs, PBR baking, and the preserved re-bakeable source (same QTM3D flow as the Python sidecar). Without it we'd have to ingest their finished GLB and lose Phases 5–9.
• License audit of the GGUF weight pack (ilintar/trellis2-gguf): confirms the bundled DINOv3 conversion complies with Meta's DINOv3 License (redistribution-with-conditions) and that the background-removal model is the MIT BiRefNet, not CC-BY-NC RMBG-2.0. Extend docs/trellis2-dependencies.md.
• Same UI/CLI/MCP surface (--backend trellis2 unchanged); progress mapping from trellis.cpp output; cancellation via process kill.
• scripts/check-trellis2-restricted-deps.sh + Trellis2GuardTest still apply (trellis.cpp is nvdiffrast/nvdiffrec-free by design).

Non-goals

Replacing the Python sidecar (it stays as the upstream-exact reference runtime) or macOS support (tracked separately).

🤖 Generated with Claude Code

[fernandotonon]

Progress: the fork (fernandotonon/trellis.cpp, branch macos-metal-support) builds clean on macOS/Apple M5 with the ggml Metal backend enabled and now has the key enabler — a --dump-post PATH flag that emits the raw decoded mesh + sparse PBR volume (the existing TRELLIS_DUMP_POST debug layout) and exits before remesh/UV/bake. QtMeshEditor side is implemented: Trellis2Predictor gained the RuntimeKind::TrellisCpp flavor (env QTMESH_TRELLIS2_CLI(+_MODELS) / QSettings ai/trellis2Cli(+Models) / PATH; preferred over the Python sidecar), parses the [k/7] stage lines for progress, imports the dump via Trellis2Interchange::readTrellisCppDump, and runs the same native game-ready + bake pipeline. 25/25 unit tests green. GGUF weight-pack audit confirmed by the trellis.cpp README: conditioner is the UNGATED timm DINOv3 mirror and bg-removal is MIT ZhengPeng7/BiRefNet (not RMBG-2.0). E2E generation on the M5 pending the ~10 GB weight download.

🤖 Generated with Claude Code

[fernandotonon]

Milestone: full TRELLIS.2 → QtMeshEditor pipeline verified on Apple Silicon Metal (first known end-to-end TRELLIS.2 run on a Mac).

Numbers (Apple M5, 24 GB, res 512, seed 42):

• trellis.cpp inference (all stages on Metal): image → 4.86M-tri raw mesh + 2.43M-voxel PBR volume in ~9.3 min, peak RSS 5.6 GB
• QtMeshEditor native half (via --dump-post + the new QTMESH_TRELLIS2_IMPORT re-bake hook): weld → simplify 4.86M → 11,784 tris → xatlas + 2905px 4-channel bake in 27.5 s, peak RSS 1.0 GB

Fixes that fell out of the first real-density run (all committed):

1. trellis.cpp fork: ggml_conv_3d lowers to IM2COL_3D which ggml-metal lacks → SS decoder uses ggml_conv_3d_direct on __APPLE__ (Metal implements CONV_3D directly).
2. TriangleGrid: density-scaled cells (fixed-64 grid put hundreds of tris per cell at 4.8M tris) + true O(r²) shell iteration; made closest() const/thread-safe.
3. Trellis2Bake::bake: two-pass parallel texel sampling (serial rasterize → all-cores sample/write).
4. makeGameReady: voxel-scale weld (voxelSize/8, the upstream reference weld — raw dual-grid is non-manifold seam soup, QEM was stuck at 2.67M tris) + uncapped second QEM pass + meshopt_simplifySloppy as guaranteed-budget fallback.
5. SparseVolumeSampler: alpha-weighted trilinear + coverage gate + deep opaque-only nearest fallback — TRELLIS wraps surfaces in an alpha≈0.007 skin (up to ~3-5 voxels thick in patches) whose black values a naive trilinear mixes in (measured 37% black texels → 0.2% after).

Known follow-up: bake the detail normal from a lightly remeshed source (raw dual-grid normals are noisy — upstream's narrow-band DC remesh exists for this); optionally parallelize bakeDetailNormal the same way as bake.

🤖 Generated with Claude Code

[fernandotonon]

Render-quality round complete — the game-ready TRELLIS.2 asset now renders clean (solid correct color, no holes, no seam speckle). Root causes found & fixed, in order of discovery:

1. Holes/shattered look — raw dual-grid output ships large flipped-winding patches (~17% conflicting directed edges measured); invisible under backface culling AND they confused QEM (which is why the sloppy fallback was needed). New Trellis2Bake::unifyWinding (BFS orientation propagation + outward signed-volume per component, unit-tested) — after it, plain QEM hits the budget exactly (4.86M → 14,938 tris at err 0.001).
2. Smooth shading normals — BakeResult/Result now carry the bake's position-welded normals; MeshGenBuilder uses them instead of recomputing from the seam-split index buffer.
3. Mip-safe atlas background — after border dilation the remaining background is flood-filled with the mesh's average baked color (thousands of small charts on this topology made the old neutral background bleed at higher mip levels).
4. THE white-speckle killer: baked diffuse alpha — low-alpha rim texels (TRELLIS's transparent skin) alpha-blended at render time and flashed bright in every viewer. Game-ready bakes are opaque by design → diffuse alpha is now 255.
5. GUI runtime discovery: trellis.cpp paths persisted via QSettings ai/trellis2Cli / ai/trellis2CliModels so the app detects the runtime without env vars (the "needs Python" hint disappears; the hint text itself now describes both flavors).

Also added BakeOptions::sourceNormalSmoothIterations (Laplacian smoothing of the source normal field before the detail bake — kills voxel-scale glitter in the normal map). 26/26 tests green.

🤖 Generated with Claude Code