trellis2: macOS/Apple Silicon path — trellis-mac (MPS) sidecar now, Metal/MoltenVK contribution to trellis.cpp later

[fernandotonon]

Follow-up to the TRELLIS.2 backend (ai/trellis2/) and to #966 (trellis.cpp runtime flavor).

Why

Neither the Python/CUDA sidecar nor trellis.cpp (CUDA/Vulkan/ROCm, no Metal) runs TRELLIS.2 on macOS today, but Macs are a first-class QtMeshEditor platform.

Near term — trellis-mac MPS sidecar env

shivampkumar/trellis-mac (MIT) runs the full TRELLIS.2 pipeline on PyTorch MPS with Metal ports of the sparse kernels — reportedly ~5 min end-to-end on an M4 Pro 24 GB with full base-color/metallic/roughness output.

• Extend ai/trellis2/install.py with a macOS flavor that provisions the trellis-mac fork (pin a revision; apply the same o_voxel.__init__ lazy-postprocess patch if applicable) — the existing generate.py/QTM3D contract should carry over nearly unchanged (inference only; QtMeshEditor keeps the native bake).
• Same license traps, same fixes: DINOv3 is gated (user's own HF login), RMBG-2.0 (CC-BY-NC) stays bypassed via our U²-Net keepAlpha matte. Re-audit trellis-mac's diffs in docs/trellis2-dependencies.md.
• Verify output parity/orientation vs the CUDA reference on a few fixtures.

Longer term — Metal on trellis.cpp

Options being evaluated (feasibility research in progress; results to be attached here):

1. Run trellis.cpp's Vulkan backend via MoltenVK — cheapest if their shaders avoid MoltenVK-unsupported features (e.g. cooperative matrix).
2. Enable GGML's Metal backend for the dense ops and let custom sparse ops fall back to CPU via ggml_backend_sched — hybrid, incremental.
3. Full Metal kernels for the sparse ops, potentially adapting trellis-mac's MIT Metal sparse-conv work — most effort, best result; would be contributed upstream to pwilkin/trellis.cpp.

• Attach the feasibility findings + pick the path
• Upstream contribution PR(s) as needed

🤖 Generated with Claude Code

[fernandotonon]

Metal feasibility research results (trellis.cpp)

Verified by source inspection of a clone of pwilkin/trellis.cpp (pushed 2026-08-21) + its pinned pwilkin/ggml fork.

Decisive finding: there are no custom sparse GPU kernels to port. trellis.cpp implements TRELLIS.2's sparse conv as host-built neighbor tables + stock ggml_get_rows/ggml_mul_mat (docs/spec/22-sparse_conv.md), and sparse attention as ggml_flash_attn_ext with zero-padded K/V (head dims 64/128). The only custom GPU kernels in the whole repo are deformable conv2d (BiRefNet preprocessing) and QEM decimation — both with existing CPU fallbacks. The entire mesh postprocess (dual contouring, xatlas, texture bake, meshoptimizer) is pure CPU. The vendored ggml fork's patches are CUDA-only, so its Metal backend is current upstream and untouched, and make_backend()'s non-CUDA path enumerates ggml_backend_dev_* generically — a Metal device would be discovered with essentially no dispatch changes. Op audit against ggml-metal-device.m::supports_op: every op they use (conv_2d/3d, flash_attn_ext hd64/128, set_rows/get_rows, pad, gelu_erf, step, concat, …) is Metal-supported.

Recommended path — contribute a ggml-Metal enable upstream, not our own implementation:

1. 1–3 day MoltenVK spike first: their custom shaders need only core Vulkan 1.2, and ggml-vulkan has explicit MoltenVK code paths — likely produces a working (slow) binary to validate numerics on Apple Silicon. Known caveats: no coopmat (slow matmuls), no bfloat16 (use TRELLIS_FA_FAST=1 F16 K/V).
2. Real backend: GGML_METAL=ON wiring + CPU fallbacks for the two custom kernels — est. 1–2 weeks to first mesh, 2–4 weeks robust. Risks: (i) >2^31-element tensors in the sparse stage (the maintainer patched CUDA for exactly this; Metal will likely need the same 64-bit-indexing round or tighter tiling), (ii) memory — 16.5 GB weights + multi-GB activations → realistically 48–64 GB unified memory for large scenes, and Metal maxBufferLength may force buffer splitting, (iii) M1 lacks bfloat → default F16 FA there. Note: trellis.cpp uses no ggml_backend_sched, so unsupported ops fail rather than fall back — adding sched or per-op guards is cheap insurance.
3. Optional follow-up PR: Metal port of the 6.6 KB QEM decimation shader if postprocess profiling justifies it.

Upstream receptiveness: no CONTRIBUTING.md but a friendly merge record — all visible PRs are external, 3 merged, and an open PR adds Windows ARM64 + Qualcomm Hexagon NPU support, i.e. new-platform PRs are entertained. No Metal/macOS issue exists upstream yet — we should open one there before starting.

trellis-mac reuse: low — it's a PyTorch-MPS port (mtlgemm sparse GEMM, mtldiffrast) solving problems trellis.cpp already solved differently; at most MSL reference material. It remains the right near-term Mac sidecar while the Metal backend lands.

🤖 Generated with Claude Code

[fernandotonon]

Upstream PRs are open: pwilkin/trellis.cpp#44 (macOS/Apple Silicon via ggml Metal — CMake enable + conv_3d_direct fix for the missing IM2COL_3D op + README/measurements: M5, res 512, 560.9 s, 5.6 GB peak, all stages on Metal) and pwilkin/trellis.cpp#45 (--dump-post raw mesh + PBR volume output, the QtMeshEditor handoff). Both branches build independently; the fork's macos-metal-support remains the combined working branch QtMeshEditor points at.

🤖 Generated with Claude Code