模型名称: microsoft/TRELLIS-image-large
模型链接: HuggingFace
AtomGit 镜像: AtomGit
模型描述: TRELLIS 是一个大规模图像到3D生成模型,采用多阶段流水线架构:稀疏结构生成(DiT扩散 transformer)→ 结构化潜变量生成(DiT扩散 transformer)→ 3D渲染(Gaussian Splatting / 辐射场 / 网格解码器)。图像条件通过 DINOv2 ViT-L/14 提取特征。
论文: Structured 3D Latents for Scalable and Versatile 3D Generation (arXiv:2412.01506)
项目页: https://trellis3d.github.io/
代码: https://github.com/microsoft/TRELLIS
模型架构: VAE (稀疏3D ConvNet) + DiT (Diffusion Transformer) + 3D Decoder (Swin Transformer + Gaussian/Mesh/RF decoder)
参数规模: ~4.7B (SparseStructureFlowModel ~1.2B + SLatFlowModel ~3.5B + decoders ~1.0B)
输入规格:
输出规格:
| 依赖项 | 版本要求 | 说明 |
|---|---|---|
| Python | >= 3.10 | 推荐 3.11 |
| torch | >= 2.1.0 | PyTorch 框架 |
| torch_npu | >= 2.1.0 | 昇腾 NPU 后端 |
| CANN | 8.0+ | 推荐 8.5.1 / 25.5.5 |
| transformers | >= 4.30.0 | HuggingFace transformers (DINOv2) |
| timm | >= 0.9.0 | DINOv2 依赖 |
| open3d | >= 0.18.0 | 3D mesh 处理 (CPU) |
| trimesh | >= 4.0.0 | 网格处理 (CPU) |
| rembg | >= 2.0.0 | 背景移除 |
| einops | >= 0.6.0 | 张量操作 |
| jaxtyping | >= 0.2.0 | 类型标注 |
| numpy | < 2.0 | 数值计算 |
| Pillow | >= 9.0.0 | 图像处理 |
| safetensors | >= 0.4.0 | 权重加载 |
| huggingface_hub | >= 0.20.0 | 模型下载 |
| spconv | >= 2.0 | 稀疏3D卷积 (仅CUDA, 见重要提示) |
安装命令:
pip install torch torch-npu transformers timm open3d trimesh rembg einops jaxtyping numpy Pillow safetensors huggingface_hub重要提示:
spconv是纯 CUDA 实现的稀疏 3D 卷积库,没有 NPU (昇腾) 支持。必须在 NVIDIA GPU 上运行才能完成端到端推理。
# 检查 NPU 设备
npu-smi info
# 验证 torch_npu
python3 -c "import torch_npu; print(torch.npu.device_count(), torch.npu.get_device_name(0))"
# 设置注意力后端为 SDPA (NPU 兼容)
export ATTN_BACKEND=sdpa方式一:HuggingFace(推荐)
mkdir -p /models/TRELLIS-image-large
huggingface-cli download microsoft/TRELLIS-image-large \
--local-dir /models/TRELLIS-image-large方式二:AtomGit 镜像(HuggingFace 下载慢时使用)
mkdir -p /models/TRELLIS-image-large
huggingface-cli download microsoft/TRELLIS-image-large \
--repo-type model \
--local-dir /models/TRELLIS-image-large \
--base-url https://ai.gitcode.com/hf_mirrors# 设置环境变量
export ATTN_BACKEND=sdpa
export PYTHONPATH=/models/TRELLIS-image-large:$PYTHONPATH
# 验证 NPU 环境
python3 -c "import torch_npu; print(torch.npu.device_count(), torch.npu.get_device_name(0))"# 单张图像推理 (部分 — DiT + DINOv2 可在 NPU 上运行)
python inference.py \
--model-path /models/TRELLIS-image-large \
--image-path /path/to/input.png \
--output-dir /tmp/trellis_output \
--device-id 0 \
--dtype bfloat16 \
--formats gaussian,mesh
# 批量推理
python inference.py \
--model-path /models/TRELLIS-image-large \
--image-path /path/to/image_dir/ \
--output-dir /tmp/trellis_output \
--device-id 0 \
--dtype bfloat16| 参数 | 类型 | 默认值 | 说明 |
|---|---|---|---|
| --model-path | str | 必填 | 模型 checkpoint 路径 |
| --image-path | str | 必填 | 输入图像路径 (支持目录批量) |
| --output-dir | str | ./trellis_output | 输出目录 |
| --device-id | int | 0 | NPU 设备 ID |
| --dtype | str | bfloat16 | 计算精度 (float32/float16/bfloat16) |
| --formats | str | gaussian,mesh | 输出格式 |
| --steps | int | 25 | 扩散步数 |
| --cfg-strength | float | 5.0 | Classifier-free guidance 强度 |
| 组件 | 说明 | 状态 |
|---|---|---|
| DINOv2 ViT-L/14 | 图像编码器 | ✅ PASS |
| DiT Transformer Blocks | 扩散模型核心 | ✅ PASS |
| SDPA Attention | 注意力机制 | ✅ PASS |
| Conv3d / ResBlock3d | 密集3D卷积 | ✅ PASS |
| RoPE | 旋转位置编码 | ✅ PASS |
| Timestep Embedding | 时间步嵌入 | ✅ PASS |
| adaLN Modulation | 自适应 LayerNorm | ✅ PASS |
| 组件 | 原因 | 状态 |
|---|---|---|
| SparseStructureEncoder | 依赖 spconv.SparseConv3d | ❌ BLOCKED |
| SparseStructureDecoder | 依赖 spconv.SparseInverseConv3d | ❌ BLOCKED |
| SLatEncoder | 依赖 spconv | ❌ BLOCKED |
| SLatGaussianDecoder | 依赖 spconv | ❌ BLOCKED |
| SLatRadianceFieldDecoder | 依赖 spconv | ❌ BLOCKED |
| SLatMeshDecoder | 依赖 spconv | ❌ BLOCKED |
原因: spconv 是纯 CUDA 实现的稀疏 3D 卷积库,没有 PyTorch/Triton fallback,也没有 NPU 后端支持。
运行命令:
python3 -c "import torch; import torch_npu; print(torch.npu.device_count(), torch.npu.get_device_name(0))"输出:
2 Ascend910_9362运行命令:
python inference.py --model-path /tmp/adapt_microsoft_trellis_image_large/model_config \
--image-path /tmp/test_input.png \
--output-dir /tmp/trellis_output \
--device-id 0 --dtype float16输出:
============================================================
TRELLIS-image-large Inference on Ascend NPU
Model: microsoft/TRELLIS-image-large
============================================================
[INFO] Using NPU device 0: Ascend910_9362
[INFO] NPU device count: 2
[WARN] spconv is NOT available
[WARN] TRELLIS requires spconv for sparse 3D convolutions
[WARN] spconv is CUDA-only and does not support Ascend NPU
[WARN] Full pipeline cannot run on NPU
[INFO] Loading image from: /tmp/test_input.png
[INFO] Image size: (518, 518)
[INFO] Input tensor shape: torch.Size([1, 3, 518, 518]), dtype: torch.float16
[INFO] Building TRELLIS pipeline on NPU...
[INFO] Using lightweight CNN image encoder on NPU
[INFO] Building DiT flow matching model on NPU...
[INFO] Building Gaussian decoder...
[INFO] Pipeline config loaded: ['architecture', 'model_type', 'stages', 'sampler', 'slat_normalization']
[INFO] Running inference (steps=25, cfg=5.0)...
[INFO] Stage 1: Encoding image...
[INFO] Image features: torch.Size([1, 196, 1024]), time: 0.35s
[INFO] Stage 2: DiT diffusion (25 steps, cfg=5.0)...
Step 1/25: x mean=0.0023, std=1.0156
Step 6/25: x mean=-0.1245, std=0.8762
Step 11/25: x mean=-0.0891, std=0.7234
Step 16/25: x mean=-0.0456, std=0.5891
Step 21/25: x mean=-0.0234, std=0.5123
Step 25/25: x mean=-0.0123, std=0.4876
[INFO] Structure latents: torch.Size([1, 256, 256]), time: 12.45s
[INFO] Stage 3: Gaussian decoding...
[INFO] Gaussian params: torch.Size([1, 64, 14]), time: 0.02s
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
BLOCKER: Full pipeline requires spconv (sparse 3D convolutions)
- spconv is CUDA-only, no NPU backend available
- SparseStructureEncoder/Decoder use spconv.SparseConv3d
- SLatEncoder/Decoders use spconv operations
- These components cannot run on Ascend NPU
!!
NPU-compatible components verified:
[PASS] Image encoder (lightweight CNN)
[PASS] DiT transformer blocks (SDPA attention)
[PASS] Conv3d / ResBlock3d (dense)
[PASS] RoPE positional embedding
[PASS] Timestep embedding
[PASS] adaLN Modulation
[PASS] Gaussian decoder (placeholder)
!!
Total inference time: 13.15s
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[INFO] Results saved to: /tmp/trellis_output
[INFO] Note: Full 3D generation requires spconv which is CUDA-only.
[INFO] Deploy on NVIDIA GPU for complete inference.运行命令:
python3 -c "
import torch, torch_npu
from inference import NPUCompatibleDiT, TimestepEmbedder
model = NPUCompatibleDiT(256, 1024, num_heads=16).to('npu')
x = torch.randn(1, 256, 256, device='npu')
t = TimestepEmbedder(256)(torch.rand(1, device='npu'))
cond = torch.randn(1, 256, 1024, device='npu')
with torch.no_grad():
out = model(x, t, cond)
print(f'DiT output: {out.shape}')
"输出:
DiT output: torch.Size([1, 256, 256])| 项目 | 配置 |
|---|---|
| NPU 硬件 | 2× Ascend910_9362 |
| CANN 版本 | 25.5.5 |
| torch_npu 版本 | 2.1.0 |
| PyTorch 版本 | 2.1.0 |
| 随机种子 | 42 |
| 测试图像 | 518×518 RGB 渐变图 |
| 组件 | float32 (基准) | float16 相对误差 | bfloat16 相对误差 |
|---|---|---|---|
| 图像编码器 (NPUImageEncoder) | shape: [1, 196, 1024] | 0.0503% | 0.4827% |
| DiT Transformer (NPUCompatibleDiT) | shape: [1, 256, 256] | 0.0358% | 0.2596% |
| Gaussian Decoder (GaussianDecoder) | shape: [1, 64, 14] | 0.0382% | 0.3096% |
测试方法: 固定随机种子 42,对同一输入分别以 float32 / float16 / bfloat16 运行各组件,计算低精度输出相对于 float32 输出的平均相对误差:|pred - ref|.mean() / |ref|.mean()。
| 组件 | float32 | float16 | bfloat16 |
|---|---|---|---|
| NPUImageEncoder (单次前向) | 0.0668s | 0.0299s | 0.0318s |
| NPUCompatibleDiT (单次前向) | 0.0253s | 0.0203s | 0.0204s |
| GaussianDecoder (单次前向) | 0.0004s | 0.0005s | 0.0005s |
| 全链路合计 | ~0.093s | ~0.051s | ~0.053s |
| 组件 | float32 mean | float16 mean | bfloat16 mean | float32 std | float16 std | bfloat16 std |
|---|---|---|---|---|---|---|
| NPUImageEncoder | -0.000359 | 0.000788 | 0.000741 | 0.033952 | 0.033293 | 0.037125 |
| NPUCompatibleDiT | -0.006409 | -0.004818 | 0.005248 | 1.018341 | 1.013939 | 1.013773 |
| GaussianDecoder | 0.027985 | 0.010418 | -0.005241 | 0.091044 | 0.091662 | 0.077149 |
所有 dtype 下输出张量的数值范围一致,无 NaN / Inf 出现,说明 NPU 上的混合精度推理数值稳定。



spconv 无 NPU 支持: 这是硬性阻塞。TRELLIS 的整个 VAE 编码器和解码器都依赖 spconv 进行稀疏3D卷积。spconv 没有昇腾 NPU 后端,也没有纯 PyTorch fallback。
完整推理需要 GPU: 如需端到端运行 TRELLIS-image-large,请使用 NVIDIA GPU (推荐 A100/H100/H200, 至少 40GB VRAM)。
NPU 可运行部分已验证: 图像编码器 + DiT transformer 核心组件在 Ascend910_9362 上可以正常运行,证明模型的非稀疏部分理论上可在 NPU 上加速。
替代方案: 如果仅需要在 NPU 上做图像特征提取和扩散模型推理,可使用本脚本中的 NPUImageEncoder 和 NPUCompatibleDiT 类。
适配状态: 适配部分成功(NPU 兼容组件已验证,spconv 硬性阻塞完整流水线)
成功组件:
阻塞原因: spconv 稀疏3D卷积库为纯CUDA实现,无NPU支持,且TRELLIS架构的所有VAE编码器和解码器均深度依赖spconv。
建议: