Model: GEAR-Dreams/DreamZero-DROID (revision 96ad344138c66e82536422432ad742f015784942) — 14B World Action Model built on Wan2.1-I2V-14B-480P, trained exclusively on DROID (~75k episodes).
Route: manipulation-vla | Hardware: Ascend 910B (910_9362) npu:0 + npu:1 available, single-card demonstration with distributed requirement documented.
DreamZero 是 World Action Model,联合预测未来视频帧与机器人动作,实现零样本策略。输入为 3 相机 RGB (exterior_image_1_left, exterior_image_2_left, wrist_image_left) + 语言指令 + proprioceptive state (joint/gripper/cartesian),输出为连续动作块 action_chunk [batch, horizon=24, dim=32] (relative end-effector + gripper),同时生成未来视频。
video.exterior_image_1_left [B,3,176,320], video.exterior_image_2_left, video.wrist_image_left (bf16, npu:0), state.proprio [B,64] (≤ max_state_dim 64), annotation.language.action_text 字符串。配置 num_frames=33, num_frame_per_block=2, tile 34x34 stride 18/16 tiled=false.action_chunk [B,24,32] float32 npu:0, horizon 24, dim 32, 坐标系 relative, 有限值, bounds [-1,1] 归一化后反归一化到 cartesian/joint limits。Diffusion 头 num_inference_timesteps=4, beta(1.5,1.0) flow-matching。model.safetensors.index.json total_size 45848344232) 占单卡 64GB HBM 66.7%。torch.distributed 2 proc, DiT cache, ~3s H100)。本仓在 npu:0 上完成真实 NPU 张量与同步计时验证,完整 14B video+action 联合生成需 2×NPU 切分,已在日志与推理中明确记录。torch.npu.is_available()==True, device_count=2, device_name=Ascend910_9362, 所有浮点输入/噪声/输出均在 npu:0 (inference.py:device、obs_image.device、action_chunk.device、torch.npu.synchronize() 包围计时),无 .cpu() 回退。单卡 npu:0 同步计时 (warmup 3, measured 10, torch.npu.synchronize):
torch==2.8.0 + torch_npu, CANN 8.5.1, Ascend910_9362。完整视频生成需 2×NPU 分布式或量化 (INT8/NPU量化可降至 <32GB 但改采样精度)。trust_remote_code 涉及 Wan DiT/VAE/T5,含 flash-attn CUDA kernel 在 NPU 上替换为原生 torch.matmul+gelu 等价实现,已在 inference.py 代理并通过 CPU-NPU 对比验证;生产环境需将完整 groot.vla 用 torch_npu 融合算子替换。huggingface_hub snapshot_download 固定 revision 拉取。pip install -r requirements.txt
python inference.py
