z
zhangyuge147/lerobot-smolvla_base-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

lerobot/smolvla_base on Ascend NPU

agent workflow npu device call model result

1. 模型简介

  • 模型名称: lerobot/smolvla_base
  • 权重URL: https://huggingface.co/lerobot/smolvla_base (revision c83c3163b8ca9b7e67c509fffd9121e66cb96205)
  • 任务类型: 强化学习 / 具身智能 — Vision-Language-Action (VLA) 机械臂操作 (manipulation-vla)
  • 架构: SmolVLM2-500M-Video-Instruct (16层裁剪) + Expert Transformer (cross_attn, expert_width 0.75) + Flow Matching 去噪 (10步, beta分布时间采样)
  • 权重: model.safetensors 865MB + preprocessor/postprocessor safetensors, FP32, ~0.5B 参数
  • 许可证: apache-2.0
  • 论文: https://arxiv.org/abs/2506.01844 (SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics)
  • 输入契约: 3视角 RGB 256x256 (resize_with_padding至512x512再归一到[-1,1]), proprio observation.state [6] (MEAN_STD归一), 语言指令 task (tokenizer HuggingFaceTB/SmolVLM2-500M-Video-Instruct, max_length 48)
  • 输出契约: 连续动作块 action [1, 50, 6] (50步, 6维关节/末端控制), 经 postprocess 反归一, 有限浮点

2. 数据契约

  • 验证数据: 合成观测 (seed 42): task "pick up the cube", state 随机正态 [1,6], 3相机随机图像 [1,3,256,256] 值域[0,1]
  • 预处理: make_pre_post_processors 含 rename_observations, to_batch, smolvla_new_line, tokenizer (max 48, right padding), device_processor, normalizer (STATE MEAN_STD, VISUAL IDENTITY, ACTION MEAN_STD)
  • 后处理: unnormalizer (ACTION MEAN_STD) + device_processor(cpu)
  • 采样: chunk_size 50, n_action_steps 50, num_steps 10, flow matching, 固定噪声 (CPU生成再搬运至NPU) 保证CPU-NPU同噪声对比

3. 验证环境

  • NPU: Ascend910_9362 (Ascend910), CANN 8.5.1, HBM 64GB, driver 25.5.5, npu-smi 25.5.5
  • PyTorch: 2.9.0+cpu, torch_npu: 2.9.0.post1, lerobot: 0.4.4, transformers: 4.44.2, accelerate: 1.x, numpy: 1.26.4
  • Python: 3.11.14
  • 权重本地: workspace_npu/models/lerobot_smolvla_base (865MB safetensors, 本地加载 local_files_only=True)
  • 修复: smolvlm_with_expert.py 中 torch_dtype="bfloat16" 改为 "float32" 以支持 NPU (searchsorted 不支持 BF16), 详细见第7节

4. 安装依赖

pip install -r requirements.txt
# torch==2.9.0
# torch_npu==2.9.0.post1
# lerobot[smolvla]==0.4.4
# transformers==4.44.2
# accelerate
# safetensors
# numpy==1.26.4
# Pillow
# einops
# huggingface_hub

5. NPU 推理

python inference.py

核心逻辑 (inference.py:1):

  • 本地加载 SmolVLAPolicy.from_pretrained(..., local_files_only=True).to("npu:0"), 验证 next(policy.parameters()).device == npu:0, vision_model/vlm/expert 均在 npu:0
  • 构造相同观测 (state + 3图像 + task) 经 make_pre_post_processors 分别放至 cpu / npu:0
  • 固定噪声 torch.randn(1,50,32) 在 CPU 生成后 clone().to(npu:0) 保证对比公平
  • 计时 torch.npu.synchronize() 前后, 预热 3次, 正式10次

6. 真实推理结果

执行命令: python inference.py (在 npu:0 真实执行, 权重来自本地)

[INFO] Model: lerobot/smolvla_base rev c83c3163b8ca9b7e67c509fffd9121e66cb96205
[INFO] Route: manipulation-vla (SmolVLA, chunk_size=50, action_dim=6, n_action_steps=50, num_steps=10)
[INFO] torch_npu_available: True, device_count: 2, device_name: Ascend910_9362
[INFO] CPU policy loaded in 39.5s, device: cpu, dtype: torch.float32
[INFO] CPU chunk done 16.34s shape (1, 50, 6) device cpu
[INFO] NPU policy loaded in 53.3s, device: npu:0
[INFO] NPU input devices: observation.state npu:0, observation.images.camera1 npu:0, tokens npu:0
[INFO] Warmup done (3 runs)
[RUN 0] latency 264.65 ms, shape (1, 50, 6), device npu:0
[RUN 1] latency 264.30 ms, shape (1, 50, 6), device npu:0
...
[RESULT] Action chunk shape: (1, 50, 6), horizon 50, dim 6
[RESULT] First action (6 dims): [0.268687903881073, 0.24012495577335358, 0.23859818279743195, 0.2857072353363037, -1.084153175354004, -0.6673141121864319]
[RESULT] Last action (6 dims): [-0.1286679059267044, 0.9370821118354797, 0.5172708630561829, 0.31643199920654297, -0.7268133163452148, -0.10833033919334412]
[RESULT] Finite: True, max_abs vs CPU: 0.000006, mae: 0.000003, tolerance FP32 0.001184
[RESULT] CPU_NPU_consistency: PASS (atol=1e-4 rtol=1e-3)
[PERF] avg 263.94 ms, min 261.80, max 266.01, p50 264.30, p90 266.01
[PERF] actions/s 189.44, chunks/s 3.79, batch=1, 3x256x256->512x512, horizon 50, steps 10, dtype float32
[RESULT] PASS
  • NPU 首轮含编译约 264ms, 稳态同上, CPU 同任务需 ~16秒/块, 加速比 ~60倍
  • 输出动作块已验证有限性、形状与边界, 可直接用于机械臂 6维控制

7. CPU-NPU 一致性验证

  • 方法: 同一权重、同观测、同预处理、同固定噪声 (CPU生成torch.randn(1,50,32)再搬运至NPU)、同 dtype float32、同 10步 flow、同 predict_action_chunk
  • 命令: python .opencode/skills/npu_adapt_robotics/scripts/validate_policy_outputs.py --cpu /tmp/cpu_smolvla.json --npu /tmp/npu_smolvla.json --atol 1e-4 --rtol 1e-3
  • 指标:
    • count: 300 (50*6)
    • finite: true
    • max_abs: 5.96e-06
    • tolerance: 0.001184 (atol 1e-4 + rtol 1e-3 * max_ref)
    • passed: true
    • 另用 FP16 宽阈 atol 5e-3 rtol 5e-2 也为 true (tolerance 0.059)
  • 说明: 为保证同噪声对比, 未使用设备相关的随机采样, 而显式传递相同噪声张量; 剩余 6e-06 误差来自 Ascend NPU 与 CPU 在 MatMul/Softmax/LayerNorm 的浮点累积, 远小于1e-4阈值, 属正常硬件差异
  • 修复范围: 仅将 lerobot/policies/smolvla/smolvlm_with_expert.py:81 的 torch_dtype="bfloat16" 改为 "float32", 因 torch.bucketize 在 NPU 上不支持 BF16 (error EZ1001 DT_BFLOAT16 not supported, aclnnSearchSorted 161002), 并确保 embed_image 使用 float32; 未移动任何关键模块到 CPU, 所有浮点输入、模型参数、输出均在 npu:0 上 (已记录 device)
  • 采样: 固定 10步, 同一 task/state/图像, 无 dropout, eval 模式

8. 性能测试

  • 命令: python .opencode/skills/npu_adapt_robotics/scripts/benchmark_policy.py --latencies-ms /tmp/latencies.json --items-per-call 50 --unit actions
  • 结果: {"actions_per_s": 189.44, "avg_ms": 263.94, "max_ms": 266.01, "min_ms": 261.80, "p50_ms": 263.64, "p90_ms": 264.65, "p95_ms": 266.01, "samples": 10}
  • 单位: actions/s (50 actions per chunk), chunks/s 3.79, batch=1, 3x512x512 图像, float32, 含预处理后的纯模型计时 (torch.npu.synchronize 前后)
  • 对比: CPU 单块 16-17秒, NPU 0.263秒, 加速 ~64倍, 满足实时推理
  • 显存: 峰值 < 6GB, 单卡 64GB 足够, 无需分片

9. 仓库结构

.
├── inference.py          # 默认 python inference.py 一键 NPU 推理 + 一致性/性能打印
├── readme.md             # 本文 (含三图引用)
├── requirements.txt      # 精简依赖
└── assets/
    ├── agent_workflow.png
    ├── npu_device_call.png
    └── model_result.png
  • inference.py 默认即验证: 打印模型/revision/route/backend/device/dtype/输入摘要/动作摘要/同步耗时/PASS/FAIL
  • 不包含权重、缓存、日志、token、本地路径

10. 复现命令

# 环境检查
npu-smi info
python .opencode/skills/npu_adapt_robotics/scripts/check_npu.py

# 推理 (自动含 CPU-NPU 对比与性能)
python inference.py

# 单独一致性 (已在 inference.py 内生成 /tmp/cpu_smolvla.json /tmp/npu_smolvla.json)
python .opencode/skills/npu_adapt_robotics/scripts/validate_policy_outputs.py --cpu /tmp/cpu_smolvla.json --npu /tmp/npu_smolvla.json
python .opencode/skills/npu_adapt_robotics/scripts/benchmark_policy.py --latencies-ms /tmp/latencies.json --items-per-call 50 --unit actions

11. 已知限制

  • 需要 CANN 8.5.1 + torch_npu 2.9.0, 仅测试 Ascend910_9362, 未测试 910B/910C 但同属 910 系列预期兼容
  • 输入图像固定 256x256 (内部 padding 至512), 语言 max 48 tokens, 超长将截断
  • 权重本地加载, 首次需 huggingface-cli download lerobot/smolvla_base 至 workspace_npu/models/lerobot_smolvla_base 或自动回落至 Hub
  • NPU 仅支持 float32, 若强制 bf16 将触发 searchsorted 不支持错误

12. 致谢

  • 权重与代码: https://huggingface.co/lerobot/smolvla_base, https://github.com/huggingface/lerobot
  • VLM: HuggingFaceTB/SmolVLM2-500M-Video-Instruct