z
z_studio/vla_so101_pick_n_place_full_expert-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

anikitakis/vla_so101_pick_n_place_full_expert on Ascend NPU

1. 模型简介

  • 模型来源: anikitakis/vla_so101_pick_n_place_full_expert
  • 模型类型: 机器人视觉-语言-动作(Vision-Language-Action, VLA)策略(library_name: lerobot), 基于 LeRobot 在 anikitakis/pick_n_place_dagger_3cam_dagger_only 数据集上微调, 面向 Fourier SO-101(so_follower,6 自由度机械臂)的 pick and place 抓取放置任务
  • 任务指令: "put the annotated object in the box"(把标注物体放入盒子)
  • 模型架构: SmolVLA1(LeRobot 官方实现),核心设计:
    • VLM 骨干:HuggingFaceTB/SmolVLM2-500M-Video-Instruct(SigLIP 视觉塔 + SmolLM2 文本模型),取前 16 层(num_vlm_layers=16);
    • Action Expert:宽度为 VLM 0.75 倍的 decoder-only 专家(expert_hidden_size=720, self_attn_every_n_layers=2 交叠自注意力层),通过交叉注意力读取 VLM 的 图像/语言/状态 KV 缓存(attention_mode=cross_attn);
    • 投影头:state_proj、action_in/out_proj、action_time_mlp;
    • 流匹配(Flow Matching)解码:从高斯噪声出发,经 10 步 Euler 采样(num_steps=10)生成 50 步动作块(chunk_size=50);
    • 输入规格:3 路 480×640×3 相机(observation.images.camera1/2/3, 训练时缩放/填充至 512×512,resize_imgs_with_padding=[512,512])、 6 维本体状态(内部 pad 至 max_state_dim=32)、tokenizer_max_length=48 的任务文本;
    • 输出规格: (1, 50, 6) 连续动作块 —— SO-101 6 维关节控制量;
    • 参数规模: ~450M(SmolVLM2-500M 骨干 16 层 + 轻量 action expert)。
  • 权重: 单文件 model.safetensors(906MB,VLM 权重内嵌其中, 无需额外下载 SmolVLM2 权重,仅需其 config + tokenizer/processor 约 5MB,见 §4.3)
  • 论文: SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics
  • 参考实现: https://github.com/huggingface/lerobot

本目录完成其在昇腾 Ascend NPU(torch_npu 推理引擎)上的完整适配与部署, 包含可运行的推理脚本 inference.py、批量测试驱动 run_test_cases.py、依赖清单 requirements.txt、运行用 venv 环境与 60 组验证结果。

2. 验证环境

组件版本
torch2.9.0(CPU build + torch_npu 扩展)
torch-npu2.9.0.post1+gitee7ba04
transformers4.57.6
lerobot0.4.4
numpy1.26.4
CANN8.5.1
Python3.11.14
NPUAscend 910B(Atlas 800T A2,64GB HBM × 2)
vllm-ascend / sglangN/A(本模型不使用)2

模型权重路径:

权重路径
SmolVLA 策略(本模型)/data/models/anikitakis/vla_so101_pick_n_place_full_expert
SmolVLM2-500M-Video-Instruct(config/tokenizer/processor)./backbone/SmolVLM2-500M-Video-Instruct

3. NPU 适配说明

SmolVLA 为 LeRobot 自定义策略架构(SmolVLAForActionPrediction),不在 vllm-ascend / sglang 的模型注册表内,且输出为连续动作而非自回归 token, 因此推理引擎选用 torch_npu2,通过 LeRobot 官方 SmolVLAPolicy 完成推理。

适配点CUDA 默认NPU 处理方式
设备分配config.device='cuda'覆盖为 npu:0,权重经 safetensors 直接加载到 NPU,policy.to(npu:0)
VLM 骨干vlm_model_name='HuggingFaceTB/SmolVLM2-500M-Video-Instruct'(需下载权重)VLM 权重内嵌于 checkpoint,config 覆盖为本地 backbone/ 目录(仅 config+tokenizer),并置 load_vlm_weights=False 跳过额外下载,离线加载
torch.bucketize/searchsorted(SmolVLM2 视觉塔动态分辨率 patch 定位)支持 bf16torch_npu 该算子不支持 bf16 输入(报 aclnnSearchSorted ... DT_BFLOAT16);inference.py 内置最小兼容层,将 bf16/fp16 输入临时转 fp32 计算(返回 int64 下标,数值等价)3
归一化统计量训练时由数据集统计从 checkpoint policy_preprocessor_step_5_normalizer_processor.safetensors 加载 state/action 的 mean/std,与 config 维度(6)对齐
数据预处理/后处理LeRobot make_pre_post_processors复用官方流水线并注入上述统计量:图像归一化、语言指令分词、action 反归一化;与策略配置对齐(observation.images.camera1/2/3)
去噪噪声sample_noise(随机)由 --seed 显式生成 N(0,I) 噪声作为流匹配采样起点,保证多次运行可复现

关键实现(均在本目录 inference.py,不修改任何三方库源码):

  1. 读取 checkpoint config.json,剔除 draccus 无法识别的训练/推送字段;
  2. 覆盖 cfg.device='npu:0'、cfg.vlm_model_name=<本地 backbone>、cfg.load_vlm_weights=False;
  3. 从 policy_preprocessor_step_5_normalizer_processor.safetensors 加载归一化统计量;
  4. SmolVLAPolicy(cfg, dataset_stats=stats) 构造策略,load_state_dict 全量加载 checkpoint 权重(0 missing / 0 unexpected);
  5. _patch_npu_ops() 对 torch.bucketize / torch.searchsorted 做 bf16→fp32 兼容;
  6. 构造 LeRobot 数据集帧(多视角图像 (C,H,W) float32、state、语言指令), preprocess(frame) → policy.predict_action_chunk(batch, noise=noise) → postprocess(actions)。

4. 分步推理操作流程

4.1 环境依赖

# 使用清华镜像安装 Python 依赖(本目录已创建 venv,继承系统级 torch/torch-npu)
cd /opt/atomgit/model_adapt/vla_so101_pick_n_place_full_expert-NPU
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple

torch / torch-npu 需与 CANN 版本严格匹配,请使用昇腾官方安装方式(见 requirements.txt 文末版本对照表)。本目录 venv/ 使用 python3 -m venv --system-site-packages venv 创建,自动继承系统级 torch / torch-npu,仅需在 venv 内安装 lerobot 及依赖。

4.2 创建运行环境(首次)

# 创建 venv(继承系统 torch/torch-npu 等)
python3 -m venv --system-site-packages venv

# 安装 lerobot[smolvla] 与依赖(清华镜像)
./venv/bin/pip install "lerobot[smolvla]==0.4.4" -i https://pypi.tuna.tsinghua.edu.cn/simple
./venv/bin/pip install numpy==1.26.4 -i https://pypi.tuna.tsinghua.edu.cn/simple

4.3 准备模型权重

# SmolVLA 策略权重已本地化于 /data/models/anikitakis/vla_so101_pick_n_place_full_expert
# SmolVLM2-500M-Video-Instruct 仅需 config + tokenizer/processor(约 5MB),已本地化于
# 本目录 backbone/SmolVLM2-500M-Video-Instruct(权重内嵌于 checkpoint,无需下载)

4.4 运行推理

cd /opt/atomgit/model_adapt/vla_so101_pick_n_place_full_expert-NPU
./venv/bin/python inference.py \
    --task "put the annotated object in the box" \
    --state "0,0,0,0,0,0" \
    --obj-pos "0.4,0.35" \
    --seed 42 \
    --output-dir output/my_run

生成的产物位于 output/my_run/:

文件说明
action_chunk.npy预测动作 chunk,(1, 50, 6) float32
report.json加载/推理耗时、设备、动作统计、NPU 显存、确定性等指标

--help 可查看全部参数(--model、--backbone、--device、--task、--state、 --obj-pos、--num-steps、--seed、--no-determinism-check 等)。

4.5 批量测试(可选)

# 加载模型一次,批量执行 60 组测试用例(约 20s)
./venv/bin/python run_test_cases.py
# 结果汇总:output/test_cases_summary.json,单组结果:output/test_case_01/…/60/

5. 测试用例与输出结果

共 60 组测试用例,覆盖 12 条任务指令 × 5 组(状态 / 物体位置 / 种子)× 去噪步数(10 / 12 / 15),全部在 Ascend 910B(npu:0)上实测通过 (成功 60/60,全部输出数值有限,状态 SUCCESS)。

5.1 展示用例 A(默认指令)

输入:合成三视角桌面图像(含标注物体,位置 0.4,0.35),state=全零, 语言指令 "put the annotated object in the box",seed=42,10 步去噪。

./venv/bin/python inference.py \
    --task "put the annotated object in the box" \
    --state "0,0,0,0,0,0" \
    --obj-pos "0.4,0.35" \
    --seed 42 \
    --output-dir output/showcase_A

输出(Ascend 910B NPU 实测):

[INFO] 昇腾推理引擎: torch_npu | NPU 设备: npu:0  (共 2 卡)
[加载] 策略结构构造完成: 16.7s
[加载] checkpoint 权重加载完成: 0.1s (missing=0 项归一化缓冲)
[加载] 模型已就绪 @ npu:0  (dtype=torch.float32, 共 450M 参数)
[INFO] 预处理 batch keys:
       observation.images.camera1: (1, 3, 480, 640) torch.float32 npu:0
       observation.images.camera2: (1, 3, 480, 640) torch.float32 npu:0
       observation.images.camera3: (1, 3, 480, 640) torch.float32 npu:0
       observation.language.attention_mask: (1, 48) torch.bool npu:0
       observation.language.tokens: (1, 48) torch.int64 npu:0
       observation.state: (1, 6) torch.float32 npu:0
[INFO] 运行推理(flow-matching 去噪)...
[INFO] 推理完成: 0.60s
[INFO] 二次前向耗时: 0.30s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
[校验] ✓ 数值范围: [-87.941, 84.531]
[INFO] 动作已保存: output/showcase_A/action_chunk.npy  shape=(1, 50, 6)
[INFO] 报告已保存: output/showcase_A/report.json

==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-25.5466  37.0873 -29.2864  34.8874   0.7446   3.5933]
==== 状态: SUCCESS ====

output/showcase_A/report.json(关键字段):

{
  "device": "npu:0",
  "engine": "torch_npu",
  "task": "put the annotated object in the box",
  "state": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  "obj_pos": [0.4, 0.35],
  "seed": 42,
  "num_steps": 10,
  "chunk_size": 50,
  "action_dim": 6,
  "deterministic": true,
  "action_first_step": [-25.5466, 37.0873, -29.2864, 34.8874, 0.7446, 3.5933],
  "action_mean_abs": 43.04,
  "infer_time_s": 0.60,
  "npu_max_memory_allocated_gb": 1.36,
  "status": "SUCCESS"
}

5.2 展示用例 B(不同指令 + 关节零位状态)

输入:物体位置 0.3,0.3,state 取训练均值附近(SO-101 关节零位), 指令 "grasp the red block and put it in the container",seed=7。

./venv/bin/python inference.py \
    --task "grasp the red block and put it in the container" \
    --state="-87.5,90.0,-81.0,86.5,-7.8,16.2" \
    --obj-pos "0.3,0.3" \
    --seed 7 \
    --output-dir output/showcase_B

注意:--state 首维为负值时需用 --state=-87.5,... 的等号写法(argparse 约定)。

输出(关键行):

[INFO] 推理完成: 0.60s
[INFO] 二次前向耗时: 0.30s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-87.2587  95.5665 -90.5581  90.6815  -7.8226  23.841 ]
==== 状态: SUCCESS ====

5.3 展示用例 C(不同指令 + 15 步去噪)

输入:物体位置 0.5,0.45,state 关节偏移,指令 "move the cube into the container", seed=2024,15 步去噪。

./venv/bin/python inference.py \
    --task "move the cube into the container" \
    --state="-85.0,92.0,-78.0,84.0,-5.0,14.0" \
    --obj-pos "0.5,0.45" \
    --seed 2024 \
    --num-steps 15 \
    --output-dir output/showcase_C

输出(关键行):

[INFO] 推理完成: 0.71s
[INFO] 二次前向耗时: 0.46s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-85.7613  95.7447 -86.5018  85.9018  -5.729   23.7636]
==== 状态: SUCCESS ====

5.4 全量测试用例(60 组)

执行:./venv/bin/python run_test_cases.py(单进程加载模型一次,批量推理,约 20s)。 输出表:action_first_step 为动作块第 1 步 6 维关节控制量; mean_abs / std / min / max 为整段动作块统计;infer 为单组推理耗时。 每组用例均可通过 --task --state --obj-pos --seed --num-steps 以 inference.py 独立复现(同参数输出一致)。

#任务指令本体状态 state物体 objseed步数第 1 步动作(6 维关节控制量)mean_absinfer(s)
01put the annotated object in the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-25.55, 37.09, -29.29, 34.89, 0.74, 3.5943.040.59
02put the annotated object in the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-87.81, 96.28, -91.30, 90.99, -8.02, 23.3571.270.29
03put the annotated object in the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-84.49, 95.38, -86.32, 87.10, -5.25, 19.4968.840.30
04put the annotated object in the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-88.95, 94.37, -93.83, 91.85, -9.33, 18.5662.050.30
05put the annotated object in the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.28, 93.45, -83.70, 84.77, -7.53, 22.5470.910.29
06place the object into the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-27.58, 37.22, -34.53, 39.37, -1.41, 2.9644.880.30
07place the object into the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.54, 96.23, -90.37, 88.84, -9.20, 24.0471.780.29
08place the object into the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-84.95, 94.57, -86.04, 86.24, -6.22, 17.0768.150.30
09place the object into the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-88.67, 92.69, -92.24, 89.72, -10.21, 15.0765.210.30
10place the object into the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.73, 93.09, -83.30, 83.67, -8.25, 21.7371.210.30
11grasp the red block and put it in the container0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-33.67, 37.87, -28.98, 33.73, 0.35, 4.0445.660.30
12grasp the red block and put it in the container-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-87.26, 95.57, -90.56, 90.68, -7.82, 23.8468.900.30
13grasp the red block and put it in the container-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-83.78, 93.94, -87.10, 89.28, -5.27, 18.3661.250.30
14grasp the red block and put it in the container-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-86.85, 91.22, -91.09, 91.02, -8.69, 14.1654.200.30
15grasp the red block and put it in the container-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-87.26, 92.11, -83.16, 85.29, -7.14, 23.8867.240.29
16pick up the object and drop it in the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-28.35, 35.30, -31.32, 36.75, -1.24, 6.2943.140.30
17pick up the object and drop it in the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.65, 94.93, -89.31, 89.73, -8.04, 24.3670.070.30
18pick up the object and drop it in the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-85.16, 94.06, -85.79, 87.98, -5.26, 20.7864.880.30
19pick up the object and drop it in the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.28, 91.53, -90.61, 90.46, -8.95, 17.7456.020.29
20pick up the object and drop it in the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.59, 92.13, -82.86, 85.07, -7.06, 23.9469.030.29
21move the annotated object to the target container0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-29.88, 36.82, -29.51, 35.68, -0.28, 6.9149.680.29
22move the annotated object to the target container-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-87.86, 97.35, -91.89, 91.59, -8.22, 25.4371.100.30
23move the annotated object to the target container-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-84.66, 96.15, -87.41, 88.31, -5.57, 21.7967.940.29
24move the annotated object to the target container-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.33, 95.14, -94.22, 91.78, -9.57, 19.7959.140.30
25move the annotated object to the target container-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.32, 94.17, -84.09, 85.09, -7.73, 24.4070.760.30
26put the object in the bin0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-26.04, 39.14, -33.47, 37.80, -1.79, 12.1446.830.30
27put the object in the bin-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.64, 95.77, -90.91, 90.37, -8.38, 24.1071.450.30
28put the object in the bin-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-85.41, 95.31, -86.48, 87.04, -5.65, 20.5469.440.30
29put the object in the bin-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.95, 93.61, -93.29, 90.91, -9.53, 19.7866.570.30
30put the object in the bin-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.91, 92.77, -83.78, 84.84, -7.62, 22.6471.130.29
31place the object into the goal box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-28.86, 39.07, -35.59, 38.60, -0.91, 4.4846.600.30
32place the object into the goal box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.49, 97.01, -90.87, 89.37, -8.70, 24.6371.620.29
33place the object into the goal box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-85.05, 95.34, -86.36, 86.52, -5.76, 19.0968.400.29
34place the object into the goal box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.02, 93.55, -92.81, 90.09, -9.97, 17.1366.570.29
35place the object into the goal box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.55, 93.83, -83.70, 83.83, -7.92, 22.6571.120.29
36transfer the object to the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-26.59, 34.37, -32.42, 36.72, -1.20, 1.8143.060.30
37transfer the object to the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.37, 96.30, -91.17, 90.14, -8.57, 24.5071.960.30
38transfer the object to the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-84.87, 94.95, -86.33, 87.02, -5.72, 20.0569.300.30
39transfer the object to the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-88.86, 93.30, -92.89, 90.66, -9.97, 16.8468.320.29
40transfer the object to the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.53, 93.22, -83.82, 84.69, -7.94, 23.5571.440.28
41take the object and put it in the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-27.23, 34.75, -32.10, 36.89, -1.00, 3.7342.880.29
42take the object and put it in the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-86.89, 96.65, -92.60, 91.90, -8.30, 22.8367.100.30
43take the object and put it in the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-83.17, 94.45, -87.98, 89.67, -5.53, 18.0056.520.29
44take the object and put it in the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-87.38, 92.24, -92.86, 91.96, -9.59, 15.8353.670.29
45take the object and put it in the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-86.97, 92.90, -84.16, 85.63, -7.71, 22.3263.810.30
46move the cube into the container0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-28.47, 39.64, -33.77, 35.77, 0.00, 5.6347.360.29
47move the cube into the container-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.79, 97.40, -91.17, 89.07, -8.98, 26.3671.930.29
48move the cube into the container-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-85.58, 96.19, -86.32, 85.72, -6.24, 23.6769.650.31
49move the cube into the container-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.80, 95.07, -93.82, 89.67, -10.06, 18.1169.710.30
50move the cube into the container-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-89.38, 93.83, -83.83, 83.57, -8.10, 24.8271.540.31
51insert the object into the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-26.77, 35.16, -32.95, 37.16, -0.55, 4.6345.150.31
52insert the object into the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.59, 97.19, -90.98, 89.05, -8.83, 24.5372.110.31
53insert the object into the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-85.38, 95.94, -86.38, 86.07, -5.96, 20.4169.310.30
54insert the object into the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-89.17, 94.06, -92.94, 89.87, -10.00, 17.2969.220.30
55insert the object into the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.88, 93.74, -83.56, 83.66, -7.96, 23.0771.520.30
56put the object away into the box0.0,0.0,0.0,0.0,0.0,0.00.4,0.354210-23.10, 39.86, -33.82, 36.17, -0.86, 6.3944.470.30
57put the object away into the box-87.5,90.0,-81.0,86.5,-7.8,16.20.3,0.30710-88.27, 96.53, -90.17, 89.23, -8.98, 24.3371.260.30
58put the object away into the box-85.0,92.0,-78.0,84.0,-5.0,14.00.5,0.45202410-84.76, 96.22, -85.93, 86.04, -5.96, 19.6668.960.30
59put the object away into the box-90.0,88.0,-84.0,88.0,-9.0,18.00.6,0.5012312-88.99, 93.45, -92.23, 89.94, -10.26, 16.0967.780.30
60put the object away into the box-88.0,91.0,-80.0,85.0,-6.5,15.50.3,0.40515-88.37, 93.79, -83.52, 83.86, -8.13, 23.0771.100.30

5.5 验证结论

  • 可运行性:60/60 组用例全部 SUCCESS,输出形状 (1,50,6) 正确、全部数值有限、 无 NaN/Inf;
  • 确定性:展示用例二次前向输出一致(deterministic: true),去噪噪声由 --seed 显式控制,同参数可复现;
  • 输入敏感性:不同任务指令、本体状态、物体位置、去噪步数均产生不同的动作输出 (如用例 A 第 1 步首维 -25.55 vs 用例 B -87.26),且数值位于 SO-101 关节动作量级 (±100 内),说明模型在 NPU 上完整加载并正确推理。

6. 性能参考

测试条件:单张 Ascend 910B(npu:0),10 步 flow-matching 去噪,3 视角 480×640 图像 + 6 维 state,bf16 骨干 / fp32 投影头混合精度(忠于原权重)。

指标数值
模型加载(构造 + 权重 + 流水线)~27 s(首次数次运行,含算子初始化)
单组推理(10 步去噪)~0.3–0.6 s
动作 chunk 规模50 步 × 6 维
NPU 峰值显存~1.4 GB
批量 60 组用例总耗时~20 s(单进程复用模型)

7. 注意事项

  1. 推理引擎:SmolVLA 为 VLA 策略,输出连续动作而非 token,非自回归 LLM, 不适用 vllm-ascend / sglang,推理引擎为 torch_npu。2
  2. VLM 骨干离线化:权重内嵌于 checkpoint,仅需本地 backbone/ 目录提供 config + tokenizer/processor;运行脚本默认设置 HF_HUB_OFFLINE=1(见 run_test_cases.py 环境变量),不访问外网。
  3. bf16 searchsorted 限制:SmolVLM2 视觉塔在 NPU 上需将 bucketize/ searchsorted 的 bf16 输入转 fp32(inference.py 已内置,见 §3)。3
  4. 合成观测:无真实相机/机械臂时使用合成桌面图像(含标注物体圆盘与目标框)。 接入真实数据时,把 inference.py 中 build_synthetic_observation 替换为 实际传感器帧即可,图像须为 (C,H,W) float32 [0,1](480×640),state 为 6 维。
  5. 语言指令:帧中需同时提供顶层 task(供 tokenizer 读取)与 observation.language.task/instruction 观测字段。
  6. 去噪步数:--num-steps 默认 10(checkpoint 训练配置),增大可提升采样 质量但线性增加耗时;展示用例 C 验证了 15 步配置。
  7. 确定性:噪声由 --seed 显式生成;predict_action_chunk 会就地修改噪声 张量,连续调用必须传入独立拷贝(脚本已用 noise.clone())。
  8. 文件写限流:本工作区为 SFS Turbo 存储,突发连续写可能偶发 EDQUOT 限流,脚本输出写盘已带自动重试。
  9. 精度说明:本目录验证的是可运行性(SUCCESS)与确定性,不含与 CUDA 的 逐位数值对标;如需精度对比,请在 NVIDIA 环境复跑相同输入。

8. 参考资料

  • anikitakis/vla_so101_pick_n_place_full_expert (HuggingFace)
  • SmolVLA 论文
  • LeRobot 文档
  • SmolVLM2-500M-Video-Instruct (ModelScope)
  • SmolVLM2-500M-Video-Instruct (HuggingFace)
  • vllm-ascend 文档

贡献者: z_studio | 赛道: 模型适配赛道

Footnotes

  1. "SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics", HuggingFace, arXiv:2506.01844。 ↩

  2. vllm-ascend / sglang 面向自回归大语言模型,SmolVLA 为 Vision-Language-Action 策略(输出连续动作),使用 torch_npu 原生推理。 ↩ ↩2 ↩3

  3. 昇腾算子 searchsorted/bucketize 的 bf16 支持列表不含 DT_BFLOAT16,转 fp32 后结果完全等价(返回 int64 下标)。 ↩ ↩2