library_name: lerobot),
基于 LeRobot 在 anikitakis/pick_n_place_dagger_3cam_dagger_only 数据集上微调,
面向 Fourier SO-101(so_follower,6 自由度机械臂)的 pick and place 抓取放置任务"put the annotated object in the box"(把标注物体放入盒子)HuggingFaceTB/SmolVLM2-500M-Video-Instruct(SigLIP 视觉塔 +
SmolLM2 文本模型),取前 16 层(num_vlm_layers=16);expert_hidden_size=720,
self_attn_every_n_layers=2 交叠自注意力层),通过交叉注意力读取 VLM 的
图像/语言/状态 KV 缓存(attention_mode=cross_attn);state_proj、action_in/out_proj、action_time_mlp;num_steps=10)生成 50 步动作块(chunk_size=50);observation.images.camera1/2/3,
训练时缩放/填充至 512×512,resize_imgs_with_padding=[512,512])、
6 维本体状态(内部 pad 至 max_state_dim=32)、tokenizer_max_length=48 的任务文本;(1, 50, 6) 连续动作块 —— SO-101 6 维关节控制量;model.safetensors(906MB,VLM 权重内嵌其中,
无需额外下载 SmolVLM2 权重,仅需其 config + tokenizer/processor 约 5MB,见 §4.3)本目录完成其在昇腾 Ascend NPU(torch_npu 推理引擎)上的完整适配与部署,
包含可运行的推理脚本 inference.py、批量测试驱动 run_test_cases.py、依赖清单
requirements.txt、运行用 venv 环境与 60 组验证结果。
| 组件 | 版本 |
|---|---|
| torch | 2.9.0(CPU build + torch_npu 扩展) |
| torch-npu | 2.9.0.post1+gitee7ba04 |
| transformers | 4.57.6 |
| lerobot | 0.4.4 |
| numpy | 1.26.4 |
| CANN | 8.5.1 |
| Python | 3.11.14 |
| NPU | Ascend 910B(Atlas 800T A2,64GB HBM × 2) |
| vllm-ascend / sglang | N/A(本模型不使用)2 |
模型权重路径:
| 权重 | 路径 |
|---|---|
| SmolVLA 策略(本模型) | /data/models/anikitakis/vla_so101_pick_n_place_full_expert |
| SmolVLM2-500M-Video-Instruct(config/tokenizer/processor) | ./backbone/SmolVLM2-500M-Video-Instruct |
SmolVLA 为 LeRobot 自定义策略架构(SmolVLAForActionPrediction),不在
vllm-ascend / sglang 的模型注册表内,且输出为连续动作而非自回归 token,
因此推理引擎选用 torch_npu2,通过 LeRobot 官方 SmolVLAPolicy 完成推理。
| 适配点 | CUDA 默认 | NPU 处理方式 |
|---|---|---|
| 设备分配 | config.device='cuda' | 覆盖为 npu:0,权重经 safetensors 直接加载到 NPU,policy.to(npu:0) |
| VLM 骨干 | vlm_model_name='HuggingFaceTB/SmolVLM2-500M-Video-Instruct'(需下载权重) | VLM 权重内嵌于 checkpoint,config 覆盖为本地 backbone/ 目录(仅 config+tokenizer),并置 load_vlm_weights=False 跳过额外下载,离线加载 |
torch.bucketize/searchsorted(SmolVLM2 视觉塔动态分辨率 patch 定位) | 支持 bf16 | torch_npu 该算子不支持 bf16 输入(报 aclnnSearchSorted ... DT_BFLOAT16);inference.py 内置最小兼容层,将 bf16/fp16 输入临时转 fp32 计算(返回 int64 下标,数值等价)3 |
| 归一化统计量 | 训练时由数据集统计 | 从 checkpoint policy_preprocessor_step_5_normalizer_processor.safetensors 加载 state/action 的 mean/std,与 config 维度(6)对齐 |
| 数据预处理/后处理 | LeRobot make_pre_post_processors | 复用官方流水线并注入上述统计量:图像归一化、语言指令分词、action 反归一化;与策略配置对齐(observation.images.camera1/2/3) |
| 去噪噪声 | sample_noise(随机) | 由 --seed 显式生成 N(0,I) 噪声作为流匹配采样起点,保证多次运行可复现 |
关键实现(均在本目录 inference.py,不修改任何三方库源码):
config.json,剔除 draccus 无法识别的训练/推送字段;cfg.device='npu:0'、cfg.vlm_model_name=<本地 backbone>、cfg.load_vlm_weights=False;policy_preprocessor_step_5_normalizer_processor.safetensors 加载归一化统计量;SmolVLAPolicy(cfg, dataset_stats=stats) 构造策略,load_state_dict 全量加载
checkpoint 权重(0 missing / 0 unexpected);_patch_npu_ops() 对 torch.bucketize / torch.searchsorted 做 bf16→fp32 兼容;(C,H,W) float32、state、语言指令),
preprocess(frame) → policy.predict_action_chunk(batch, noise=noise) → postprocess(actions)。# 使用清华镜像安装 Python 依赖(本目录已创建 venv,继承系统级 torch/torch-npu)
cd /opt/atomgit/model_adapt/vla_so101_pick_n_place_full_expert-NPU
pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simpletorch / torch-npu 需与 CANN 版本严格匹配,请使用昇腾官方安装方式(见
requirements.txt文末版本对照表)。本目录venv/使用python3 -m venv --system-site-packages venv创建,自动继承系统级 torch / torch-npu,仅需在 venv 内安装 lerobot 及依赖。
# 创建 venv(继承系统 torch/torch-npu 等)
python3 -m venv --system-site-packages venv
# 安装 lerobot[smolvla] 与依赖(清华镜像)
./venv/bin/pip install "lerobot[smolvla]==0.4.4" -i https://pypi.tuna.tsinghua.edu.cn/simple
./venv/bin/pip install numpy==1.26.4 -i https://pypi.tuna.tsinghua.edu.cn/simple# SmolVLA 策略权重已本地化于 /data/models/anikitakis/vla_so101_pick_n_place_full_expert
# SmolVLM2-500M-Video-Instruct 仅需 config + tokenizer/processor(约 5MB),已本地化于
# 本目录 backbone/SmolVLM2-500M-Video-Instruct(权重内嵌于 checkpoint,无需下载)cd /opt/atomgit/model_adapt/vla_so101_pick_n_place_full_expert-NPU
./venv/bin/python inference.py \
--task "put the annotated object in the box" \
--state "0,0,0,0,0,0" \
--obj-pos "0.4,0.35" \
--seed 42 \
--output-dir output/my_run生成的产物位于 output/my_run/:
| 文件 | 说明 |
|---|---|
action_chunk.npy | 预测动作 chunk,(1, 50, 6) float32 |
report.json | 加载/推理耗时、设备、动作统计、NPU 显存、确定性等指标 |
--help 可查看全部参数(--model、--backbone、--device、--task、--state、
--obj-pos、--num-steps、--seed、--no-determinism-check 等)。
# 加载模型一次,批量执行 60 组测试用例(约 20s)
./venv/bin/python run_test_cases.py
# 结果汇总:output/test_cases_summary.json,单组结果:output/test_case_01/…/60/共 60 组测试用例,覆盖 12 条任务指令 × 5 组(状态 / 物体位置 / 种子)×
去噪步数(10 / 12 / 15),全部在 Ascend 910B(npu:0)上实测通过
(成功 60/60,全部输出数值有限,状态 SUCCESS)。
输入:合成三视角桌面图像(含标注物体,位置 0.4,0.35),state=全零,
语言指令 "put the annotated object in the box",seed=42,10 步去噪。
./venv/bin/python inference.py \
--task "put the annotated object in the box" \
--state "0,0,0,0,0,0" \
--obj-pos "0.4,0.35" \
--seed 42 \
--output-dir output/showcase_A输出(Ascend 910B NPU 实测):
[INFO] 昇腾推理引擎: torch_npu | NPU 设备: npu:0 (共 2 卡)
[加载] 策略结构构造完成: 16.7s
[加载] checkpoint 权重加载完成: 0.1s (missing=0 项归一化缓冲)
[加载] 模型已就绪 @ npu:0 (dtype=torch.float32, 共 450M 参数)
[INFO] 预处理 batch keys:
observation.images.camera1: (1, 3, 480, 640) torch.float32 npu:0
observation.images.camera2: (1, 3, 480, 640) torch.float32 npu:0
observation.images.camera3: (1, 3, 480, 640) torch.float32 npu:0
observation.language.attention_mask: (1, 48) torch.bool npu:0
observation.language.tokens: (1, 48) torch.int64 npu:0
observation.state: (1, 6) torch.float32 npu:0
[INFO] 运行推理(flow-matching 去噪)...
[INFO] 推理完成: 0.60s
[INFO] 二次前向耗时: 0.30s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
[校验] ✓ 数值范围: [-87.941, 84.531]
[INFO] 动作已保存: output/showcase_A/action_chunk.npy shape=(1, 50, 6)
[INFO] 报告已保存: output/showcase_A/report.json
==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-25.5466 37.0873 -29.2864 34.8874 0.7446 3.5933]
==== 状态: SUCCESS ====output/showcase_A/report.json(关键字段):
{
"device": "npu:0",
"engine": "torch_npu",
"task": "put the annotated object in the box",
"state": [0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
"obj_pos": [0.4, 0.35],
"seed": 42,
"num_steps": 10,
"chunk_size": 50,
"action_dim": 6,
"deterministic": true,
"action_first_step": [-25.5466, 37.0873, -29.2864, 34.8874, 0.7446, 3.5933],
"action_mean_abs": 43.04,
"infer_time_s": 0.60,
"npu_max_memory_allocated_gb": 1.36,
"status": "SUCCESS"
}输入:物体位置 0.3,0.3,state 取训练均值附近(SO-101 关节零位),
指令 "grasp the red block and put it in the container",seed=7。
./venv/bin/python inference.py \
--task "grasp the red block and put it in the container" \
--state="-87.5,90.0,-81.0,86.5,-7.8,16.2" \
--obj-pos "0.3,0.3" \
--seed 7 \
--output-dir output/showcase_B注意:
--state首维为负值时需用--state=-87.5,...的等号写法(argparse 约定)。
输出(关键行):
[INFO] 推理完成: 0.60s
[INFO] 二次前向耗时: 0.30s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-87.2587 95.5665 -90.5581 90.6815 -7.8226 23.841 ]
==== 状态: SUCCESS ====输入:物体位置 0.5,0.45,state 关节偏移,指令 "move the cube into the container",
seed=2024,15 步去噪。
./venv/bin/python inference.py \
--task "move the cube into the container" \
--state="-85.0,92.0,-78.0,84.0,-5.0,14.0" \
--obj-pos "0.5,0.45" \
--seed 2024 \
--num-steps 15 \
--output-dir output/showcase_C输出(关键行):
[INFO] 推理完成: 0.71s
[INFO] 二次前向耗时: 0.46s | 确定性(两次输出一致): 是
[校验] ✓ 形状正确: (1, 50, 6)
[校验] ✓ 全部数值有限
==== 推理输出示例(action chunk 第 1 步,action_dim=6)====
[-85.7613 95.7447 -86.5018 85.9018 -5.729 23.7636]
==== 状态: SUCCESS ====执行:
./venv/bin/python run_test_cases.py(单进程加载模型一次,批量推理,约 20s)。 输出表:action_first_step为动作块第 1 步 6 维关节控制量;mean_abs/std/min/max为整段动作块统计;infer为单组推理耗时。 每组用例均可通过--task --state --obj-pos --seed --num-steps以inference.py独立复现(同参数输出一致)。
| # | 任务指令 | 本体状态 state | 物体 obj | seed | 步数 | 第 1 步动作(6 维关节控制量) | mean_abs | infer(s) |
|---|---|---|---|---|---|---|---|---|
| 01 | put the annotated object in the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -25.55, 37.09, -29.29, 34.89, 0.74, 3.59 | 43.04 | 0.59 |
| 02 | put the annotated object in the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -87.81, 96.28, -91.30, 90.99, -8.02, 23.35 | 71.27 | 0.29 |
| 03 | put the annotated object in the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -84.49, 95.38, -86.32, 87.10, -5.25, 19.49 | 68.84 | 0.30 |
| 04 | put the annotated object in the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -88.95, 94.37, -93.83, 91.85, -9.33, 18.56 | 62.05 | 0.30 |
| 05 | put the annotated object in the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.28, 93.45, -83.70, 84.77, -7.53, 22.54 | 70.91 | 0.29 |
| 06 | place the object into the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -27.58, 37.22, -34.53, 39.37, -1.41, 2.96 | 44.88 | 0.30 |
| 07 | place the object into the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.54, 96.23, -90.37, 88.84, -9.20, 24.04 | 71.78 | 0.29 |
| 08 | place the object into the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -84.95, 94.57, -86.04, 86.24, -6.22, 17.07 | 68.15 | 0.30 |
| 09 | place the object into the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -88.67, 92.69, -92.24, 89.72, -10.21, 15.07 | 65.21 | 0.30 |
| 10 | place the object into the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.73, 93.09, -83.30, 83.67, -8.25, 21.73 | 71.21 | 0.30 |
| 11 | grasp the red block and put it in the container | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -33.67, 37.87, -28.98, 33.73, 0.35, 4.04 | 45.66 | 0.30 |
| 12 | grasp the red block and put it in the container | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -87.26, 95.57, -90.56, 90.68, -7.82, 23.84 | 68.90 | 0.30 |
| 13 | grasp the red block and put it in the container | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -83.78, 93.94, -87.10, 89.28, -5.27, 18.36 | 61.25 | 0.30 |
| 14 | grasp the red block and put it in the container | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -86.85, 91.22, -91.09, 91.02, -8.69, 14.16 | 54.20 | 0.30 |
| 15 | grasp the red block and put it in the container | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -87.26, 92.11, -83.16, 85.29, -7.14, 23.88 | 67.24 | 0.29 |
| 16 | pick up the object and drop it in the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -28.35, 35.30, -31.32, 36.75, -1.24, 6.29 | 43.14 | 0.30 |
| 17 | pick up the object and drop it in the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.65, 94.93, -89.31, 89.73, -8.04, 24.36 | 70.07 | 0.30 |
| 18 | pick up the object and drop it in the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -85.16, 94.06, -85.79, 87.98, -5.26, 20.78 | 64.88 | 0.30 |
| 19 | pick up the object and drop it in the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.28, 91.53, -90.61, 90.46, -8.95, 17.74 | 56.02 | 0.29 |
| 20 | pick up the object and drop it in the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.59, 92.13, -82.86, 85.07, -7.06, 23.94 | 69.03 | 0.29 |
| 21 | move the annotated object to the target container | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -29.88, 36.82, -29.51, 35.68, -0.28, 6.91 | 49.68 | 0.29 |
| 22 | move the annotated object to the target container | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -87.86, 97.35, -91.89, 91.59, -8.22, 25.43 | 71.10 | 0.30 |
| 23 | move the annotated object to the target container | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -84.66, 96.15, -87.41, 88.31, -5.57, 21.79 | 67.94 | 0.29 |
| 24 | move the annotated object to the target container | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.33, 95.14, -94.22, 91.78, -9.57, 19.79 | 59.14 | 0.30 |
| 25 | move the annotated object to the target container | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.32, 94.17, -84.09, 85.09, -7.73, 24.40 | 70.76 | 0.30 |
| 26 | put the object in the bin | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -26.04, 39.14, -33.47, 37.80, -1.79, 12.14 | 46.83 | 0.30 |
| 27 | put the object in the bin | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.64, 95.77, -90.91, 90.37, -8.38, 24.10 | 71.45 | 0.30 |
| 28 | put the object in the bin | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -85.41, 95.31, -86.48, 87.04, -5.65, 20.54 | 69.44 | 0.30 |
| 29 | put the object in the bin | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.95, 93.61, -93.29, 90.91, -9.53, 19.78 | 66.57 | 0.30 |
| 30 | put the object in the bin | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.91, 92.77, -83.78, 84.84, -7.62, 22.64 | 71.13 | 0.29 |
| 31 | place the object into the goal box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -28.86, 39.07, -35.59, 38.60, -0.91, 4.48 | 46.60 | 0.30 |
| 32 | place the object into the goal box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.49, 97.01, -90.87, 89.37, -8.70, 24.63 | 71.62 | 0.29 |
| 33 | place the object into the goal box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -85.05, 95.34, -86.36, 86.52, -5.76, 19.09 | 68.40 | 0.29 |
| 34 | place the object into the goal box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.02, 93.55, -92.81, 90.09, -9.97, 17.13 | 66.57 | 0.29 |
| 35 | place the object into the goal box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.55, 93.83, -83.70, 83.83, -7.92, 22.65 | 71.12 | 0.29 |
| 36 | transfer the object to the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -26.59, 34.37, -32.42, 36.72, -1.20, 1.81 | 43.06 | 0.30 |
| 37 | transfer the object to the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.37, 96.30, -91.17, 90.14, -8.57, 24.50 | 71.96 | 0.30 |
| 38 | transfer the object to the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -84.87, 94.95, -86.33, 87.02, -5.72, 20.05 | 69.30 | 0.30 |
| 39 | transfer the object to the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -88.86, 93.30, -92.89, 90.66, -9.97, 16.84 | 68.32 | 0.29 |
| 40 | transfer the object to the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.53, 93.22, -83.82, 84.69, -7.94, 23.55 | 71.44 | 0.28 |
| 41 | take the object and put it in the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -27.23, 34.75, -32.10, 36.89, -1.00, 3.73 | 42.88 | 0.29 |
| 42 | take the object and put it in the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -86.89, 96.65, -92.60, 91.90, -8.30, 22.83 | 67.10 | 0.30 |
| 43 | take the object and put it in the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -83.17, 94.45, -87.98, 89.67, -5.53, 18.00 | 56.52 | 0.29 |
| 44 | take the object and put it in the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -87.38, 92.24, -92.86, 91.96, -9.59, 15.83 | 53.67 | 0.29 |
| 45 | take the object and put it in the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -86.97, 92.90, -84.16, 85.63, -7.71, 22.32 | 63.81 | 0.30 |
| 46 | move the cube into the container | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -28.47, 39.64, -33.77, 35.77, 0.00, 5.63 | 47.36 | 0.29 |
| 47 | move the cube into the container | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.79, 97.40, -91.17, 89.07, -8.98, 26.36 | 71.93 | 0.29 |
| 48 | move the cube into the container | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -85.58, 96.19, -86.32, 85.72, -6.24, 23.67 | 69.65 | 0.31 |
| 49 | move the cube into the container | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.80, 95.07, -93.82, 89.67, -10.06, 18.11 | 69.71 | 0.30 |
| 50 | move the cube into the container | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -89.38, 93.83, -83.83, 83.57, -8.10, 24.82 | 71.54 | 0.31 |
| 51 | insert the object into the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -26.77, 35.16, -32.95, 37.16, -0.55, 4.63 | 45.15 | 0.31 |
| 52 | insert the object into the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.59, 97.19, -90.98, 89.05, -8.83, 24.53 | 72.11 | 0.31 |
| 53 | insert the object into the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -85.38, 95.94, -86.38, 86.07, -5.96, 20.41 | 69.31 | 0.30 |
| 54 | insert the object into the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -89.17, 94.06, -92.94, 89.87, -10.00, 17.29 | 69.22 | 0.30 |
| 55 | insert the object into the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.88, 93.74, -83.56, 83.66, -7.96, 23.07 | 71.52 | 0.30 |
| 56 | put the object away into the box | 0.0,0.0,0.0,0.0,0.0,0.0 | 0.4,0.35 | 42 | 10 | -23.10, 39.86, -33.82, 36.17, -0.86, 6.39 | 44.47 | 0.30 |
| 57 | put the object away into the box | -87.5,90.0,-81.0,86.5,-7.8,16.2 | 0.3,0.30 | 7 | 10 | -88.27, 96.53, -90.17, 89.23, -8.98, 24.33 | 71.26 | 0.30 |
| 58 | put the object away into the box | -85.0,92.0,-78.0,84.0,-5.0,14.0 | 0.5,0.45 | 2024 | 10 | -84.76, 96.22, -85.93, 86.04, -5.96, 19.66 | 68.96 | 0.30 |
| 59 | put the object away into the box | -90.0,88.0,-84.0,88.0,-9.0,18.0 | 0.6,0.50 | 123 | 12 | -88.99, 93.45, -92.23, 89.94, -10.26, 16.09 | 67.78 | 0.30 |
| 60 | put the object away into the box | -88.0,91.0,-80.0,85.0,-6.5,15.5 | 0.3,0.40 | 5 | 15 | -88.37, 93.79, -83.52, 83.86, -8.13, 23.07 | 71.10 | 0.30 |
(1,50,6) 正确、全部数值有限、
无 NaN/Inf;deterministic: true),去噪噪声由 --seed
显式控制,同参数可复现;测试条件:单张 Ascend 910B(
npu:0),10 步 flow-matching 去噪,3 视角 480×640 图像 + 6 维 state,bf16 骨干 / fp32 投影头混合精度(忠于原权重)。
| 指标 | 数值 |
|---|---|
| 模型加载(构造 + 权重 + 流水线) | ~27 s(首次数次运行,含算子初始化) |
| 单组推理(10 步去噪) | ~0.3–0.6 s |
| 动作 chunk 规模 | 50 步 × 6 维 |
| NPU 峰值显存 | ~1.4 GB |
| 批量 60 组用例总耗时 | ~20 s(单进程复用模型) |
backbone/ 目录提供
config + tokenizer/processor;运行脚本默认设置 HF_HUB_OFFLINE=1(见
run_test_cases.py 环境变量),不访问外网。bucketize/ searchsorted 的 bf16 输入转 fp32(inference.py 已内置,见 §3)。3inference.py 中 build_synthetic_observation 替换为
实际传感器帧即可,图像须为 (C,H,W) float32 [0,1](480×640),state 为 6 维。task(供 tokenizer 读取)与
observation.language.task/instruction 观测字段。--num-steps 默认 10(checkpoint 训练配置),增大可提升采样
质量但线性增加耗时;展示用例 C 验证了 15 步配置。--seed 显式生成;predict_action_chunk 会就地修改噪声
张量,连续调用必须传入独立拷贝(脚本已用 noise.clone())。EDQUOT
限流,脚本输出写盘已带自动重试。贡献者: z_studio | 赛道: 模型适配赛道
"SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Robotics", HuggingFace, arXiv:2506.01844。 ↩
vllm-ascend / sglang 面向自回归大语言模型,SmolVLA 为 Vision-Language-Action 策略(输出连续动作),使用 torch_npu 原生推理。 ↩ ↩2 ↩3
昇腾算子 searchsorted/bucketize 的 bf16 支持列表不含 DT_BFLOAT16,转
fp32 后结果完全等价(返回 int64 下标)。 ↩ ↩2