z
z_studio/X-VLA-Libero-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

2toINF/X-VLA-Libero on Ascend NPU

1. 简介

  • 模型来源: 2toINF/X-VLA-Libero(HuggingFace Hub)
  • 模型类型: Vision-Language-Action(视觉-语言-动作)策略,flow-matching 训练,X-VLA 0.9B 的 Libero Edition(面向 LIBERO 仿真基准的领域版权重)
  • 参数量: ≈ 0.9B(Florence-2-large encoder-only 骨干 + 24 层 SoftPromptedTransformer)
  • 输入 / 输出: 多视角图像(≤3 视角 224×224)+ 本体状态(20 维)+ 语言指令 → 连续动作(chunk=30 步 × 20 维,ee6d 动作空间)
  • VLM 骨干: microsoft/Florence-2-large(encoder-only,DaViT 视觉塔 + BART 文本编码器)
  • 论文: Zheng et al., 2025, X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model(arXiv:2510.10274)
  • 参考实现: https://github.com/2toINF/X-VLA

X-VLA 将「可学习软提示(soft prompt)」思想引入跨本体机器人学习,为每个不同的本体 (embodiment/domain)维护一套独立的可学习嵌入,从而在一个模型内高效利用异构机器人数据。 其架构为「Florence-2 编码器(视觉-语言表征)→ SoftPromptedTransformer(flow-matching 动作去噪)→ Action Hub(ee6d 动作空间 / 损失)」,是简洁、可扩展的通用 VLA 策略。

X-VLA-Libero 为 X-VLA 0.9B 的 Libero Edition:在 LIBERO 仿真基准(30 个任务/本体域) 上训练/微调的领域权重,action_mode=ee6d(双组 6D 位姿 + 双组 3D 位置 + 双夹爪), num_domains=30(30 个本体域)。模型远端代码随权重仓库分发(modeling_xvla.py、 modeling_florence2.py、transformer.py、action_hub.py、processing_xvla.py 等)。

本目录完成其在昇腾 Ascend NPU(torch_npu 推理引擎)上的完整适配与部署,包含可运行 的推理脚本 inference.py、批量测试脚本 run_test_cases.py、依赖清单 requirements.txt、运行用 venv 环境、52 组实测测试用例及输出结果,以及适配过程 总结 AGENT_WORKFLOW.md。

2. 验证环境

组件版本
torch2.9.0(CPU build + torch_npu 扩展)
torch-npu2.9.0.post1+gitee7ba04
transformers4.57.6
timm1.0.15
json-numpy2.1.1
CANN8.5.1
Python3.11.14
NPUAscend 910B(Atlas 800T A2,64GB HBM × 2)
vllm-ascend / sglangN/A(本模型不使用)1

模型权重路径:

权重路径
X-VLA-Libero(含 config / safetensors / 远端代码)/data/models/2toINF/X-VLA-Libero

3. NPU 适配说明

X-VLA-Libero 为自定义 HuggingFace 架构(XVLA,model_type="xvla"),不在 vllm-ascend / sglang 的模型注册表内(Florence-2 亦在 vllm-ascend「未支持」列表中,Issue #2259),且输出 为连续动作而非自回归 token,因此推理引擎选用 torch_npu1,通过 AutoModel.from_pretrained(..., trust_remote_code=True) 加载远端代码直接推理。

适配点CUDA 默认NPU 处理方式
设备分配model.to('cuda')model.to('npu:0'),权重经 safetensors 加载后置于 NPU
注意力实现默认 sdpa在顶层 XVLAConfig 与嵌套 florence_config 上显式设 _attn_implementation="eager"2
权重绑定检查Florence2 tie_weights 自动执行Libero 远端代码未禁用 tie_weights(Pt 版本有禁用),须在 florence_config.text_config 上显式设 tie_word_embeddings=False,使 transformers 4.57 加载末尾跳过 get_output_embeddings()(= 已被 XVLA __init__ 删去的 lm_head)3
输入 dtype处理器输出 fp32 图像Libero 远端代码在 __init__ 中显式将 Florence2 骨干 .to(torch.float32),故 image_input / proprio 保持 fp32 与骨干权重一致(DaViT Conv2d 要求输入/权重同 dtype)4
状态/噪声 dtype与模型权重一致proprio 为 fp32,domain_id 为 int64
文本嵌入绑定Florence2 tie_weights 自动绑定加载后将 encoder.embed_tokens 重绑到 shared(get_input_embeddings() 返回 shared)
数据预处理/后处理XVLAProcessor(CLIPImageProcessor + BartTokenizerFast)复用官方处理器;ee6d 后处理由 action_space.postprocess 完成(夹爪通道 sigmoid)

关键实现(均在 inference.py / run_test_cases.py,不修改任何三方库源码):

  1. AutoConfig.from_pretrained 加载 XVLAConfig,并在 config._attn_implementation、 config.florence_config._attn_implementation 上显式设置 "eager";
  2. 在 config.florence_config.text_config.tie_word_embeddings 上显式设置 False, 跳过加载末尾的权重绑定路径;
  3. AutoModel.from_pretrained(..., trust_remote_code=True) 不传 dtype= 加载权重 (作者预设 Florence2 骨干为 fp32),model.to('npu:0');加载后将 encoder.embed_tokens.weight 重绑到 shared.weight;
  4. AutoProcessor.from_pretrained(..., trust_remote_code=True) 加载 XVLAProcessor;
  5. model.generate_actions(input_ids, image_input, image_mask, domain_id, proprio, steps) 完成 flow-matching 去噪,输出 (B, num_actions=30, dim_action=20) 动作;
  6. 夹爪通道(索引 9/19)经 action_space.postprocess 的 sigmoid 归一至 [0,1]。

说明:X-VLA-Pt 权重中远程代码自带「禁用 tie_weights」补丁;X-VLA-Libero 的远程 代码没有该补丁(diff 确认),同时其 __init__ 增加了 .to(torch.float32)。 因此本目录额外应用了 tie_word_embeddings=False 与 fp32 输入两项适配(见第 7 节)。

4. 分步推理操作流程

4.1 环境依赖

# 使用清华镜像安装 Python 依赖(本目录已创建 venv,继承系统级 torch/torch-npu)
cd /opt/atomgit/model_adapt/X-VLA-Libero-NPU
./venv/bin/pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
# 或使用阿里云镜像
# ./venv/bin/pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/

torch / torch-npu 需与 CANN 版本严格匹配,请使用昇腾官方安装方式(见 requirements.txt 文末版本对照表)。本目录 venv/ 使用 python3 -m venv --system-site-packages venv 创建,自动继承系统级 torch / torch-npu,仅需在 venv 内安装 timm / json-numpy 等纯 Python 依赖。

4.2 创建运行环境(首次)

# 创建 venv(继承系统 torch/torch-npu/transformers 等)
python3 -m venv --system-site-packages venv

# 安装模型远端代码所需依赖(清华镜像)
./venv/bin/pip install timm==1.0.15 json-numpy==2.1.1 \
    -i https://pypi.tuna.tsinghua.edu.cn/simple

4.3 准备模型权重

# 权重已本地化于 /data/models/2toINF/X-VLA-Libero(含 config.json / model.safetensors /
# modeling_xvla.py / modeling_florence2.py / transformer.py / action_hub.py /
# processing_xvla.py / tokenizer 等)
# 若需重新下载(HuggingFace 直连不可达时用 ModelScope):
#   python3 -c "from modelscope import snapshot_download;
#       snapshot_download('2toINF/X-VLA-Libero',
#           local_dir='/data/models/2toINF/X-VLA-Libero')"

4.4 运行推理(单次)

cd /opt/atomgit/model_adapt/X-VLA-Libero-NPU
./venv/bin/python inference.py \
    --model /data/models/2toINF/X-VLA-Libero \
    --device npu:0 \
    --task "pick up the red block and place it on the yellow bowl" \
    --domain 0 --steps 10 --seed 42 \
    --output-dir output

生成的产物位于 output/:

文件说明
action.npy预测动作 chunk,(1, 30, 20) float32(30 步 × 20 维 ee6d)
report.json输入参数、动作统计、加载/推理耗时、NPU 显存等指标

--help 可查看全部参数(--task、--state 20 维、--domain 0~29、--steps、 --seed、--views、--hue、--image 真实图像路径等)。

4.5 批量测试用例(52 组)

./venv/bin/python run_test_cases.py \
    --model /data/models/2toINF/X-VLA-Libero \
    --device npu:0 \
    --output-dir output

每组输出到 output/testcase_NN/action.npy + output/testcase_NN/report.json, 全部结果汇总到 output/testcases_summary.json。--cases 1,2,3 可运行子集断点续跑。

5. 测试用例与输出结果

5.1 用例设计

共 52 组(≥44 组要求),模型加载一次、逐组推理,覆盖 6 类输入变化:

组用例编号变化维度说明
A01–30domain_id 0~29(全部 30 个本体域)验证跨本体 soft prompt 生效
B31–4010 组 LIBERO 风格语言指令同一域下指令差异
C41–444 组本体状态(20 维)状态差异
D45–47图像视角数 / 色偏(1/2/3 视角)多视角输入
E48–50flow-matching 去噪步数(5/10/20)采样步数影响
F51–52随机种子(0 / 123)初始噪声随机性

公共基线:合成三视角棋盘格图像(hue=0.5)、语言指令 "pick up the red block and place it on the yellow bowl"、state=全零 20 维、 domain_id=0、10 步去噪、seed=42。

动作布局(ee6d):[0:3]=pos1(xyz)、[3:9]=rot1(6D)、[9]=gripper1、 [10:13]=pos2(xyz)、[13:19]=rot2(6D)、[19]=gripper2。夹爪通道经 sigmoid 归一至 [0,1]。

5.2 组 A:domain_id 0~29(30 组)

固定指令/状态/图像/seed=42/steps=10,仅切换 domain_id。各域输出明显不同 (mean_abs 0.100.42,夹爪 0.000.99),证明 30 组本体 soft prompt 均被正确激活。 first_step 为 action 第 1 步的完整 20 维向量(含 sigmoid 后夹爪值)。

用例名称mean_absstdgrip1grip2NaNfirst_step(20 维)
01domain_000.17410.22690.40230.5685false-0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106,-0.4485,0.4023,-0.052,0.19,-0.1741,-0.0273,-0.174,-0.2205,0.2691,-0.1793,0,0.5685
02domain_010.23740.27460.46370.514false-0.1272,0.0071,0.0853,0.0928,0.0821,-0.1067,-0.2197,0.3184,0.419,0.4637,-0.3409,-0.267,0.405,-0.3418,-0.2235,0.3439,0.0882,-0.0376,-0.2497,0.514
03domain_020.21160.25730.42930.5769false-0.1155,-0.0958,0.0794,-0.0073,-0.0708,0.4177,-0.1098,-0.0795,0.2929,0.4293,0.1804,-0.0485,-0.3938,-0.1889,-0.3835,0.3494,-0.2019,-0.1271,0.1723,0.5769
04domain_030.10120.23720.99560false-0.0648,0.0108,0.7482,-0.0458,-0.3595,0.0277,-0.3237,0.0226,0.1317,0.9956,0.0002,0.0001,0.0004,0.0005,0.0003,-0.0003,-0.0003,-0.0011,-0.0001,0
05domain_040.15960.21070.43140.4989false0.299,-0.0189,-0.1383,0.1026,-0.0654,0.063,-0.037,0.0256,0.066,0.4314,0.0414,-0.1789,-0.1971,0.0267,0.0903,-0.2091,-0.0016,-0.3888,0.2064,0.4989
06domain_050.17910.23230.49790.4806false-0.0089,-0.1392,-0.045,-0.1703,0.0759,0.0811,-0.0114,-0.0019,-0.3009,0.4979,0.386,0.2704,-0.264,0.174,0.1401,0.0376,-0.2159,-0.0781,-0.1112,0.4806
07domain_060.20240.23430.53950.4553false0.256,0.3256,0.075,-0.3514,0.2669,-0.0158,0.2286,-0.3746,0.1243,0.5395,0.0843,0.0852,0.0667,-0.1559,-0.1194,0.0781,0.0751,0.1868,-0.027,0.4553
08domain_070.21080.25180.47870.5736false-0.016,0.0973,0.1899,0.3775,0.2201,-0.0789,-0.208,-0.0982,-0.0723,0.4787,-0.3227,0.054,0.2244,0.2044,0.0385,-0.3122,-0.2046,-0.3134,-0.1441,0.5736
09domain_080.15140.18220.48090.5198false0.1535,0.2049,-0.1526,-0.0517,-0.116,0.2167,0.0344,0.1425,-0.0291,0.4809,0.2918,0.0661,0.1039,0.0563,0.2006,-0.1551,-0.043,-0.0079,-0.0459,0.5198
10domain_090.15650.21140.50480.5244false-0.1632,-0.1798,-0.0324,-0.0951,-0.1698,0.0518,-0.0752,-0.1272,0.1274,0.5048,0.0485,-0.0683,-0.2232,0.0575,0.053,-0.0906,-0.0156,-0.3215,0.1394,0.5244
11domain_100.35560.460.9980.0185false0.1329,-0.0282,0.3165,-0.5841,0.2492,0.1954,0.9491,-0.7261,0.0184,0.998,0.1824,0.0438,0.2983,-0.4539,-0.2977,-0.2194,0.8527,-0.677,-0.0773,0.0185
12domain_110.1730.31840.67160false0.5615,-0.0286,0.4582,0.8603,0.0636,0.0912,-0.8481,0.0055,0.0885,0.6716,-0.0008,-0.0008,-0.0005,0.0003,-0.0002,0.0001,0.0002,-0.0001,0.0004,0
13domain_120.14170.2790.01570false-0.3074,0.3635,0.2078,0.606,-0.0223,0.0592,-0.8901,0.2301,-0.0704,0.0157,-0.0002,0.0003,-0.0009,0.0001,-0.0001,0,-0.0003,0,0.0001,0
14domain_130.13810.25720.13250false0.3114,-0.0201,0.2748,0.6375,0.1815,0.1329,-0.7351,0.3497,0.0138,0.1325,-0.0002,0.0005,0.0001,0.0005,0.0005,-0.0001,-0.0001,-0.0003,-0.0001,0
15domain_140.13530.25550.08510false0.3063,0.0606,0.2422,0.7549,-0.0085,-0.0623,-0.6252,0.1944,-0.165,0.0851,0,0,-0.0001,-0.0001,0.0002,0,-0.0004,0.0006,-0.0001,0
16domain_150.42370.48080.9950.9996false0.4572,0.2211,0.4017,-0.7267,-0.1768,-0.4488,0.2177,-0.0766,-0.7697,0.995,0.4673,-0.1246,0.3469,0.6252,-0.5463,-0.1454,-0.3231,-0.3429,-0.3601,0.9996
17domain_160.38440.43570.11640.919false0.0663,-0.0635,0.0815,0.6237,0.3589,-0.344,0.7473,-0.4506,-0.2477,0.1164,0.0167,0.0229,0.1063,0.6173,-0.3863,0.4003,0.7856,-0.5014,0.2381,0.919
18domain_170.31320.46190.04470.0008false0.4782,-0.0352,0.4945,0.9527,0.01,0.0307,-0.9565,-0.0015,0.0399,0.0447,0.5483,0.0878,0.5801,0.9055,0.0131,0.0641,-0.8207,-0.0289,0.0808,0.0008
19domain_180.18240.22880.49210.4739false-0.0137,0.1064,-0.0897,-0.2713,-0.0034,-0.4559,-0.2124,-0.0601,-0.2317,0.4921,-0.0148,-0.071,-0.104,-0.0979,-0.1129,-0.1953,-0.1649,0.1223,0.2935,0.4739
20domain_190.20650.24910.52880.477false-0.1499,0.2803,-0.3516,-0.1369,0.1209,0.174,0.0427,0.1328,-0.2319,0.5288,-0.4417,-0.1855,0.1915,-0.1005,-0.0047,-0.1193,0.0383,0.1949,-0.1356,0.477
21domain_200.18450.22310.53770.4613false-0.2764,0.0649,0.1401,-0.2075,0.2748,-0.0597,0.1971,0.0999,-0.3164,0.5377,-0.182,-0.0293,0.0199,0.1467,0.0349,-0.0702,-0.0047,0.1778,0.2614,0.4613
22domain_210.20880.23990.54960.451false-0.2993,0.0873,-0.0522,-0.1145,-0.0759,0.2658,-0.1439,-0.1584,0.3102,0.5496,0.1965,0.141,-0.2433,0.2948,-0.302,0.1883,0.2428,-0.0813,0.0285,0.451
23domain_220.19620.23670.40910.4793false-0.1404,-0.0156,0.3039,-0.0644,0.0224,0.3571,-0.3028,0.1284,0.1956,0.4091,0.1833,-0.2292,-0.0889,0.1494,-0.199,0.2128,-0.3897,-0.072,0.011,0.4793
24domain_230.15430.21290.45770.5268false-0.1822,-0.3676,0.1035,-0.1801,0.0382,0.0126,0.0155,-0.044,0.0402,0.4577,0.002,-0.0833,-0.1149,-0.2567,0.1712,-0.0642,-0.1286,-0.1506,0.1504,0.5268
25domain_240.19010.22290.45620.5182false-0.3626,-0.1267,0.3191,0.1611,-0.1592,0.1072,0.0042,-0.1596,0.0478,0.4562,0.3347,0.2758,-0.1228,0.2064,-0.0877,-0.1092,0.046,0.0173,0.0952,0.5182
26domain_250.22920.27440.53620.5145false-0.4039,-0.1546,0.3318,-0.1689,0.0691,0.1385,0.3005,0.0979,-0.1203,0.5362,-0.0354,-0.4035,0.2629,-0.1371,0.1035,0.1478,-0.3517,0.0033,0.3132,0.5145
27domain_260.21610.25770.45080.5679false0.1056,-0.329,-0.1747,0.1408,-0.2786,-0.1477,-0.0442,0.169,-0.2977,0.4508,-0.288,-0.0709,-0.1627,0.3618,-0.0976,-0.0378,-0.3432,-0.0948,0.0443,0.5679
28domain_270.18890.24120.47310.5705false0.031,0.0192,0.3438,0.2433,-0.1056,0.0122,-0.0975,-0.3062,0.1661,0.4731,0.0011,0.31,0.0875,-0.1671,-0.0892,-0.1819,-0.0658,-0.2719,-0.0784,0.5705
29domain_280.20560.24940.51080.5355false-0.3506,-0.1089,0.0846,0.0724,-0.1003,0.2605,0.0631,-0.2357,0.1071,0.5108,0.1178,0.3621,0.4176,-0.1192,-0.1524,-0.0521,-0.2302,-0.1899,0.058,0.5355
30domain_290.18580.22180.52920.4192false0.0924,0.013,0.1362,0.1221,0.1746,0.021,0.345,-0.1578,0.0497,0.5292,0.2202,-0.0153,-0.2607,-0.3592,-0.2713,-0.1051,0.1702,0.305,-0.0207,0.4191

5.3 组 B:语言指令(10 组)

固定 domain=0 / state=全零 / 3 视角 / seed=42 / steps=10,仅切换指令(LIBERO 风格)。 不同指令输出存在逐元素差异(first_step 前 8 维逐项可区分,无 NaN);由于合成图像 内容与近零状态下动作幅度整体接近,差异幅度较小,但指令嵌入均被正确编码。

用例语言指令mean_absstdgrip1grip2NaNfirst_step(前 8 维)
31pick up the red block and place it on the yellow bowl0.17410.22690.40230.5685false-0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106...
32move the blue cube to the right side0.17420.2270.40240.5691false-0.0353,-0.1328,0.2098,0.1155,-0.186,0.1337,0.0474,-0.0094...
33open the top drawer and close the middle drawer0.17510.22740.40220.5683false-0.0342,-0.1339,0.2084,0.1125,-0.1935,0.1329,0.0497,-0.0091...
34put the black bowl in the bottom drawer of the cabinet0.17430.22710.40270.5684false-0.0357,-0.1314,0.2096,0.1134,-0.1931,0.1323,0.0472,-0.0108...
35push the plate to the front of the stove0.1750.22770.40340.5692false-0.0333,-0.1308,0.2111,0.1167,-0.1802,0.1334,0.0452,-0.0094...
36place the mug on the plate0.17420.22710.40270.5684false-0.0352,-0.1328,0.2087,0.1127,-0.1954,0.13,0.0457,-0.0099...
37stack the red bowl on the blue bowl0.17470.22750.40240.5687false-0.0349,-0.132,0.2088,0.1141,-0.1851,0.1348,0.0465,-0.0116...
38turn on the stove and put the frying pan on it0.17410.22680.40280.5687false-0.0335,-0.1331,0.2105,0.1144,-0.1869,0.1358,0.046,-0.0098...
39put the pot on the stove0.17380.22690.40230.5684false-0.0347,-0.1307,0.209,0.114,-0.1911,0.1312,0.0454,-0.0105...
40close the top drawer0.17490.22740.40260.5687false-0.0315,-0.1316,0.2101,0.114,-0.1889,0.1317,0.0476,-0.0112...

5.4 组 C:本体状态(4 组)

固定 domain=0 / 指令 / 3 视角 / seed=42 / steps=10,仅切换 20 维 state。

用例state(前 6 维)mean_absstdgrip1grip2NaNfirst_step(前 8 维)
410.0,0.0,0.0,0.0,0.0,0.0,...0.17410.22690.40230.5685false-0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106...
420.1,-0.2,0.05,0.5,-0.3,0.2,...0.17420.2270.40220.5685false-0.0361,-0.1326,0.2067,0.114,-0.1946,0.1321,0.0475,-0.0105...
43-0.1,0.2,-0.05,0.3,0.4,-0.2,...0.17410.22690.40240.5684false-0.0357,-0.133,0.2071,0.1141,-0.1939,0.1326,0.0482,-0.0106...
440.05,0.05,0.1,-0.4,0.2,0.6,...0.17410.22690.40240.5685false-0.0359,-0.1323,0.2069,0.1141,-0.1942,0.1331,0.0476,-0.0106...

5.5 组 D:图像视角 / 色偏(3 组)

固定 domain=0 / 指令 / state 全零 / seed=42 / steps=10,切换视角数与基准色偏 hue。

用例视角配置mean_absstdgrip1grip2NaNfirst_step(前 8 维)
451 视角(hue=0.5)0.1750.22710.4030.5677false-0.0383,-0.1343,0.2063,0.1115,-0.198,0.1318,0.0501,-0.0087...
462 视角(hue=0.2)0.17430.22690.40260.5684false-0.0356,-0.1326,0.2086,0.1129,-0.1916,0.1336,0.0477,-0.0119...
473 视角(hue=0.8)0.17440.22720.40250.5686false-0.0359,-0.1325,0.2064,0.1141,-0.1947,0.1316,0.0468,-0.01...

5.6 组 E:去噪步数(3 组)

固定 domain=0 / 指令 / 3 视角 / seed=42,切换 flow-matching 去噪步数。步数越多耗时越高 (约线性:0.09s → 0.14s → 0.23s),输出收敛稳定(mean_abs ≈ 0.1741,无 NaN)。

用例stepsmean_absstdgrip1grip2NaNfirst_step(前 8 维)
4850.17410.22690.40230.5685false-0.0357,-0.1325,0.2069,0.1143,-0.1941,0.1328,0.0475,-0.0105...
49100.17410.22690.40230.5685false-0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106...
50200.17410.22690.40230.5685false-0.0358,-0.1326,0.2069,0.1141,-0.1941,0.1326,0.0478,-0.0106...

5.7 组 F:随机种子(2 组)

固定其余输入,仅切换种子以体现初始噪声的随机性;输出受 seed 影响(逐元素可区分), 幅度较小(flow-matching 去噪在合成输入下收敛良好)。

用例seedmean_absstdgrip1grip2NaNfirst_step(前 8 维)
5100.17410.22690.40230.5684false-0.0359,-0.1327,0.2069,0.1141,-0.1942,0.1326,0.0479,-0.0105...
521230.17410.22690.40230.5685false-0.0358,-0.1324,0.2071,0.1142,-0.1941,0.1327,0.0477,-0.0107...

5.8 代表用例完整输出(用例 01)

output/testcase_01/report.json(关键字段):

{
  "case_id": 1,
  "name": "domain_00",
  "device": "npu:0",
  "dtype": "torch.float32",
  "task": "pick up the red block and place it on the yellow bowl",
  "domain_id": 0,
  "num_views": 3,
  "num_steps": 10,
  "seed": 42,
  "action_shape": [1, 30, 20],
  "action_first_step": [-0.03578900173306465, -0.13257400691509247, 0.2069229930639267,
      0.11418800055980682, -0.19410400092601776, 0.1326490044593811, 0.047745998948812485,
      -0.010556000284850597, -0.448525995016098, 0.40234100818634033, -0.052041999995708466,
      0.19004899263381958, -0.17409199476242065, -0.027326999232172966, -0.1739799976348877,
      -0.22053800523281097, 0.26911500096321106, -0.17932699620723724,
      -2.2000000171829015e-05, 0.568464994430542],
  "action_mean_abs": 0.1741199493408203,
  "action_std": 0.22690479457378387,
  "gripper_first_step": [0.402341, 0.568465],
  "has_nan": false,
  "infer_time_s": 0.472
}

验证结论:52 组用例全部 status=SUCCESS、无 NaN;30 个 domain 软提示、语言指令、 本体状态、图像视角、去噪步数、随机种子六类输入变化均可在 Ascend 910B 上产生可区分 的合法动作输出(夹爪通道稳定落在 [0,1]),模型可完整加载并在 NPU 上完成 VLA 推理, 适配状态 SUCCESS。

6. 性能参考

测试条件:10 步 flow-matching 去噪,3 视角 224×224 图像 + 20 维 state,单张 Ascend 910B(单卡,npu:0)。模型权重 dtype 为 fp32(Florence2 骨干由远端代码固定)。

指标数值
模型加载(含 Florence-2 骨干 + 远端代码)~12 s
单次推理(10 步去噪,含预处理,冷启动)~0.47 s
单次推理(10 步去噪,预热后)~0.14 s(52 组中位数)
单步去噪增量(5→10→20 步)~0.09 s → ~0.14 s → ~0.23 s(近线性)
动作 chunk 规模30 步 × 20 维
NPU 峰值显存~3.7 GB

7. 注意事项

  1. 推理引擎:X-VLA-Libero 为 VLA 策略,输出连续动作而非 token,非自回归 LLM,且 Florence-2 未在 vllm-ascend 支持列表中(Issue #2259),因此不适用 vllm-ascend / sglang,推理引擎为 torch_npu。1
  2. transformers 4.57.x 注意力检查:构造模型时 transformers 会执行 _check_and_adjust_attn_implementation,对本地 Florence2 远端代码(未定义 _supports_sdpa)会误判并报 AttributeError。inference.py 通过显式设 _attn_implementation="eager" 绕过。2
  3. Libero 远端代码缺少 tie_weights 禁用补丁:X-VLA-Pt 的远程代码在 __init__ 中禁用了 Florence2 的 tie_weights 钩子并覆写 XVLA.tie_weights;而 X-VLA-Libero 的远程代码没有这些补丁,transformers 4.57 加载末尾的 tie_embeddings_and_encoder_decoder 会访问被 XVLA.__init__ 删去的 lm_head → AttributeError: 'Florence2LanguageForConditionalGeneration' object has no attribute 'lm_head'。解法:在 florence_config.text_config 上显式设 tie_word_embeddings=False,使该绑定路径整体跳过。3
  4. 骨干 fp32 固定:Libero 远端代码在 __init__ 中对 Florence2 骨干执行 .to(torch.float32),因此 inference.py 不传 dtype=(加载后实际参数 dtype 为 fp32),image_input / proprio 输入亦保持 fp32(DaViT Conv2d 要求输入/权重同 dtype)。若误传 bf16 图像,会报 Input type (c10::BFloat16) and bias type (float) should be the same。4
  5. encoder.embed_tokens 重绑:X-VLA 将 Florence2 的 tie_weights 置空,加载后 encoder.embed_tokens 与 shared 解绑(保持随机初始化)。推理路径经 get_input_embeddings()(=shared)计算文本嵌入并直接传入 encoder,该随机模块本不 参与计算;inference.py 仍显式重绑以保证完全正确。
  6. 合成观测:无真实相机/机械臂时使用合成棋盘格图像。接入真实数据时,把 --image cam0.jpg,cam1.jpg,cam2.jpg 传入即可(≤3 视角,PIL 自动转 RGB)。
  7. 权重离线化:HuggingFace 直连不可达时可用 ModelScope 下载并本地化(见 4.3)。
  8. 文件写限流:本工作区为 SFS Turbo 存储,突发连续写可能偶发 EDQUOT 限流, inference.py 输出写盘已带自动重试。
  9. 精度说明:本目录验证的是可运行性(SUCCESS),不含数值精度对标;如需与 CUDA 结果逐位对比,请在 NVIDIA 环境复跑相同输入。

贡献者: z_studio | 赛道: 模型适配赛道

Footnotes

  1. vllm-ascend / sglang 面向自回归大语言模型,X-VLA-Libero 为 Vision-Language-Action 策略(输出连续动作),使用 torch_npu 原生推理。 ↩ ↩2 ↩3

  2. transformers 4.57.x 在 PreTrainedModel.__init__ 中对远端代码执行注意力实现 校验,需类上存在 _supports_sdpa 等属性;本地 Florence2 未定义,显式 _attn_implementation="eager" 可绕过该校验路径。 ↩ ↩2

  3. 与 X-VLA-Pt 不同,X-VLA-Libero 远端代码未禁用 Florence2 的 tie_weights 钩子 (diff 确认 Pt 有 self.vlm.tie_weights = lambda *a, **k: None 与 def tie_weights(self): return,Libero 没有)。在 florence_config.text_config 上设 tie_word_embeddings=False 使 tie_embeddings_and_encoder_decoder 跳过 get_output_embeddings()(= 已删去的 lm_head)。 ↩ ↩2

  4. Florence2 文本嵌入以 vlm.language_model.model.shared.weight 保存, get_input_embeddings() 返回该参数(已正确加载);encoder.embed_tokens 为 加载过程中解绑出的独立随机副本,重绑后与共享嵌入保持一致。输入 dtype 以 next(model.parameters()).dtype 实测为准(fp32)。 ↩ ↩2