X-VLA 将「可学习软提示(soft prompt)」思想引入跨本体机器人学习,为每个不同的本体 (embodiment/domain)维护一套独立的可学习嵌入,从而在一个模型内高效利用异构机器人数据。 其架构为「Florence-2 编码器(视觉-语言表征)→ SoftPromptedTransformer(flow-matching 动作去噪)→ Action Hub(ee6d 动作空间 / 损失)」,是简洁、可扩展的通用 VLA 策略。
X-VLA-Libero 为 X-VLA 0.9B 的 Libero Edition:在 LIBERO 仿真基准(30 个任务/本体域)
上训练/微调的领域权重,action_mode=ee6d(双组 6D 位姿 + 双组 3D 位置 + 双夹爪),
num_domains=30(30 个本体域)。模型远端代码随权重仓库分发(modeling_xvla.py、
modeling_florence2.py、transformer.py、action_hub.py、processing_xvla.py 等)。
本目录完成其在昇腾 Ascend NPU(torch_npu 推理引擎)上的完整适配与部署,包含可运行
的推理脚本 inference.py、批量测试脚本 run_test_cases.py、依赖清单
requirements.txt、运行用 venv 环境、52 组实测测试用例及输出结果,以及适配过程
总结 AGENT_WORKFLOW.md。
| 组件 | 版本 |
|---|---|
| torch | 2.9.0(CPU build + torch_npu 扩展) |
| torch-npu | 2.9.0.post1+gitee7ba04 |
| transformers | 4.57.6 |
| timm | 1.0.15 |
| json-numpy | 2.1.1 |
| CANN | 8.5.1 |
| Python | 3.11.14 |
| NPU | Ascend 910B(Atlas 800T A2,64GB HBM × 2) |
| vllm-ascend / sglang | N/A(本模型不使用)1 |
模型权重路径:
| 权重 | 路径 |
|---|---|
| X-VLA-Libero(含 config / safetensors / 远端代码) | /data/models/2toINF/X-VLA-Libero |
X-VLA-Libero 为自定义 HuggingFace 架构(XVLA,model_type="xvla"),不在 vllm-ascend /
sglang 的模型注册表内(Florence-2 亦在 vllm-ascend「未支持」列表中,Issue #2259),且输出
为连续动作而非自回归 token,因此推理引擎选用 torch_npu1,通过
AutoModel.from_pretrained(..., trust_remote_code=True) 加载远端代码直接推理。
| 适配点 | CUDA 默认 | NPU 处理方式 |
|---|---|---|
| 设备分配 | model.to('cuda') | model.to('npu:0'),权重经 safetensors 加载后置于 NPU |
| 注意力实现 | 默认 sdpa | 在顶层 XVLAConfig 与嵌套 florence_config 上显式设 _attn_implementation="eager"2 |
| 权重绑定检查 | Florence2 tie_weights 自动执行 | Libero 远端代码未禁用 tie_weights(Pt 版本有禁用),须在 florence_config.text_config 上显式设 tie_word_embeddings=False,使 transformers 4.57 加载末尾跳过 get_output_embeddings()(= 已被 XVLA __init__ 删去的 lm_head)3 |
| 输入 dtype | 处理器输出 fp32 图像 | Libero 远端代码在 __init__ 中显式将 Florence2 骨干 .to(torch.float32),故 image_input / proprio 保持 fp32 与骨干权重一致(DaViT Conv2d 要求输入/权重同 dtype)4 |
| 状态/噪声 dtype | 与模型权重一致 | proprio 为 fp32,domain_id 为 int64 |
| 文本嵌入绑定 | Florence2 tie_weights 自动绑定 | 加载后将 encoder.embed_tokens 重绑到 shared(get_input_embeddings() 返回 shared) |
| 数据预处理/后处理 | XVLAProcessor(CLIPImageProcessor + BartTokenizerFast) | 复用官方处理器;ee6d 后处理由 action_space.postprocess 完成(夹爪通道 sigmoid) |
关键实现(均在 inference.py / run_test_cases.py,不修改任何三方库源码):
AutoConfig.from_pretrained 加载 XVLAConfig,并在 config._attn_implementation、
config.florence_config._attn_implementation 上显式设置 "eager";config.florence_config.text_config.tie_word_embeddings 上显式设置 False,
跳过加载末尾的权重绑定路径;AutoModel.from_pretrained(..., trust_remote_code=True) 不传 dtype= 加载权重
(作者预设 Florence2 骨干为 fp32),model.to('npu:0');加载后将
encoder.embed_tokens.weight 重绑到 shared.weight;AutoProcessor.from_pretrained(..., trust_remote_code=True) 加载 XVLAProcessor;model.generate_actions(input_ids, image_input, image_mask, domain_id, proprio, steps)
完成 flow-matching 去噪,输出 (B, num_actions=30, dim_action=20) 动作;action_space.postprocess 的 sigmoid 归一至 [0,1]。说明:
X-VLA-Pt权重中远程代码自带「禁用 tie_weights」补丁;X-VLA-Libero的远程 代码没有该补丁(diff确认),同时其__init__增加了.to(torch.float32)。 因此本目录额外应用了tie_word_embeddings=False与 fp32 输入两项适配(见第 7 节)。
# 使用清华镜像安装 Python 依赖(本目录已创建 venv,继承系统级 torch/torch-npu)
cd /opt/atomgit/model_adapt/X-VLA-Libero-NPU
./venv/bin/pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
# 或使用阿里云镜像
# ./venv/bin/pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/torch / torch-npu 需与 CANN 版本严格匹配,请使用昇腾官方安装方式(见
requirements.txt文末版本对照表)。本目录venv/使用python3 -m venv --system-site-packages venv创建,自动继承系统级 torch / torch-npu,仅需在 venv 内安装 timm / json-numpy 等纯 Python 依赖。
# 创建 venv(继承系统 torch/torch-npu/transformers 等)
python3 -m venv --system-site-packages venv
# 安装模型远端代码所需依赖(清华镜像)
./venv/bin/pip install timm==1.0.15 json-numpy==2.1.1 \
-i https://pypi.tuna.tsinghua.edu.cn/simple# 权重已本地化于 /data/models/2toINF/X-VLA-Libero(含 config.json / model.safetensors /
# modeling_xvla.py / modeling_florence2.py / transformer.py / action_hub.py /
# processing_xvla.py / tokenizer 等)
# 若需重新下载(HuggingFace 直连不可达时用 ModelScope):
# python3 -c "from modelscope import snapshot_download;
# snapshot_download('2toINF/X-VLA-Libero',
# local_dir='/data/models/2toINF/X-VLA-Libero')"cd /opt/atomgit/model_adapt/X-VLA-Libero-NPU
./venv/bin/python inference.py \
--model /data/models/2toINF/X-VLA-Libero \
--device npu:0 \
--task "pick up the red block and place it on the yellow bowl" \
--domain 0 --steps 10 --seed 42 \
--output-dir output生成的产物位于 output/:
| 文件 | 说明 |
|---|---|
action.npy | 预测动作 chunk,(1, 30, 20) float32(30 步 × 20 维 ee6d) |
report.json | 输入参数、动作统计、加载/推理耗时、NPU 显存等指标 |
--help 可查看全部参数(--task、--state 20 维、--domain 0~29、--steps、
--seed、--views、--hue、--image 真实图像路径等)。
./venv/bin/python run_test_cases.py \
--model /data/models/2toINF/X-VLA-Libero \
--device npu:0 \
--output-dir output每组输出到 output/testcase_NN/action.npy + output/testcase_NN/report.json,
全部结果汇总到 output/testcases_summary.json。--cases 1,2,3 可运行子集断点续跑。
共 52 组(≥44 组要求),模型加载一次、逐组推理,覆盖 6 类输入变化:
| 组 | 用例编号 | 变化维度 | 说明 |
|---|---|---|---|
| A | 01–30 | domain_id 0~29(全部 30 个本体域) | 验证跨本体 soft prompt 生效 |
| B | 31–40 | 10 组 LIBERO 风格语言指令 | 同一域下指令差异 |
| C | 41–44 | 4 组本体状态(20 维) | 状态差异 |
| D | 45–47 | 图像视角数 / 色偏(1/2/3 视角) | 多视角输入 |
| E | 48–50 | flow-matching 去噪步数(5/10/20) | 采样步数影响 |
| F | 51–52 | 随机种子(0 / 123) | 初始噪声随机性 |
公共基线:合成三视角棋盘格图像(hue=0.5)、语言指令
"pick up the red block and place it on the yellow bowl"、state=全零 20 维、
domain_id=0、10 步去噪、seed=42。
动作布局(ee6d):[0:3]=pos1(xyz)、[3:9]=rot1(6D)、[9]=gripper1、
[10:13]=pos2(xyz)、[13:19]=rot2(6D)、[19]=gripper2。夹爪通道经 sigmoid 归一至 [0,1]。
固定指令/状态/图像/seed=42/steps=10,仅切换 domain_id。各域输出明显不同
(mean_abs 0.100.42,夹爪 0.000.99),证明 30 组本体 soft prompt 均被正确激活。
first_step 为 action 第 1 步的完整 20 维向量(含 sigmoid 后夹爪值)。
| 用例 | 名称 | mean_abs | std | grip1 | grip2 | NaN | first_step(20 维) |
|---|---|---|---|---|---|---|---|
| 01 | domain_00 | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106,-0.4485,0.4023,-0.052,0.19,-0.1741,-0.0273,-0.174,-0.2205,0.2691,-0.1793,0,0.5685 |
| 02 | domain_01 | 0.2374 | 0.2746 | 0.4637 | 0.514 | false | -0.1272,0.0071,0.0853,0.0928,0.0821,-0.1067,-0.2197,0.3184,0.419,0.4637,-0.3409,-0.267,0.405,-0.3418,-0.2235,0.3439,0.0882,-0.0376,-0.2497,0.514 |
| 03 | domain_02 | 0.2116 | 0.2573 | 0.4293 | 0.5769 | false | -0.1155,-0.0958,0.0794,-0.0073,-0.0708,0.4177,-0.1098,-0.0795,0.2929,0.4293,0.1804,-0.0485,-0.3938,-0.1889,-0.3835,0.3494,-0.2019,-0.1271,0.1723,0.5769 |
| 04 | domain_03 | 0.1012 | 0.2372 | 0.9956 | 0 | false | -0.0648,0.0108,0.7482,-0.0458,-0.3595,0.0277,-0.3237,0.0226,0.1317,0.9956,0.0002,0.0001,0.0004,0.0005,0.0003,-0.0003,-0.0003,-0.0011,-0.0001,0 |
| 05 | domain_04 | 0.1596 | 0.2107 | 0.4314 | 0.4989 | false | 0.299,-0.0189,-0.1383,0.1026,-0.0654,0.063,-0.037,0.0256,0.066,0.4314,0.0414,-0.1789,-0.1971,0.0267,0.0903,-0.2091,-0.0016,-0.3888,0.2064,0.4989 |
| 06 | domain_05 | 0.1791 | 0.2323 | 0.4979 | 0.4806 | false | -0.0089,-0.1392,-0.045,-0.1703,0.0759,0.0811,-0.0114,-0.0019,-0.3009,0.4979,0.386,0.2704,-0.264,0.174,0.1401,0.0376,-0.2159,-0.0781,-0.1112,0.4806 |
| 07 | domain_06 | 0.2024 | 0.2343 | 0.5395 | 0.4553 | false | 0.256,0.3256,0.075,-0.3514,0.2669,-0.0158,0.2286,-0.3746,0.1243,0.5395,0.0843,0.0852,0.0667,-0.1559,-0.1194,0.0781,0.0751,0.1868,-0.027,0.4553 |
| 08 | domain_07 | 0.2108 | 0.2518 | 0.4787 | 0.5736 | false | -0.016,0.0973,0.1899,0.3775,0.2201,-0.0789,-0.208,-0.0982,-0.0723,0.4787,-0.3227,0.054,0.2244,0.2044,0.0385,-0.3122,-0.2046,-0.3134,-0.1441,0.5736 |
| 09 | domain_08 | 0.1514 | 0.1822 | 0.4809 | 0.5198 | false | 0.1535,0.2049,-0.1526,-0.0517,-0.116,0.2167,0.0344,0.1425,-0.0291,0.4809,0.2918,0.0661,0.1039,0.0563,0.2006,-0.1551,-0.043,-0.0079,-0.0459,0.5198 |
| 10 | domain_09 | 0.1565 | 0.2114 | 0.5048 | 0.5244 | false | -0.1632,-0.1798,-0.0324,-0.0951,-0.1698,0.0518,-0.0752,-0.1272,0.1274,0.5048,0.0485,-0.0683,-0.2232,0.0575,0.053,-0.0906,-0.0156,-0.3215,0.1394,0.5244 |
| 11 | domain_10 | 0.3556 | 0.46 | 0.998 | 0.0185 | false | 0.1329,-0.0282,0.3165,-0.5841,0.2492,0.1954,0.9491,-0.7261,0.0184,0.998,0.1824,0.0438,0.2983,-0.4539,-0.2977,-0.2194,0.8527,-0.677,-0.0773,0.0185 |
| 12 | domain_11 | 0.173 | 0.3184 | 0.6716 | 0 | false | 0.5615,-0.0286,0.4582,0.8603,0.0636,0.0912,-0.8481,0.0055,0.0885,0.6716,-0.0008,-0.0008,-0.0005,0.0003,-0.0002,0.0001,0.0002,-0.0001,0.0004,0 |
| 13 | domain_12 | 0.1417 | 0.279 | 0.0157 | 0 | false | -0.3074,0.3635,0.2078,0.606,-0.0223,0.0592,-0.8901,0.2301,-0.0704,0.0157,-0.0002,0.0003,-0.0009,0.0001,-0.0001,0,-0.0003,0,0.0001,0 |
| 14 | domain_13 | 0.1381 | 0.2572 | 0.1325 | 0 | false | 0.3114,-0.0201,0.2748,0.6375,0.1815,0.1329,-0.7351,0.3497,0.0138,0.1325,-0.0002,0.0005,0.0001,0.0005,0.0005,-0.0001,-0.0001,-0.0003,-0.0001,0 |
| 15 | domain_14 | 0.1353 | 0.2555 | 0.0851 | 0 | false | 0.3063,0.0606,0.2422,0.7549,-0.0085,-0.0623,-0.6252,0.1944,-0.165,0.0851,0,0,-0.0001,-0.0001,0.0002,0,-0.0004,0.0006,-0.0001,0 |
| 16 | domain_15 | 0.4237 | 0.4808 | 0.995 | 0.9996 | false | 0.4572,0.2211,0.4017,-0.7267,-0.1768,-0.4488,0.2177,-0.0766,-0.7697,0.995,0.4673,-0.1246,0.3469,0.6252,-0.5463,-0.1454,-0.3231,-0.3429,-0.3601,0.9996 |
| 17 | domain_16 | 0.3844 | 0.4357 | 0.1164 | 0.919 | false | 0.0663,-0.0635,0.0815,0.6237,0.3589,-0.344,0.7473,-0.4506,-0.2477,0.1164,0.0167,0.0229,0.1063,0.6173,-0.3863,0.4003,0.7856,-0.5014,0.2381,0.919 |
| 18 | domain_17 | 0.3132 | 0.4619 | 0.0447 | 0.0008 | false | 0.4782,-0.0352,0.4945,0.9527,0.01,0.0307,-0.9565,-0.0015,0.0399,0.0447,0.5483,0.0878,0.5801,0.9055,0.0131,0.0641,-0.8207,-0.0289,0.0808,0.0008 |
| 19 | domain_18 | 0.1824 | 0.2288 | 0.4921 | 0.4739 | false | -0.0137,0.1064,-0.0897,-0.2713,-0.0034,-0.4559,-0.2124,-0.0601,-0.2317,0.4921,-0.0148,-0.071,-0.104,-0.0979,-0.1129,-0.1953,-0.1649,0.1223,0.2935,0.4739 |
| 20 | domain_19 | 0.2065 | 0.2491 | 0.5288 | 0.477 | false | -0.1499,0.2803,-0.3516,-0.1369,0.1209,0.174,0.0427,0.1328,-0.2319,0.5288,-0.4417,-0.1855,0.1915,-0.1005,-0.0047,-0.1193,0.0383,0.1949,-0.1356,0.477 |
| 21 | domain_20 | 0.1845 | 0.2231 | 0.5377 | 0.4613 | false | -0.2764,0.0649,0.1401,-0.2075,0.2748,-0.0597,0.1971,0.0999,-0.3164,0.5377,-0.182,-0.0293,0.0199,0.1467,0.0349,-0.0702,-0.0047,0.1778,0.2614,0.4613 |
| 22 | domain_21 | 0.2088 | 0.2399 | 0.5496 | 0.451 | false | -0.2993,0.0873,-0.0522,-0.1145,-0.0759,0.2658,-0.1439,-0.1584,0.3102,0.5496,0.1965,0.141,-0.2433,0.2948,-0.302,0.1883,0.2428,-0.0813,0.0285,0.451 |
| 23 | domain_22 | 0.1962 | 0.2367 | 0.4091 | 0.4793 | false | -0.1404,-0.0156,0.3039,-0.0644,0.0224,0.3571,-0.3028,0.1284,0.1956,0.4091,0.1833,-0.2292,-0.0889,0.1494,-0.199,0.2128,-0.3897,-0.072,0.011,0.4793 |
| 24 | domain_23 | 0.1543 | 0.2129 | 0.4577 | 0.5268 | false | -0.1822,-0.3676,0.1035,-0.1801,0.0382,0.0126,0.0155,-0.044,0.0402,0.4577,0.002,-0.0833,-0.1149,-0.2567,0.1712,-0.0642,-0.1286,-0.1506,0.1504,0.5268 |
| 25 | domain_24 | 0.1901 | 0.2229 | 0.4562 | 0.5182 | false | -0.3626,-0.1267,0.3191,0.1611,-0.1592,0.1072,0.0042,-0.1596,0.0478,0.4562,0.3347,0.2758,-0.1228,0.2064,-0.0877,-0.1092,0.046,0.0173,0.0952,0.5182 |
| 26 | domain_25 | 0.2292 | 0.2744 | 0.5362 | 0.5145 | false | -0.4039,-0.1546,0.3318,-0.1689,0.0691,0.1385,0.3005,0.0979,-0.1203,0.5362,-0.0354,-0.4035,0.2629,-0.1371,0.1035,0.1478,-0.3517,0.0033,0.3132,0.5145 |
| 27 | domain_26 | 0.2161 | 0.2577 | 0.4508 | 0.5679 | false | 0.1056,-0.329,-0.1747,0.1408,-0.2786,-0.1477,-0.0442,0.169,-0.2977,0.4508,-0.288,-0.0709,-0.1627,0.3618,-0.0976,-0.0378,-0.3432,-0.0948,0.0443,0.5679 |
| 28 | domain_27 | 0.1889 | 0.2412 | 0.4731 | 0.5705 | false | 0.031,0.0192,0.3438,0.2433,-0.1056,0.0122,-0.0975,-0.3062,0.1661,0.4731,0.0011,0.31,0.0875,-0.1671,-0.0892,-0.1819,-0.0658,-0.2719,-0.0784,0.5705 |
| 29 | domain_28 | 0.2056 | 0.2494 | 0.5108 | 0.5355 | false | -0.3506,-0.1089,0.0846,0.0724,-0.1003,0.2605,0.0631,-0.2357,0.1071,0.5108,0.1178,0.3621,0.4176,-0.1192,-0.1524,-0.0521,-0.2302,-0.1899,0.058,0.5355 |
| 30 | domain_29 | 0.1858 | 0.2218 | 0.5292 | 0.4192 | false | 0.0924,0.013,0.1362,0.1221,0.1746,0.021,0.345,-0.1578,0.0497,0.5292,0.2202,-0.0153,-0.2607,-0.3592,-0.2713,-0.1051,0.1702,0.305,-0.0207,0.4191 |
固定 domain=0 / state=全零 / 3 视角 / seed=42 / steps=10,仅切换指令(LIBERO 风格)。
不同指令输出存在逐元素差异(first_step 前 8 维逐项可区分,无 NaN);由于合成图像
内容与近零状态下动作幅度整体接近,差异幅度较小,但指令嵌入均被正确编码。
| 用例 | 语言指令 | mean_abs | std | grip1 | grip2 | NaN | first_step(前 8 维) |
|---|---|---|---|---|---|---|---|
| 31 | pick up the red block and place it on the yellow bowl | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106... |
| 32 | move the blue cube to the right side | 0.1742 | 0.227 | 0.4024 | 0.5691 | false | -0.0353,-0.1328,0.2098,0.1155,-0.186,0.1337,0.0474,-0.0094... |
| 33 | open the top drawer and close the middle drawer | 0.1751 | 0.2274 | 0.4022 | 0.5683 | false | -0.0342,-0.1339,0.2084,0.1125,-0.1935,0.1329,0.0497,-0.0091... |
| 34 | put the black bowl in the bottom drawer of the cabinet | 0.1743 | 0.2271 | 0.4027 | 0.5684 | false | -0.0357,-0.1314,0.2096,0.1134,-0.1931,0.1323,0.0472,-0.0108... |
| 35 | push the plate to the front of the stove | 0.175 | 0.2277 | 0.4034 | 0.5692 | false | -0.0333,-0.1308,0.2111,0.1167,-0.1802,0.1334,0.0452,-0.0094... |
| 36 | place the mug on the plate | 0.1742 | 0.2271 | 0.4027 | 0.5684 | false | -0.0352,-0.1328,0.2087,0.1127,-0.1954,0.13,0.0457,-0.0099... |
| 37 | stack the red bowl on the blue bowl | 0.1747 | 0.2275 | 0.4024 | 0.5687 | false | -0.0349,-0.132,0.2088,0.1141,-0.1851,0.1348,0.0465,-0.0116... |
| 38 | turn on the stove and put the frying pan on it | 0.1741 | 0.2268 | 0.4028 | 0.5687 | false | -0.0335,-0.1331,0.2105,0.1144,-0.1869,0.1358,0.046,-0.0098... |
| 39 | put the pot on the stove | 0.1738 | 0.2269 | 0.4023 | 0.5684 | false | -0.0347,-0.1307,0.209,0.114,-0.1911,0.1312,0.0454,-0.0105... |
| 40 | close the top drawer | 0.1749 | 0.2274 | 0.4026 | 0.5687 | false | -0.0315,-0.1316,0.2101,0.114,-0.1889,0.1317,0.0476,-0.0112... |
固定 domain=0 / 指令 / 3 视角 / seed=42 / steps=10,仅切换 20 维 state。
| 用例 | state(前 6 维) | mean_abs | std | grip1 | grip2 | NaN | first_step(前 8 维) |
|---|---|---|---|---|---|---|---|
| 41 | 0.0,0.0,0.0,0.0,0.0,0.0,... | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106... |
| 42 | 0.1,-0.2,0.05,0.5,-0.3,0.2,... | 0.1742 | 0.227 | 0.4022 | 0.5685 | false | -0.0361,-0.1326,0.2067,0.114,-0.1946,0.1321,0.0475,-0.0105... |
| 43 | -0.1,0.2,-0.05,0.3,0.4,-0.2,... | 0.1741 | 0.2269 | 0.4024 | 0.5684 | false | -0.0357,-0.133,0.2071,0.1141,-0.1939,0.1326,0.0482,-0.0106... |
| 44 | 0.05,0.05,0.1,-0.4,0.2,0.6,... | 0.1741 | 0.2269 | 0.4024 | 0.5685 | false | -0.0359,-0.1323,0.2069,0.1141,-0.1942,0.1331,0.0476,-0.0106... |
固定 domain=0 / 指令 / state 全零 / seed=42 / steps=10,切换视角数与基准色偏 hue。
| 用例 | 视角配置 | mean_abs | std | grip1 | grip2 | NaN | first_step(前 8 维) |
|---|---|---|---|---|---|---|---|
| 45 | 1 视角(hue=0.5) | 0.175 | 0.2271 | 0.403 | 0.5677 | false | -0.0383,-0.1343,0.2063,0.1115,-0.198,0.1318,0.0501,-0.0087... |
| 46 | 2 视角(hue=0.2) | 0.1743 | 0.2269 | 0.4026 | 0.5684 | false | -0.0356,-0.1326,0.2086,0.1129,-0.1916,0.1336,0.0477,-0.0119... |
| 47 | 3 视角(hue=0.8) | 0.1744 | 0.2272 | 0.4025 | 0.5686 | false | -0.0359,-0.1325,0.2064,0.1141,-0.1947,0.1316,0.0468,-0.01... |
固定 domain=0 / 指令 / 3 视角 / seed=42,切换 flow-matching 去噪步数。步数越多耗时越高 (约线性:0.09s → 0.14s → 0.23s),输出收敛稳定(mean_abs ≈ 0.1741,无 NaN)。
| 用例 | steps | mean_abs | std | grip1 | grip2 | NaN | first_step(前 8 维) |
|---|---|---|---|---|---|---|---|
| 48 | 5 | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0357,-0.1325,0.2069,0.1143,-0.1941,0.1328,0.0475,-0.0105... |
| 49 | 10 | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1326,0.2069,0.1142,-0.1941,0.1326,0.0477,-0.0106... |
| 50 | 20 | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1326,0.2069,0.1141,-0.1941,0.1326,0.0478,-0.0106... |
固定其余输入,仅切换种子以体现初始噪声的随机性;输出受 seed 影响(逐元素可区分), 幅度较小(flow-matching 去噪在合成输入下收敛良好)。
| 用例 | seed | mean_abs | std | grip1 | grip2 | NaN | first_step(前 8 维) |
|---|---|---|---|---|---|---|---|
| 51 | 0 | 0.1741 | 0.2269 | 0.4023 | 0.5684 | false | -0.0359,-0.1327,0.2069,0.1141,-0.1942,0.1326,0.0479,-0.0105... |
| 52 | 123 | 0.1741 | 0.2269 | 0.4023 | 0.5685 | false | -0.0358,-0.1324,0.2071,0.1142,-0.1941,0.1327,0.0477,-0.0107... |
output/testcase_01/report.json(关键字段):
{
"case_id": 1,
"name": "domain_00",
"device": "npu:0",
"dtype": "torch.float32",
"task": "pick up the red block and place it on the yellow bowl",
"domain_id": 0,
"num_views": 3,
"num_steps": 10,
"seed": 42,
"action_shape": [1, 30, 20],
"action_first_step": [-0.03578900173306465, -0.13257400691509247, 0.2069229930639267,
0.11418800055980682, -0.19410400092601776, 0.1326490044593811, 0.047745998948812485,
-0.010556000284850597, -0.448525995016098, 0.40234100818634033, -0.052041999995708466,
0.19004899263381958, -0.17409199476242065, -0.027326999232172966, -0.1739799976348877,
-0.22053800523281097, 0.26911500096321106, -0.17932699620723724,
-2.2000000171829015e-05, 0.568464994430542],
"action_mean_abs": 0.1741199493408203,
"action_std": 0.22690479457378387,
"gripper_first_step": [0.402341, 0.568465],
"has_nan": false,
"infer_time_s": 0.472
}验证结论:52 组用例全部 status=SUCCESS、无 NaN;30 个 domain 软提示、语言指令、
本体状态、图像视角、去噪步数、随机种子六类输入变化均可在 Ascend 910B 上产生可区分
的合法动作输出(夹爪通道稳定落在 [0,1]),模型可完整加载并在 NPU 上完成 VLA 推理,
适配状态 SUCCESS。
测试条件:10 步 flow-matching 去噪,3 视角 224×224 图像 + 20 维 state,单张 Ascend 910B(单卡,
npu:0)。模型权重 dtype 为 fp32(Florence2 骨干由远端代码固定)。
| 指标 | 数值 |
|---|---|
| 模型加载(含 Florence-2 骨干 + 远端代码) | ~12 s |
| 单次推理(10 步去噪,含预处理,冷启动) | ~0.47 s |
| 单次推理(10 步去噪,预热后) | ~0.14 s(52 组中位数) |
| 单步去噪增量(5→10→20 步) | ~0.09 s → ~0.14 s → ~0.23 s(近线性) |
| 动作 chunk 规模 | 30 步 × 20 维 |
| NPU 峰值显存 | ~3.7 GB |
_check_and_adjust_attn_implementation,对本地 Florence2 远端代码(未定义
_supports_sdpa)会误判并报 AttributeError。inference.py 通过显式设
_attn_implementation="eager" 绕过。2X-VLA-Pt 的远程代码在 __init__
中禁用了 Florence2 的 tie_weights 钩子并覆写 XVLA.tie_weights;而
X-VLA-Libero 的远程代码没有这些补丁,transformers 4.57 加载末尾的
tie_embeddings_and_encoder_decoder 会访问被 XVLA.__init__ 删去的 lm_head
→ AttributeError: 'Florence2LanguageForConditionalGeneration' object has no attribute 'lm_head'。解法:在 florence_config.text_config 上显式设
tie_word_embeddings=False,使该绑定路径整体跳过。3__init__ 中对 Florence2 骨干执行
.to(torch.float32),因此 inference.py 不传 dtype=(加载后实际参数 dtype 为
fp32),image_input / proprio 输入亦保持 fp32(DaViT Conv2d 要求输入/权重同
dtype)。若误传 bf16 图像,会报 Input type (c10::BFloat16) and bias type (float) should be the same。4tie_weights 置空,加载后
encoder.embed_tokens 与 shared 解绑(保持随机初始化)。推理路径经
get_input_embeddings()(=shared)计算文本嵌入并直接传入 encoder,该随机模块本不
参与计算;inference.py 仍显式重绑以保证完全正确。--image cam0.jpg,cam1.jpg,cam2.jpg 传入即可(≤3 视角,PIL 自动转 RGB)。EDQUOT 限流,
inference.py 输出写盘已带自动重试。贡献者: z_studio | 赛道: 模型适配赛道
vllm-ascend / sglang 面向自回归大语言模型,X-VLA-Libero 为 Vision-Language-Action 策略(输出连续动作),使用 torch_npu 原生推理。 ↩ ↩2 ↩3
transformers 4.57.x 在 PreTrainedModel.__init__ 中对远端代码执行注意力实现
校验,需类上存在 _supports_sdpa 等属性;本地 Florence2 未定义,显式
_attn_implementation="eager" 可绕过该校验路径。 ↩ ↩2
与 X-VLA-Pt 不同,X-VLA-Libero 远端代码未禁用 Florence2 的 tie_weights 钩子
(diff 确认 Pt 有 self.vlm.tie_weights = lambda *a, **k: None 与
def tie_weights(self): return,Libero 没有)。在 florence_config.text_config
上设 tie_word_embeddings=False 使 tie_embeddings_and_encoder_decoder 跳过
get_output_embeddings()(= 已删去的 lm_head)。 ↩ ↩2
Florence2 文本嵌入以 vlm.language_model.model.shared.weight 保存,
get_input_embeddings() 返回该参数(已正确加载);encoder.embed_tokens 为
加载过程中解绑出的独立随机副本,重绑后与共享嵌入保持一致。输入 dtype 以
next(model.parameters()).dtype 实测为准(fp32)。 ↩ ↩2