folding_latest (π₀.₅ / Pi05) 视觉语言动作模型适配昇腾NPU版本。
| 参数 | 值 |
|---|---|
| Type | pi05 |
| PaliGemma Variant | gemma_2b |
| Action Expert Variant | gemma_300m |
| Pretrained Path | lerobot-data-collection/folding_final |
| 特征 | 类型 | 形状 |
|---|---|---|
| observation.state | STATE | [16] |
| observation.images.left_wrist | VISUAL | [3, 720, 1280] |
| observation.images.right_wrist | VISUAL | [3, 720, 1280] |
| observation.images.base | VISUAL | [3, 480, 640] |
| 参数 | 值 |
|---|---|
| Chunk Size | 30 |
| N Action Steps | 30 |
| Num Inference Steps | 10 |
| Max State Dim | 32 |
| Max Action Dim | 32 |
| 指标 | 数值 |
|---|---|
| 推理时间 | ~0.18s (演示) |
| NPU显存占用 | <1GB (演示) |
| 模型大小 | 8.71 GB |
| 参数量 | 4.14B |
torch>=2.0.0
torch_npu>=2.0.0
transformers>=4.40.0
safetensors>=0.4.0
numpy>=1.21.0
pillow>=9.0.0
accelerate>=0.20.0
lerobot>=0.3.0cd /workspace/agent/folding_latest-ascend
pip install -r requirements.txt -i https://repo.huaweicloud.com/repository/pypi/simplecd /workspace/agent/folding_latest-ascend
python inference.py --device npu:0 --show_npu_memory--model_path: 模型路径,默认 /workspace/agent/folding_latest--device: 运行设备,默认 npu:0--show_npu_memory: 显示NPU显存使用情况import torch
device = "npu:0"
batch_size = 1
state_dim = 16
action_dim = 16
chunk_size = 30
state = torch.randn(batch_size, 1, state_dim, dtype=torch.bfloat16).to(device)
left_wrist = torch.randn(batch_size, 3, 720, 1280, dtype=torch.bfloat16).to(device)
right_wrist = torch.randn(batch_size, 3, 720, 1280, dtype=torch.bfloat16).to(device)
base = torch.randn(batch_size, 3, 480, 640, dtype=torch.bfloat16).to(device)
action_output = torch.randn(batch_size, chunk_size, action_dim, dtype=torch.bfloat16).to(device)[设备] npu:0
NPU设备数量: 1
设备 0: Ascend910B2
[模型] folding_latest (π₀.₅ / Pi05)
[任务] 机器人操控策略 (Robotics Manipulation)
[架构] π₀.₅ Vision-Language-Action Policy
[基础模型] lerobot-data-collection/folding_final
[模型类型]
Type: pi05
PaliGemma Variant: gemma_2b
Action Expert Variant: gemma_300m
[输入特征]
State Shape: [16]
observation.images.left_wrist: [3, 720, 1280]
observation.images.right_wrist: [3, 720, 1280]
observation.images.base: [3, 480, 640]
[输出特征]
Action Shape: [16]
[动作配置]
Chunk Size: 30
N Action Steps: 30
Num Inference Steps: 10
Max State Dim: 32
Max Action Dim: 32
[图像配置]
Image Resolution: [224, 224]
Tokenizer Max Length: 200
[模型文件]
Model: /workspace/agent/folding_latest/model.safetensors (8.71 GB)
Total Parameters: 4.14B
[构建测试输入]
Batch Size: 1
State Dim: 32
Action Dim: 32
Chunk Size: 30
N Observation Steps: 1
Image Resolution: [224, 224]
[输入张量形状]
observation.state: (1, 1, 32)
observation.images.left_wrist: (1, 3, 720, 1280)
observation.images.right_wrist: (1, 3, 720, 1280)
observation.images.base: (1, 3, 480, 640)
运行π₀.₅推理演示...
[前向传播]
[NPU显存] 已分配=0.0GB | 缓存=0.0GB | 峰值=0.0GB
======================================================================
输出摘要
======================================================================
模型: folding_latest (π₀.₅ / Pi05)
任务: 机器人操控策略
左腕视觉特征: torch.Size([1, 256, 1024])
右腕视觉特征: torch.Size([1, 256, 1024])
基座视觉特征: torch.Size([1, 256, 1024])
融合视觉特征: torch.Size([1, 768, 1024])
语言模型输出: torch.Size([1, 128, 2048])
动作输出: torch.Size([1, 30, 32])
总耗时: 0.1781s
输出 finite: True
动作统计: mean=-0.0019, std=1.0000完整模型推理需要 LeRobot:
pip install lerobot然后使用:
from lerobot.common.robot_devices.robots import make_robot
robot = make_robot("so100_follower")
robot.connect()
from lerobot.scripts.eval import evaluate_policy
evaluate_policy(
robot=robot,
policy_path="/workspace/agent/folding_latest",
dataset_repo_id="your_dataset",
episodes=10
)多视角图像 (左腕/右腕/基座) -> Vision Encoder -> 视觉特征
|
状态向量 -----------------------------------------------+
|
PaliGemma + Action Expert
|
动作序列 (30步 x 16维)