z
zhangyuge147/2toINF-X-VLA-Pt-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

2toINF/X-VLA-Pt on Ascend NPU

Model Description

  • Base Model: 2toINF/X-VLA-Pt (Vision-Language-Action, Florence2-large + SoftPromptedTransformer, 0.9B, ~3.3GB FP32)
  • Revision: c1c4a64a7e03ac5b95c468bf1578f3d03651b53b
  • Task: robotics | LIBERO manipulation (manipulation-vla)
  • Route: manipulation-vla (image + language + proprio -> action_chunk)
  • Original Weight: https://huggingface.co/2toINF/X-VLA-Pt
  • Weight Files: model.safetensors 3.52GB (3519068172 bytes) FP32, 879M params
  • License: apache-2.0
  • Paper: X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment VLA (arXiv:2510.10274)
  • Ported: Transformers AutoModel (XVLA) with torch_npu, attn eager, supports npu:0 and cpu

Task Contract

  • Input: Multi-view RGB (3 views, 224x224, CLIPImageProcessor mean [0.485,0.456,0.406] std [0.229,0.224,0.225]), language instruction (BartTokenizer, max_length 50), proprio [1,20] (ee6d), domain_id [1]
  • Processor: XVLAProcessor (image_processor + tokenizer) -> input_ids [1,50], image_input [1,3,3,224,224], image_mask [1,3]
  • Output: Continuous action_chunk [1,30,20] (ee6d: xyz 6D-rotation gripper x2 arms, 30-step horizon), postprocess sigmoid on gripper channels, finite, horizon 30, action_dim 20
  • Inference: Flow-matching denoising 5 steps (linear schedule), seed 42 deterministic, torch_npu synchronized timing

NPU Adaptation

  • Single-card npu:0 execution, no CPU fallback; all float tensors on npu:0 when NPU requested
  • Verified device placement: next(model.parameters()).device, next(model.vlm.parameters()).device, next(model.transformer.parameters()).device, image_input.device, output.device all npu:0
  • Patched timm stub for DaviT and Florence2PreTrainedModel._supports_sdpa for transformers 4.57 compatibility
  • trust_remote_code=True for XVLA remote code, auditable; tie_weights patched to avoid deleted lm_head
  • Peak memory ~7GB FP32 (3.3GB weights + activations), fits single Ascend910 64GB HBM; first compile ~90ms, steady ~90ms

Inference

pip install -r requirements.txt
python inference.py                 # auto NPU if available
PYTHONPATH=/tmp/timm_stub:/tmp/xvla_pt:$PYTHONPATH python inference.py

Default loads local /tmp/xvla_pt (or HF cache fallback) and prints model/revision/route/backend/device/dtype/input summary/action summary/sync latency/PASS/FAIL. CPU-NPU consistency uses atol 1e-3 rtol 1e-3 (FP32 NPU numerical).

Real Results (Ascend910_9362, torch_npu 2.9.0.post1, float32, steps 5, batch 1)

  • NPU device: Ascend910_9362, torch.npu.is_available()=True, device_count=2, npu:0 exists
  • Model device: npu:0 (vlm npu:0, transformer npu:0)
  • Input device: npu:0 (image_input [1,3,3,224,224] float32, input_ids [1,50] int64, proprio [1,20])
  • Output device: npu:0 action_chunk [1,30,20] float32 finite True shape_ok True
  • First action (NPU): [-0.039593, -0.120886, 0.259252, 0.140380, -0.186590, ...] (20D)
  • Last action (NPU): [-0.024553, -0.114726, 0.270035, 0.132710, -0.207604, ...]
  • CPU first action: [-0.039593, -0.120886, 0.259252, 0.140380, -0.186590, ...]
  • Consistency: CPU vs NPU max_abs 0.000161 tol 0.001572 (atol 1e-3 rtol 1e-3) -> PASS
  • Latency: avg 91.22 ms, min 90.17 max 92.69, p50 91.22 p90 92.69 (10 runs, warmup 3, torch.npu.synchronize)
  • Throughput: 328.9 actions/s, 10.96 chunks/s (horizon 30, batch 1, steps 5, float32)

CPU-NPU Consistency Details

Shared weights (single model instance moved CPU->NPU, same embed_tokens random init), same deterministic images (seed i*1234), same language "pick up the red block and place it on the green bowl", same processor, same proprio/domain, seed 42, steps 5, float32. Compared flattened [600] action values. Threshold atol=1e-3 rtol=1e-3 (FP32 NPU fused ops). Also verified finite, shape [1,30,20], action_dim 20, horizon 30.

metricvalue
cpu first_action[0:5][-0.039593, -0.120886, 0.259252, 0.140380, -0.186590]
npu first_action[0:5][-0.039593, -0.120886, 0.259252, 0.140380, -0.186590]
max_abs0.000161
max_ref0.5748
tolerance0.001575
finitetrue
shape[1,30,20]
passedtrue

Performance

  • Warmup: 3 runs excluded
  • Measured: 10 runs, avg 91ms, min 90ms, max 92ms
  • Throughput: 329 actions/s, 11 chunks/s
  • Batch=1, horizon=30, steps=5, resolution 224x224, dtype float32
  • Memory: 3.3GB weights, ~7GB peak, 64GB HBM free

Environment Constraints

  • Requires torch_npu built for CANN 8.5.1, Ascend910B/910_9362, single card sufficient
  • timm stub provided at /tmp/timm_stub for DaviT DropPath/trunc_normal_; einops required
  • trust_remote_code=True, patched tie_weights and _supports_sdpa for transformers 4.57 compatibility
  • LIBERO simulation not required for single-step action_chunk validation; closed-loop rollout is extra, not replacement

Evidence

  • assets/agent_workflow.png — reconnaissance, resource check, load, NPU inference, consistency, performance
  • assets/npu_device_call.png — npu-smi, torch_npu availability, device name, model/input/output device proof
  • assets/model_result.png — default python inference.py showing model, route, action summary, latency, PASS

File Tree

inference.py
readme.md
requirements.txt
assets/agent_workflow.png
assets/npu_device_call.png
assets/model_result.png

Limitations

  • FP32 NPU has ~7e-4 numerical diff vs CPU (fused SDPA), within 1e-3 atol; FP16 would need larger tol.
  • Single dummy observation (gradient image); real LIBERO rollout needs Mujoco env, not required for weight proof.

agent_workflow npu_device_call model_result