waylong/smolvla-libero-npu-bb9f6ab10b
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

SmolVLA LIBERO - Ascend NPU Adapted

Compact vision-language-action model for LIBERO manipulation, adapted to Huawei Ascend 910B NPU.

Base model: lerobot/smolvla_libero https://huggingface.co/lerobot/smolvla_libero (revision 31d453f7edd78c839a8bbc39744a292686daf0de)
Route: manipulation-vla | Task: language-conditioned manipulation (LIBERO)
VLM backbone: HuggingFaceTB/SmolVLM2-500M-Video-Instruct (tokenizer 48, num_vlm_layers 16, cross_attn)
Weights: 865 MB model.safetensors + 500M VLM, served via HuggingFace Hub

Observation / Action Contract

Input (batch=1):

  • observation.state 6-dim (joint + gripper, normalized MEAN_STD, padded to max_state_dim 32)
  • observation.images.camera1/2/3 each 3×256×256 (resized with padding to 512×512, IDENTITY norm)
  • task language instruction tokenized to 48 tokens (observation.language.tokens + observation.language.attention_mask bool)

Output:

  • action_chunk shape (1, 50, 7) float32, horizon 50, action_dim 7 (delta joint + gripper), normalized with MEAN_STD
  • Flow matching sampler: num_steps=10, chunk_size=50, n_action_steps=50, max_action_dim 32 (padded, slice to 7)

NPU Adaptation Details

  • Device: npu:0 Ascend910_9362, torch_npu 2.9.0.post1, torch 2.9.0
  • Dtype: float32 for NPU compatibility (BF16 bucketize not supported on NPU, switched from BF16 to FP32, CPU and NPU share same dtype/weights/noise/seed)
  • All tensors forced to npu:0 via .to("npu:0"); first param device npu:0, input images/state/tokens on npu:0, sampler noise on npu:0, output on npu:0
  • No dummy policy, no CPU fallback for core matmuls; only NPU inference counted as PASS
  • Deterministic seed 0 for flow noise, task "put the pink block on the plate"

Real Results (single run)

Device proof:

  • Input tensors on npu:0 (state, 3 cameras, tokens, attention_mask)
  • First param device npu:0, sampler noise on npu:0, output on npu:0, torch.npu.synchronize() timed

CPU vs NPU consistency (same weights, same observation, same fixed noise 1×50×32, seed 0):

  • max_abs 0.000146, mean_abs ~6.9e-06, max_ref ~1.82, tol 0.00192 (atol 1e-4 + rtol 1e-3 * max_ref) → PASS
  • Finite check PASS, shape (1,50,7) PASS, horizon 50, action_dim 7
  • Relaxed fp16 tolerance 0.096 would also PASS, but strict FP32 tolerance already passes

Performance (10 timed runs after 3 warmup, with synchronize):

  • avg ~327 ms, min ~292 ms, max ~358 ms, p50/p90/p95 reported in log; ~152 actions/s, ~3.05 chunks/s (horizon 50, batch 1, 512×512 padded images, flow steps 10)
  • Hardware: 2×Ascend910, single card npu:0 used, no compile, no quantization

Limitations

  • Requires Ascend NPU with torch_npu and lerobot 0.4.4, transformers 4.44.2; CPU fallback is not a success claim
  • Images must be 256×256 RGB in [0,1], state 6-dim, task non-empty string; empty state not supported
  • Real robot rollout not included; success is single action-chunk numeric proof, not closed-loop reward

Inference

pip install -r requirements.txt
python inference.py

Default prints model/revision, route, backend, device, dtype, input summary, CPU/NPU action summaries, sync latency, PASS/FAIL.

Evidence

agent workflow npu device call model result