waylong/smolvla_libero-npu
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

SmolVLA LIBERO — Ascend NPU Adaptation

Model: HuggingFaceVLA/smolvla_libero (revision 6721902bc4d61e50a3bfdb11dfb4cb626f05d102)
Route: manipulation-vla (SmolVLA) — 2x RGB (256×256) + state(8) + language(48 tokens) → continuous action chunk [50,7]
VLM: HuggingFaceTB/SmolVLM2-500M-Instruct (SigLIP + SmolLM2 500M) + small expert (cross-attn, 10 flow-matching steps)
Task: LIBERO spatial (pick-up black bowl between plate and ramekin and place on plate)
Hardware: Ascend 910 (Ascend910_9362) — single npu:0 — CPU fallback is failure

Input / Output Contract

  • Observations: observation.images.image [3,256,256] and observation.images.image2 [3,256,256] (RGB 0-1, resized with padding to 512×512, normalized to [-1,1] for SigLIP), observation.state [8] padded to 32, observation.language.tokens [48] + mask (SmolVLM2 tokenizer, task = natural language instruction, max_length 48).
  • Output: action_chunk [50,7] (7-DoF: XYZ + quat/gripper, normalized MEAN_STD, executed horizon n_action_steps=1). Raw chunk kept as raw_output; normalized values compared for consistency.

Real Results (this build)

  • CPU first action (seed 42/123): [-0.156, 0.984, 0.176, -0.172, -0.579, 0.172, 1.039]
  • NPU first action: [-0.156, 0.997, 0.197, -0.172, -0.573, 0.169, 1.038] — shape [1,50,7], dtype float32, device npu:0
  • CPU-NPU consistency: max_abs 0.024, mean 0.0047, tol 0.135 (BF16 atol 5e-3 rtol 5e-2, max_ref 2.60), finite true — PASS via validate_policy_outputs.py
  • Performance (NPU, synchronized): warmup 3 + 10 measured — avg 581 ms, min 571 ms, max 593 ms, p50 582 ms, 86 actions/s, 1.72 chunks/s (batch 1, horizon 50, steps 10, dtype BF16 weights / FP32 compute). CPU single inference 31.8 s (no amp).
  • Device proof: npu-smi info shows Ascend910_9362, torch.npu.is_available()==True, model params/image/state/noise/output all on npu:0, timing wrapped with torch.npu.synchronize().

How to Run

pip install -r requirements.txt  # torch 2.9.0 + torch_npu 2.9.0.post1 + lerobot 0.4.4 + transformers 4.57.1
export HF_HUB_OFFLINE=1  # if local /tmp/smolvla_libero exists, otherwise downloads from HuggingFaceVLA/smolvla_libero
python inference.py

inference.py loads from /tmp/smolvla_libero if present else from HuggingFaceVLA/smolvla_libero (fixed revision), moves to npu:0, runs identical batch on CPU/NPU with fixed noise seed, prints model/revision/route/backend/device/dtype, input summary, action summaries, synchronized latency and PASS/FAIL.

NPU Adaptation Notes

  • Patches for NPU: transformers SigLIP bucketize cast to float32 (BF16 unsupported on NPU searchsorted), lerobot eager attention where condition cast to bool (Long mask not accepted on strict torch 2.9). No model weights changed, no dummy policy.
  • VLM cache: HuggingFaceTB/SmolVLM2-500M-Instruct shares blob b9bfd... with SmolVLM2-500M-Video-Instruct (same commit 7b375e), local cache symlinked — no remote code added.
  • Limits: single-card only (1.2 GB policy + 1.9 GB VLM < 64 GB HBM), trust_remote_code=False.

Evidence

  • assets/agent_workflow.png — full workflow, consistency and performance
  • assets/npu_device_call.png — npu-smi, torch.npu availability, device name and tensor devices
  • assets/model_result.png — default python inference.py result

workflow npu result