HuggingFaceVLA/smolvla_libero on Ascend NPU #NPU

Model Introduction

Model Name: HuggingFaceVLA/smolvla_libero
Revision: main
Model Weight URL: https://huggingface.co/HuggingFaceVLA/smolvla_libero
Model Type: Vision-Language-Action (VLA)
Task Type: reinforcement-learning / manipulation-vla (image + state + language → continuous action_chunk)
Architecture Family: SmolVLA – SmolVLM2-500M-Instruct (vision encoder + language backbone + flow-matching action expert, 10 steps)
Pipeline Tag: robotics
Weights: model.safetensors 1.2 GB, 604.93M params, dtype float32
License: Apache-2.0
Paper: SmolVLA: A Compact Efficient Vision-Language-Action Model (https://huggingface.co/papers/2506.01844) – LeRobot https://github.com/huggingface/lerobot

SmolVLA-LIBERO is the LIBERO fine-tune of SmolVLA base for LIBERO simulation. Input: 2-view RGB 256×256 (image + image2), proprio state 8-dim, language instruction. Output: 50-step continuous action chunk [B,50,7] with flow-matching denoising (10 steps, max_action_dim 32, max_state_dim 32). Trained with LeRobot on unknown dataset (base_model lerobot/smolvla_base).

Data Contract

Field	Value
Input Format	`observation.state` [B,8] float32 + `observation.images.image` [B,3,256,256] float32 + `observation.images.image2` [B,3,256,256] float32 + `task` str
Vision	2 cameras, 256×256, IDENTITY normalization, resized with padding to 512×512 internally, processor handles tokenization
Language	Instruction `Pick up the red block` → tokens [1,6] via SmolVLM2 tokenizer (max_length 48, padding longest)
State	proprio [1,8] float32 random ~ N(0,0.1) sample `[0.0336,0.0128,0.0234]`, normalized via MEAN_STD (preprocessor step5)
Output	`actions` [B,50,7] float32 raw normalized + postprocessor unnormalizer → [1,50,7]
Horizon / Chunk	50 (chunk_size), action_dim 7, n_action_steps 1
Sampler	Flow-matching, 10 diffusion steps, noise fixed seed 0 shape [1,50,32] (max_action_dim 32) deterministic CPU/NPU consistency
Dtype	float32 (model weights float32, compute float32)
Batch	1 (demo)
Preprocessing	Processor handles resize/padding, state/action normalization (MEAN_STD, eps 1e-8)

No simulation environment needed; synthetic deterministic observations ensure reproducible action chunk.

Environment

Python 3.11.14
torch 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, lerobot 0.4.4, transformers 4.57.1
huggingface_hub 0.34.4, numpy 1.26.4, Pillow 11.x, safetensors
Ascend CANN 8.5.1, torch.npu.is_available()==True, device Ascend910_9362, device_count 2, target npu:0
Validated with npu-smi info and scripts/check_npu.py

Installation

pip install -r requirements.txt

Model weights: auto-loaded from /opt/atomgit/adapt-npu-agent/working/models/smolvla_libero if present, else from https://huggingface.co/HuggingFaceVLA/smolvla_libero via snapshot_download (uses HF_ENDPOINT mirror https://hf-mirror.com). Needs 1.2 GB download.

NPU Inference

Default command (runs synthetic demo with NPU):

python inference.py

Expected output includes:

model_name, revision, route manipulation-vla, backend SmolVLA
local_dir, npu_available True, device_name Ascend910_9362, dtype torch.float32
input_features [observation.images.image, observation.images.image2, observation.state], chunk_size 50, num_steps 10
CPU baseline: action shape [1,50,7], sample, finite, min/max/mean
first_param device: npu:0, npu observation.state device: npu:0, NPU action shape [1,50,7] device: npu:0
first run (compile) latency with torch.npu.synchronize() and stable avg
Benchmark p50/p90/p95 and throughput
Consistency max_abs 0.000031 mae 0.000007 tolerance 0.002058 PASS
Final PASS

Real Results (Verified NPU)

Run: python inference.py (batch 1, 2 views 256×256, horizon 50, dim 7, float32, npu:0, seed 42/0, steps 10)

Input:

observation.state [1,8] sample [0.0336,0.0128,0.0234,0.0230, -0.0686, -0.0614, 0.0366], normalized then [0.7641,-0.1418,-1.9579,-8.5663,0.1192,...]
images [1,3,256,256] x2 random U[0,1]
language.tokens [1,6] for Pick up the red block

Output:

CPU actions [1,50,7] sample first [-1.5637,-0.0280,-0.5052,-0.0145,0.0093,1.0972,1.0003] last [-0.3730,1.4840,-0.1728,0.9703,-0.4480,0.8200,-0.8331] mean 0.135658 std 0.700425
NPU actions [1,50,7] sample first [-1.5637,-0.0280,-0.5052,-0.0145,0.0093,1.0972,1.0003] last [-0.3730,1.4840,-0.1728,0.9704,-0.4480,0.8200,-0.8331] mean 0.135664
Decoded (post) first [-0.4618,0.0762,-0.3150,-3.0e-05,0.0062,0.0803,0.9494]
finite True, shape verified, horizon 50, dim 7, bounds checked

Timing (npu:0, synchronized):

First run (compile): 795.18 ms
Stable avg: 464.13 ms (warmup 3, test 10, min 455.05 max 472.92 p50 464.68 p90 472.92)
Throughput: 107.73 actions/s, 2.15 chunks/s (50 actions per chunk)

No CPU fallback; all tensors on npu:0, model on npu:0.

Consistency Verification

Saved CPU/NPU raw actions to .json and compared:

python .opencode/skills/npu_adapt_robotics/scripts/validate_policy_outputs.py --cpu /tmp/smolvla_cpu.json --npu /tmp/smolvla_npu.json --atol 1e-4 --rtol 1e-3

Result: count 350 passed True finite True max_abs 3.09e-05 mae 7e-06 tolerance 0.002057 (fp32). Shape [1,50,7] verified, horizon/dim correct, finite. Strict FP32 pass.

Benchmark via benchmark_policy.py logic: avg 464.13 ms, 2.15 chunks/s.

Performance

Warmup 3, test 10, synchronized:

compile/first_run 795.18 ms, avg 464.13 ms, min 455.05 ms, max 472.92 ms, p50 464.68 ms, p90 472.92 ms, p95 472.92 ms
batch 1, horizon 50, act_dim 7, dtype float32, steps 10, device npu:0
throughput 107.73 actions/s (50 per chunk at 7 dim), 2.15 chunks/s
Includes SmolVLM2 encoding + flow-matching expert (10 steps)

Evidence

Three PNGs generated by xterm.js from real logs (render_xterm_evidence.mjs --style raw):

assets/agent_workflow.png – workflow (inspect, NPU check, inference, consistency, benchmark)
assets/npu_device_call.png – npu-smi + torch.npu availability + model param device + input/output device
assets/model_result.png – full python inference.py output

Images rendered from UTF-8 stdout/stderr, prompt atomgit@pod-a94f8701860f4700b161b00e290de466:~$ is display label only.

agent workflow npu device call model result

Limitations

Only LIBERO embodiment tested (horizon 50, 7 DoF); other embodiments not validated.
Synthetic observations (deterministic random) – not real LIBERO env rollout; closed-loop success not evaluated per skill rule 6.
Flow-matching only with fixed seed 0/42 and 10 steps; stochastic sampling not compared.
Float32 only; bfloat16/float16 not tested.
Requires 1.2 GB weights + SmolVLM2 tokenizer vocab; internet needed first run via HF mirror.