jsjs1258/Nablaaa-ppo-Pyramids-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

Nablaaa/ppo-Pyramids NPU Adaptation

Model Introduction

Nablaaa/ppo-Pyramids is a PPO (Proximal Policy Optimization) agent trained with Unity ML-Agents on the Pyramids environment. The policy maps a 172-dim vector observation (56+56+56+4) to a discrete 5-way single-branch action.

  • Original model: https://huggingface.co/Nablaaa/ppo-Pyramids
  • Revision: 51afdc57fa90b739589ca4b9082ac738191eb606
  • Architecture: Concat 172 -> Linear 172×512 Swish -> Linear 512×512 Swish -> Linear 512×5 + masked Softmax + ArgMax/Multinomial
  • Observation space: obs_0 56 + obs_1 56 + obs_2 56 + obs_3 4 = 172 float32 (no normalizer)
  • Action space: discrete 5 (action_masks all-ones, deterministic = argmax)
  • Params: ~355K FP32 (~1.4 MB)
  • Checkpoint: Pyramids/Pyramids-3000009.pt (3,000,009 steps, also available as Pyramids-3000009.onnx)
  • Framework: PyTorch + torch_npu on Ascend 910

This repository adapts the real PPO policy to run on a single Ascend NPU (npu:0). All model parameters and every observation/action tensor are placed on npu:0; inference.py loads the real weights from HF cache and produces real logits/probs and deterministic discrete actions.

Policy / Action Contract

FieldValue
Routeoffline-rl / classic-rl-policy (ML-Agents PPO Pyramids discrete)
Observation dim172 (56+56+56+4)
Action dim5 discrete (0..4 single branch)
Hidden512×2, Swish = x·sigmoid(x)
Outputlogits[5], probs=softmax(masked_logits), deterministic=argmax
Horizon1 decision per forward
Masks[1,1,1,1,1]

Environment

  • Ascend 910 NPU (single card npu:0, CANN 8.5.1)
  • Python 3.11, PyTorch 2.9.0, torch_npu 2.9.0, huggingface_hub, onnx, numpy
  • Device name: Ascend910_9362, torch.npu.is_available() = True

Installation

pip install -r requirements.txt

NPU Inference

python inference.py --device npu
# or CPU
python inference.py --device cpu
# auto-select NPU if available
python inference.py

Weights are loaded from /data/adapt-npu-agent/hf/hub/models--Nablaaa--ppo-Pyramids/snapshots/51afdc57fa90b739589ca4b9082ac738191eb606/Pyramids/Pyramids-3000009.pt if cached, otherwise via huggingface_hub (HF_ENDPOINT=https://hf-mirror.com).

Real NPU Result

Executed on npu:0 (Ascend910_9362) with synchronized timing (torch.npu.synchronize() around forward):

  • Input batch=1 (seed 0): obs0_mean 0.0014, obs3 [0.498,-0.523,-0.251,-1.055]
  • Logits shape (1,5) device npu:0 dtype float32: [-1.94938, 19.62939, -14.29364, -7.64803, -2.16591]
  • Probs: [4.25e-10, 1.0, 1.85e-15, 1.42e-12, 3.42e-10]
  • Deterministic action: [1]
  • Latency avg 0.974 ms, min 0.933 ms, max 1.008 ms, p50 0.982 ms, throughput 1026 decisions/s (batch 1, includes post-sync)

Batch 5 Verification (used for CPU-NPU consistency)

Inputs: 5 varied observations ([0.1]*172 etc., see validate_npu.py)

  • CPU logits first row: [-1.63797, 2.07002, -3.21489, 0.55549, 0.40300] -> det [1,4,1,4,1]
  • NPU logits first row: [-1.63797, 2.07002, -3.21489, 0.55549, 0.40300] -> det [1,4,1,4,1]

CPU-NPU Consistency (same weights, same observations, same sampler, seed 42)

MetricValue
Max abs diff (logits)2.15e-06
Mean abs diff5.50e-07
Tolerance1e-4 + 1e-3*max_ref = 6.89e-03
FiniteTrue
Actions matchTrue (all 5 decisions)
ResultPASS

Assets

Evidence screenshots rendered via xterm.js from real logs:

  • assets/agent_workflow.png – workflow, model discovery, CPU/NPU validation steps
  • assets/npu_device_call.png – npu-smi info + torch.npu.is_available() + device proof
  • assets/model_result.png – python inference.py --device npu real output on npu:0

agent workflow npu device call model result

Limitations

  • Single-card FP32 only; no quantization.
  • Observation must be 172-dim float32; action is discrete single-step (horizon 1), not trajectory.

References

  • Original Hub: https://huggingface.co/Nablaaa/ppo-Pyramids
  • Unity ML-Agents: https://github.com/Unity-Technologies/ml-agents