jsjs1258/HumanCompatibleAI-ppo-seals-CartPole-v0-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

HumanCompatibleAI/ppo-seals-CartPole-v0 NPU Adaptation

#NPU

Model Introduction

HumanCompatibleAI/ppo-seals-CartPole-v0 is a PPO (Proximal Policy Optimization) agent trained with the RL Zoo and stable-baselines3 on the seals/CartPole-v0 environment. It learns to balance a pole on a cart by choosing between two discrete actions (push left / push right).

  • Original model: https://huggingface.co/HumanCompatibleAI/ppo-seals-CartPole-v0
  • Algo: PPO (SB3 MlpPolicy)
  • Architecture: ActorCriticPolicy (MLP 64-64) with separate actor (action_net) and critic (value_net) heads
  • Observation: 4-dim CartPole state [cart_pos, cart_vel, pole_angle, pole_ang_vel]
  • Action: 2 discrete actions (Categorical distribution)
  • Framework: PyTorch + torch_npu (Ascend 910)
  • Params: ~8.2K (FP32)

This repository adapts the real trained PPO policy to run on a single Ascend NPU (npu:0). All policy parameters and every input/output tensor are explicitly placed on npu:0; inference.py loads the real weights from the local checkpoint directory and produces the real policy action (action logits, probabilities and value estimate).

Policy / Action Contract

FieldValue
Routeclassic-rl-policy (SB3 PPO)
Observation dim4 (cart_pos, cart_vel, pole_angle, pole_ang_vel)
Action dim2 discrete (CartPole push left/right)
ActorLinear(4→64) → ReLU → Linear(64→64) → ReLU → Linear(64→2)
CriticLinear(4→64) → ReLU → Linear(64→64) → ReLU → Linear(64→1)
Outputaction logits, action probabilities, state value, deterministic action
Samplerdeterministic (argmax) for verification
Seed (training)705888933

Data Contract

The model is evaluated on fixed, deterministic CartPole observation batches (real observations from the environment space, no random initialization at verification time). inference.py uses a fixed 5-observation batch of float32 CartPole state vectors.

The weights are loaded from the local checkpoint directory (PPO_SEALS_LOCAL_DIR) if provided, otherwise downloaded from HuggingFace Hub (HF_ENDPOINT mirror) on first run.

Environment

  • Ascend 910 NPU (single card, npu:0), CANN 8.5.1 / Ascend Toolkit
  • Python 3.11, PyTorch 2.9.0, torch_npu
  • stable-baselines3 2.9.0, numpy, huggingface_hub

Installation

pip install -r requirements.txt

NPU Inference

python inference.py

To load weights from a local directory instead of downloading:

PPO_SEALS_LOCAL_DIR=/path/to/model python inference.py

Real NPU Result

All computation runs on npu:0 (Ascend910_9362). The 5-observation batch produces:

  • Action logits: shape (5, 2), dtype float32, device npu:0
  • Deterministic actions: [0, 1, 1, 0, 1]
  • State value: shape (5, 1), device npu:0

CPU-NPU Consistency (same weights, same observations)

MetricValue
Max abs diff (action logits)2.98e-07
Max abs diff (value)3.05e-05
Toleranceatol=1e-4, rtol=1e-3
Deterministic actions matchTrue
All outputs finiteTrue
ResultPASS

Performance (npu:0, synchronized, batch=5)

MetricValue
Avg latency0.31 ms
Min / Max0.29 / 0.34 ms
p50 / p90 / p950.30 / 0.32 / 0.34 ms
Throughput~16,385 decisions/s

Evidence

  • agent workflow
  • NPU device call
  • model result

Limitations

  • This is a lightweight MLP policy (~8.2K params); throughput is dominated by host/device sync overhead rather than compute.
  • Verification uses deterministic action sampling (argmax) for reproducible CPU-NPU consistency; stochastic sampling is supported but not required for the consistency proof.
  • The replay.mp4 and rollouts.npz included in the upstream repo are demonstration artifacts and are not used for NPU inference.