atlasleong/ppo-cartpole-v1-npu
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

PPO agent playing CartPole-v1 — Ascend NPU inference candidate

This directory is a complete, reproducible Ascend NPU inference candidate for the pinned upstream model HumanCompatibleAI/ppo-CartPole-v1.

  • Pinned source revision: 08b61dd5dca1ee9f1a0f2d7a0fcf7740b0af5dac
  • Pinned real checkpoint: ppo-CartPole-v1.zip — 139331 bytes — SHA256 9bc70c310a4e48d5695193a20d481540ff2ace5ee7f7fa91283ed656c9a9460c
  • Architecture route: Stable-Baselines3 PPO with MlpPolicy (ActorCriticPolicy).
  • Inference engine: torch_npu (torch 2.9.0+cpu + torch_npu 2.9.0, CANN-matched).
  • Target device: physical npu:3 (Ascend910B4-1).
  • Deterministic test: observation [0, 0, 0, 0] → action / value / log_prob.

The policy/model forward runs wholly on npu:3. The CPU only orchestrates Python, deserializes the checkpoint, and preprocesses the observation. There is no CPU offload and no CPU fallback; any NPU initialization, placement mismatch, or execution error makes the process exit nonzero.

Model introduction / 模型简介与来源

FieldValue
Model IDHumanCompatibleAI/ppo-CartPole-v1
Sourcehttps://huggingface.co/HumanCompatibleAI/ppo-CartPole-v1
Pinned revision08b61dd5dca1ee9f1a0f2d7a0fcf7740b0af5dac
pipeline_tagreinforcement-learning
library_namestable-baselines3
NPU tagsascend, npu, torch_npu, hardware: NPU
Algo / policyPPO / MlpPolicy (ActorCriticPolicy, MLP [64, 64])
Observation spaceBox(-4.8…, 4.8…, (4,), float32) (CartPole-v1)
Action spaceDiscrete(2)
Reported rewardmean_reward = 500.00 +/- 0.00 (model-index, from upstream card)

License (without guessing)

The upstream model card declares no license field in its YAML front matter. This was verified directly from the Hugging Face Hub model metadata (cardData contains no license key) on 2026-08-22. Accordingly the front matter above uses license: other to denote a non-standard / undeclared license, and the upstream card's own tag set is preserved. The weights are treated as license-not-declared; the Stable-Baselines3 library itself is MIT-licensed, but that does not license the model weights. Do not assume a permissive license for redistribution without upstream confirmation.

Environment dependencies / 环境依赖清单

Tested in this container (/root/ascend-model-pipeline/jobs/rl26/ppo-cartpole-v1/ppo-cartpole-v1-npu):

ComponentVersion / value
OSLinux 5.10.0 .oe2203sp4.aarch64 (openEuler, aarch64)
Python3.11.14
CANN8.5.1 (V100R001C25SPC002B220)
torch2.9.0+cpu (image-provided, CANN-matched — do not replace)
torch_npu2.9.0 (image-provided, CANN-matched — do not replace)
stable-baselines32.7.0
gymnasium1.2.0
numpy1.26.4
cloudpickle3.1.2
npu-smi25.5.1
HardwareAscend910B4-1 (8 devices; this candidate uses physical npu:3)

requirements.txt intentionally lists only non-torch runtime dependencies. The CANN-matched torch / torch_npu pair is pre-installed in the image and must never be replaced by ordinary PyPI wheels.

Installation (Aliyun mirror only)

All pip installs below use the Aliyun PyPI mirror only:

# 1. Load the CANN environment (idempotent).
source /usr/local/Ascend/ascend-toolkit/set_env.sh

# 2. Install only the non-torch runtime dependencies from Aliyun.
pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/

Equivalently, pin the mirror for the whole shell:

export PIP_INDEX_URL=https://mirrors.aliyun.com/pypi/simple/
pip install -r requirements.txt

⚠️ Never install torch or torch_npu from PyPI. They are provided by the image as a CANN-matched pair (torch 2.9.0+cpu + torch_npu 2.9.0) and must stay untouched.

Inference steps / 分步推理操作

# 1. Load the CANN environment (required for NPU runtime).
source /usr/local/Ascend/ascend-toolkit/set_env.sh

# 2. Sanity-check the NPU devices.
npu-smi info

# 3. Run the deterministic NPU inference (device npu:3, 8 s hold window for npu-smi
#    PID correlation; log is teed to logs/npu_inference.log).
cd /root/ascend-model-pipeline/jobs/rl26/ppo-cartpole-v1/ppo-cartpole-v1-npu
HOLD_SECONDS=8 ./run_inference.sh

# 4. Inspect the structured result.
cat logs/npu_result.json

run_inference.sh sources set_env.sh, keeps the physical device mapping (ASCEND_RT_VISIBLE_DEVICES defaults to 0,1,2,3,4,5,6,7 so npu:3 == physical device 3), and exits with the Python exit code.

Complete test case and output / 完整测试用例与输出

Command:

cd /root/ascend-model-pipeline/jobs/rl26/ppo-cartpole-v1/ppo-cartpole-v1-npu && HOLD_SECONDS=8 ./run_inference.sh

Exit code: 0. PID: 3156813 (correlated live with npu-smi on physical npu:3).

Actual output (logs/npu_inference.log):

2026-08-22T08:07:48.010+00:00 === ppo-CartPole-v1 NPU inference candidate ===
2026-08-22T08:07:48.010+00:00 source_revision=08b61dd5dca1ee9f1a0f2d7a0fcf7740b0af5dac  route=Stable-Baselines3 PPO  device=npu:3  pid=3156813
2026-08-22T08:07:49.353+00:00 NPU available: devices=8 requested=npu:3 name=Ascend910B4-1
2026-08-22T08:07:49.353+00:00 artifact ok: ppo-CartPole-v1.zip bytes=139331 sha256=9bc70c310a4e48d5695193a20d481540ff2ace5ee7f7fa91283ed656c9a9460c
2026-08-22T08:07:49.438+00:00 checkpoint loaded to npu:3 in 0.0847s (deserialize only, no forward)
2026-08-22T08:07:49.438+00:00 policy param devices: ['npu:3']
2026-08-22T08:07:49.438+00:00 policy buffer devices: []
2026-08-22T08:07:49.439+00:00 input obs=[0.0, 0.0, 0.0, 0.0] dtype=torch.float32 device=npu:3 shape=[1, 4]
2026-08-22T08:07:49.439+00:00 NPU_READY pid=3156813 device=npu:3 marker=08b61dd5 ts=2026-08-22T08:07:49.439606+00:00 hold=8.0s
2026-08-22T08:07:49.439+00:00 holding 8.0s (external wrapper can run npu-smi now)...
2026-08-22T08:07:57.983+00:00 npu-smi proc evidence: rc=0
NPU ID                         : 3
	Chip Count                     : 1

	Process id:3156813 Process name:python            Process memory(MB):116
	Chip ID                        : 0
2026-08-22T08:07:58.195+00:00 forward ok in 0.211901s  action=[1] value=[[49.979976654052734]] log_prob=[-2.86102294921875e-06]
2026-08-22T08:07:58.196+00:00 NPU_DONE pid=3156813 device=npu:3 marker=08b61dd5 ts=2026-08-22T08:07:58.196463+00:00
2026-08-22T08:07:59.303+00:00 RESULT OK exit_code=0 total=10.186s -> logs/npu_result.json
EXIT_CODE=0

Key result values (all on npu:3):

OutputValueShapedtype
action1[1]int64
value49.979976654052734[1, 1]float32
log_prob-2.86102294921875e-06[1]float32

Placement evidence in logs/npu_result.json (top level): engine: torch_npu, npu_available: true, npu_device_name: Ascend910B4-1, source_revision: 08b61dd5…, input_device: npu:3, model_device: npu:3, output_device: npu:3, exit_code: 0.

Evidence screenshots

Rendered from the real final transcript / sanitized log / result JSON (no fabricated values):

  • assets/agent_workflow.png — agent workflow: read-only upstream routing, pinned-revision checkpoint verification, NPU init, NPU forward, and result JSON.
  • assets/npu_device_call.png — live npu-smi device / process call on physical npu:3 (PID correlation during the hold window).
  • assets/model_result.png — final model result: action / value / log_prob, shapes, devices, versions, timing and PID.
  • assets/modelagent_live_session.png — sanitized excerpts from the real ModelAgent review conversation, followed by the independently rerun final PID and acceptance result.
  • evidence/modelagent_summary.log — detailed sanitized ModelAgent interaction record and its relationship to the fresh independent NPU acceptance evidence.
  • logs/strict_checkpoint_audit.log — independent state-dict audit: all 12 source tensors exactly matched the NPU-loaded policy, with no model forward executed by the audit.

Limitations

  • License is not declared upstream. See the "License" section above; verify before any redistribution.
  • Ascend910B4 does not support fp64 (double). torch_npu casts double tensors to float32 and emits a non-fatal [W822] warning during the first forward. Outputs are correct float32.
  • SB3 training-hyperparameter deserialization. The pinned checkpoint stores learning_rate / clip_range / lr_schedule as Python 3.8 code objects that cannot be cloudpickled under Python 3.11. These are training-time hyperparameters (not weights) and are replaced through SB3 custom_objects with the documented values (0.001, 0.2, ConstantSchedule(0.001)). The real policy.pth weights are loaded unchanged.
  • SB3 CPU-oriented warning. PPO warns that it is "primarily intended to run on the CPU" when placed on a non-CPU accelerator during training; this candidate intentionally runs inference on NPU, so that specific misleading message is filtered.
  • Device mapping. This candidate is validated for physical npu:3. On another host with a different ASCEND_RT_VISIBLE_DEVICES mapping, verify npu:3 still selects the intended physical device.
  • No environment launch. Gym / Unity / robot hardware is not launched; the loaded policy network is invoked directly with the deterministic observation [0, 0, 0, 0].
  • Checkpoint not shipped. The immutable checkpoint is referenced at its source path (/work/pipeline/models/rl26/ppo-cartpole-v1/ppo-CartPole-v1.zip); it is not copied into this delivery.