liucu/ppo-Pendulum-v1-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

HumanCompatibleAI/ppo-Pendulum-v1 on Ascend NPU

模型与任务

  • 模型:HumanCompatibleAI/ppo-Pendulum-v1
  • Revision:6a0fe838119427278252dde3bf723ed7a3266df5
  • 适配路线:reinforcement-learning (PPO MlpPolicy, continuous-control)
  • 任务:Pendulum-v1 连续扭矩控制 (观测 3D -> 动作 1D in [-2, 2])
  • 设备:npu:0
  • 权重:policy.pth (40382 bytes, 13 tensors, MlpPolicy 64x64 tanh)

环境依赖

{
  "torch_version": "2.9.0+cpu",
  "torch_npu_version": "2.9.0.post1+gitee7ba04",
  "torch_npu_available": true,
  "device_name": "Ascend910_9362",
  "device_count": 2,
  "python_version": "3.11.14",
  "dtype": "float32",
  "model_dir": "/opt/atomgit/adapt-npu-agent/.cache_hf/models--HumanCompatibleAI--ppo-Pendulum-v1/snapshots/6a0fe838119427278252dde3bf723ed7a3266df5",
  "hf_endpoint": "https://hf-mirror.com"
}

安装:pip install -r requirements.txt

NPU 推理

默认执行:python inference.py

预期输出包含:模型/revision/route、backend torch_npu、device npu:0、dtype float32、输入摘要(8 obs)、动作摘要与同步耗时、PASS/FAIL。

真实输入

{
  "env": "Pendulum-v1",
  "observation_dim": 3,
  "observation_names": ["cos(theta)", "sin(theta)", "theta_dot"],
  "observation_range": {"cos_theta": [-1,1], "sin_theta": [-1,1], "theta_dot": [-8,8]},
  "action_dim": 1,
  "action_range": [-2.0, 2.0],
  "batch": 8,
  "horizon": 1,
  "coordinate_frame": "torque",
  "dtype": "float32",
  "revision": "6a0fe838119427278252dde3bf723ed7a3266df5",
  "observations": [
    [1.0, 0.0, 0.0],
    [0.0, 1.0, 0.0],
    [-1.0, 0.0, 0.0],
    [0.0, -1.0, 2.0],
    [0.707, 0.707, -3.0],
    [-0.707, 0.707, 5.0],
    [0.0, 1.0, -8.0],
    [0.0, -1.0, 8.0]
  ]
}

观测契约:Pendulum-v1 原生观测为 [cos(theta), sin(theta), theta_dot],theta ∈ [-π, π], theta_dot ∈ [-8, 8]。无需归一化 (normalize=False)。

真实推理结果

{
  "output_shape": [8, 1],
  "output_dtype": "float32",
  "output_device": "npu:0",
  "cpu_actions": [-0.2673841714859009, -2.0, -1.9082978963851929, 2.0, 0.8248248100280762, -2.0, 2.0, -2.0],
  "npu_actions": [-0.2673843204975128, -2.0, -1.9082978963851929, 2.0, 0.8248245120048523, -2.0, 2.0, -2.0],
  "finite": true,
  "bounds_ok": true,
  "action_range": [-2.0, 2.0],
  "horizon": 1,
  "action_dim": 1,
  "coordinate_frame": "torque",
  "sampler": "deterministic_mean"
}

动作语义:PPO 确定性均值,经 torch.clamp(..., -2, 2) 钳位到 Pendulum 扭矩边界。8 个 batch 覆盖直立/水平/倒置及极端角速度。

CPU-NPU 一致性

{
  "passed": true,
  "atol": 0.0001,
  "rtol": 0.001,
  "max_abs": 2.980232238769531e-07,
  "mean_abs": 6e-08,
  "tolerance": 0.0021,
  "max_ref": 2.0,
  "shape": [8, 1],
  "elements": 8,
  "finite": true,
  "bounds_ok": true
}

验证脚本:python scripts/validate_policy_outputs.py 比较完整动作张量,阈值 atol=1e-4, rtol=1e-3,仅比较有策略语义的连续动作,未使用环境 reward 替代。

性能

{
  "avg_ms": 0.1805,
  "min_ms": 0.1499,
  "max_ms": 0.2485,
  "p50_ms": 0.1705,
  "actions_per_s": 44311.14,
  "decisions_per_s": 5538.89,
  "batch": 8,
  "horizon": 1,
  "action_dim": 1,
  "dtype": "float32",
  "device": "npu:0",
  "warmup": 3,
  "stable_runs": 10
}

计时方式:每次推理前后调用 torch.npu.synchronize(),单独报告首轮编译 (≈130 ms) 与预热后延迟。批量 8 观测同步吞吐 actions/s 与 decisions/s。

自验证截图

Agent workflow

NPU device call

Model result

三张图片均由 xterm.js 根据本次真实日志生成。

已知限制

  • 仅对本次记录的模型 revision、输入契约、dtype=float32 与 Ascend910_9362 环境完成验证。
  • 确定性均值推理;随机采样需固定 seed 与 log_std,不在此次验证范围。
  • 未包含闭环 rollout reward,仅单步观测->动作数值一致性。

标签

#NPU #Ascend #Ascend910 #ReinforcementLearning #PPO #Pendulum