PPO CartPole-v1 强化学习策略模型适配昇腾NPU版本。
| 参数 | 值 |
|---|---|
| Policy | MlpPolicy |
| Batch Size | 256 |
| Clip Range | lin_0.2 |
| Entropy Coef | 0.0 |
| GAE Lambda | 0.8 |
| Gamma | 0.98 |
| Learning Rate | lin_0.001 |
| N Environs | 8 |
| N Epochs | 20 |
| N Steps | 32 |
| N Timesteps | 100000 |
| 参数 | 值 |
|---|---|
| Observation Space | Box(4,) |
| Action Space | Discrete(2) |
| Max Episode Steps | 500 |
| 指标 | 数值 |
|---|---|
| 推理时间 | ~0.17s (演示) |
| NPU显存占用 | <1GB (演示) |
| 模型大小 | ~136 KB |
torch>=2.0.0
torch_npu>=2.0.0
numpy>=1.21.0
gymnasium>=0.29.0
stable-baselines3>=2.0.0cd /workspace/agent1/ppo-CartPole-v1-ascend
pip install -r requirements.txt -i https://repo.huaweicloud.com/repository/pypi/simplecd /workspace/agent1/ppo-CartPole-v1-ascend
python inference.py --device npu:0 --show_npu_memory--model_path: 模型路径,默认 /workspace/agent2/HumanCompatibleAI/ppo-CartPole-v1--device: 运行设备,默认 npu:0--show_npu_memory: 显示NPU显存使用情况import torch
device = "npu:0"
batch_size = 1
input_dim = 4
observation = torch.randn(batch_size, input_dim, dtype=torch.float32).to(device)
action_probs = policy_net(observation)
action = torch.argmax(action_probs, dim=-1)[设备] npu:0
NPU设备数量: 1
设备 0: Ascend910B2
[模型] ppo-CartPole-v1
[任务] 强化学习策略推理 (Reinforcement Learning)
[算法] PPO (Proximal Policy Optimization)
[环境] CartPole-v1
[模型配置]
Policy: MlpPolicy
Batch Size: 256
Clip Range: lin_0.2
Entropy Coef: 0.0
GAE Lambda: 0.8
Gamma: 0.98
Learning Rate: lin_0.001
N Environs: 8
N Epochs: 20
N Steps: 32
N Timesteps: 100000
[CartPole环境配置]
Observation Space: Box(4,)
Action Space: Discrete(2)
Max Episode Steps: 500
[模型文件]
Model: /workspace/agent2/HumanCompatibleAI/ppo-CartPole-v1/ppo-CartPole-v1.zip (136.07 KB)
[构建测试输入]
Batch Size: 1
Input Dimension: 4
Hidden Dimension: 64
Output Dimension (Actions): 2
[输入张量形状]
Observation: (1, 4)
运行PPO CartPole推理演示...
======================================================================
加载策略网络到NPU...
======================================================================
======================================================================
前向传播...
======================================================================
======================================================================
NPU 显存状态
======================================================================
[NPU显存] 已分配=0.0000GB | 缓存=0.0020GB | 峰值=0.0000GB
======================================================================
NPU 信息 (npu-smi info)
======================================================================
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 103.9 41 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 3487 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1165077 | python | 118 |
+===========================+===============+====================================================+
======================================================================
输出摘要
======================================================================
模型: ppo-CartPole-v1
任务: 强化学习策略推理
输入形状: torch.Size([1, 4])
动作概率形状: torch.Size([1, 2])
选定动作: 1
总耗时: 0.1662s
输出 finite: True
动作概率分布:
动作0: 0.4602
动作1: 0.5398完整模型推理需要 stable-baselines3:
pip install stable-baselines3 gymnasium然后使用:
from stable_baselines3 import PPO
import gymnasium as gym
model_path = "/workspace/agent2/HumanCompatibleAI/ppo-CartPole-v1/ppo-CartPole-v1.zip"
model = PPO.load(model_path)
env = gym.make("CartPole-v1")
obs, info = env.reset()
done = False
while not done:
action, _states = model.predict(obs, deterministic=True)
obs, reward, terminated, truncated, info = env.step(action)
done = terminated or truncated
env.close()Cart状态 (位置, 速度, 角度, 角速度)
|
MlpPolicy网络
|
全连接层 (4 -> 64)
|
ReLU激活
|
全连接层 (64 -> 64)
|
ReLU激活
|
输出层 (64 -> 2)
|
Softmax
|
动作概率分布