CoMotion 多人3D运动估计模型 NPU 部署指南

1. 模型概述及场景

CoMotion（Concurrent Multi-person 3D Motion） 是 Apple 于 2025 年发表在 ICLR 上的一种多人3D运动估计方法，能够从单目摄像头视频流中检测并追踪多人的详细3D姿态。该系统在拥挤场景中保持时间一致的预测，能够处理复杂姿态和遮挡情况。模型同时执行强逐帧检测和学习的姿态更新来逐帧追踪人物，而非跨时间匹配检测结果，从而实现在遮挡情况下的在线追踪。

论文：CoMotion: Concurrent Multi-person 3D Motion

作者：Alejandro Newell, Peiyun Hu, Lahav Lipson, Stephan R. Richter, Vladlen Koltun

主要能力：

单目视频多人3D姿态检测与追踪
时间一致的姿态预测
拥挤场景下的遮挡处理
输出 SMPL 姿态参数、渲染视频、MOT 格式跟踪结果

依赖组件：

组件	说明	许可证
ml-comotion	主模型代码	Apple Software License
chumpy	自动微分库	MIT License
aitviewer	3D可视化渲染库	MIT License

2. 定制适配代码

本项目基于以下开源项目进行 NPU 适配修改：

ml-comotion — 主模型代码
- 原作者：Alejandro Newell, Peiyun Hu, Lahav Lipson, Stephan R. Richter, Vladlen Koltun
- 代码许可：Apple Software License（详见 LICENSE.md）
- 模型权重许可：Apple ML Research Model License，仅限研究用途（详见 LICENSE_MODEL.md）
- 修改内容：设备切换 NPU、导入路径调整、相机参数调整、新增无头渲染脚本和依赖列表
chumpy — 自动微分库
- 原作者：Matthew Loper / Max-Planck-Gesellschaft
- 许可：MIT License（详见 LICENSE.txt）
- 修改内容：修复 global 变量作用域问题
aitviewer — 3D可视化渲染库
- 原作者：ETH Zurich, Manuel Kaufmann, Velko Vechev, Dario Mylonopoulos
- 许可：MIT License（详见 LICENSE）
- 修改内容：移除 USD 依赖、PyQt5 迁移 PyQt6、SMPL 兼容性修复、Python 版本限制移除

修改说明：适配华为昇腾 NPU 运行环境，定制适配代码下载后覆盖源码。

3. 修改点总结

本项目基于原始开源代码进行了 NPU 适配修改，修改内容分为以下几类：

3.1 ml-comotion 修改（4个文件）

文件	修改类型	说明
`demo.py`	修改	1) 设备选择从 CUDA 切换到 NPU（`torch.device("npu:0")`）；2) 导入路径加 `src.` 前缀；3) 相机位置调整至原点；4) Billboard 距离从 100.0 调整为 10.0
`src/comotion_demo/utils/dataloading.py`	修改	默认相机内参矩阵焦距从 `2max_res` 调整为 `1.05max_res`，适配 NPU 渲染场景
`demo.sh`	新增	无头渲染启动脚本，使用 Xvfb + Mesa/swrast 软件渲染，支持无物理显示器的容器化/NPU 环境
`requirements.txt`	新增	固定依赖版本列表，包含 `torch-npu==2.5.1` 等华为昇腾 NPU 依赖

3.2 chumpy 修改（1个文件）

文件	修改类型	说明
`chumpy/api_compatibility.py`	修改	修复 `global` 变量 `which_passed` 的作用域声明问题，将 `global` 从 `__main__` 块移到模块级别，解决 Python 兼容性问题

3.3 aitviewer 修改（11个文件）

文件	修改类型	说明
`setup.py`	修改	移除 `usd-core` 依赖；硬编码 PyQt6 替代 PyQt5；移除 Python < 3.11 版本上限
`viewer.py`	修改	移除 USD 导出功能；PyQt5Window → PyQt6Window
`pyqt5_window.py`	修改	PyQt5 → PyQt6 导入迁移
`imgui_integration.py`	修改	PyQt5 → PyQt6 导入迁移
`smpl.py`	修改	使手部姿态参数可选，兼容基础 SMPL 模型（无手部参数时不报错）
`billboard.py`	修改	注释所有 USD（pxr）导入和调用
`meshes.py`	修改	注释所有 USD（pxr）导入和调用
`plane.py`	修改	注释所有 USD（pxr）导入和调用
`light.py`	修改	注释所有 USD（pxr）导入和调用
`node.py`	修改	注释所有 USD（pxr）导入和调用
`usd.py`	修改	注释所有 USD（pxr）导入和调用

3.4 修改目的总结

类别	解决的问题
NPU 适配	将 PyTorch 后端从 CUDA 切换到华为昇腾 NPU
无头渲染	使用 Xvfb + Mesa 软件渲染，支持无显示器的服务器环境
依赖精简	移除 `usd-core`（目标环境不可用），迁移 PyQt5 → PyQt6
模型兼容	修复 SMPL 基础模型（无手部参数）兼容性问题
Python 版本	移除 Python < 3.11 限制，支持 Python 3.11.6
Bug 修复	修复 chumpy 中 `global` 变量作用域问题

4. 准备运行环境

4.1 版本配套表

配套	版本	说明
CANN	8.2.0	华为昇腾计算框架
Python	3.11.6	-
torch	2.5.1	PyTorch
torch_npu	2.5.1	华为 NPU PyTorch 适配层

4.2 硬件环境

Atlas 800T A2（8×64G）
部署卡类型：910B2
部署方式：单卡
操作系统：ARM（OpenEuler 24.03 LTS）

4.3 镜像下载

地址：Ascend Hub
版本：2.1.RC2-800I-A2-py311-openeuler24.03-lts（ARM 架构）

5. 运行指导

5.1 下载源码与定制代码

(1) 下载原始源码

git clone https://github.com/apple/ml-comotion.git
git clone https://github.com/mattloper/chumpy.git
git clone https://github.com/eth-ait/aitviewer.git

(2) 下载定制适配代码

定制代码

git clone https://gitcode.com/Ascend-SACT/ml-comotion-npu.git

(3) 用定制代码覆盖源码

将定制代码覆盖到对应的原始源码目录：

# 假设原始源码和 ml-comotion-npu 在同一父目录下
cp -rf ml-comotion-npu/ml-comotion/* ml-comotion/
cp -rf ml-comotion-npu/chumpy/* chumpy/
cp -rf ml-comotion-npu/aitviewer/* aitviewer/

5.2 安装系统依赖

yum install qt6-qtbase-devel qt6-qttools-devel qt6-qtdeclarative-devel \
    ffmpeg ffmpeg-devel xorg-x11-server-Xvfb mesa-dri-drivers mesa-vulkan-drivers

5.3 安装 Python 依赖

(1) ml-comotion

cd ml-comotion
pip install -r requirements.txt
cd ..

(2) chumpy

cd chumpy
pip install -r requirements.txt
pip3 install .
cd ..

(3) PyQt6

从PyQt6下载 PyQt6-6.7.1-1-cp38-abi3-manylinux_2_28_aarch64.whl

pip install PyQt6-6.7.1-1-cp38-abi3-manylinux_2_28_aarch64.whl

(4) aitviewer

cd aitviewer
pip3 install .
cd ..

5.4 下载模型权重

文件名	下载地址	解压后存放路径	说明
`demo_checkpoints.tar.gz`	demo_checkpoints	`ml-comotion/src/comotion_demo/data`	预训练权重（含检测和更新阶段）
`SMPL_python_v.1.1.0.zip`	SMPL_python	`ml-comotion/src/comotion_demo/data/smpl`	选择“Download version 1.1.0 for Python 2.7 (female/male/neutral, 300 shape PCs)”，将 `basicmodel_neutral_lbs_10_207_0_v1.1.0.pkl` 重命名为 `SMPL_NEUTRAL.pkl` 拷贝到存放路径

5.5 执行推理

cd ml-comotion
bash demo.sh -i path/to/teaser_5s.mp4 -o results

可选参数：

--start-frame：指定起始帧
--num-frames：指定处理帧数
--skip-visualization：跳过可视化渲染

示例输出日志：

[root@pm-d12e0109 ml-comotion]# bash demo.sh -i path/to/teaser_5s.mp4 -o results
Running CoMotion: 0it [00:00, ?it/s]2026-03-21 14:51:51 - INFO - yield_image_from_video - Yielding 1000000000 frames from path/to/teaser_5s.mp4 (121.0 in total).
2026-03-21 14:51:51 - INFO - yield_image_from_video - Starting from frame 0
Running CoMotion: 121it [00:27,  4.44it/s]
2026-03-21 14:52:19 - INFO - visualize_poses - Rendering SMPL video: path/to/teaser_5s.mp4
2026-03-21 14:52:19 - INFO - yield_image_from_video - Yielding 1000000000 frames from path/to/teaser_5s.mp4 (121.0 in total).
2026-03-21 14:52:19 - INFO - yield_image_from_video - Starting from frame 0
2026-03-21 14:52:20 - INFO - get_window_cls - Attempting to load window class: moderngl_window.context.headless.Window
2026-03-21 14:52:20 - INFO - print_context_info - Context Version:
2026-03-21 14:52:20 - INFO - print_context_info - ModernGL: 5.12.0
2026-03-21 14:52:20 - INFO - print_context_info - vendor: Mesa
2026-03-21 14:52:20 - INFO - print_context_info - renderer: llvmpipe (LLVM 17.0.6, 128 bits)
2026-03-21 14:52:20 - INFO - print_context_info - version: 4.5 (Core Profile) Mesa 24.0.3
2026-03-21 14:52:20 - INFO - print_context_info - python: 3.11.6 (main, Feb 19 2025, 18:13:39) [GCC 12.3.1 (openEuler 12.3.1-38.oe2403)]
2026-03-21 14:52:20 - INFO - print_context_info - platform: linux
2026-03-21 14:52:20 - INFO - print_context_info - code: 450
2026-03-21 14:52:20 - INFO - _load_source - Loading: fragment_picking/frag_pick.glsl
2026-03-21 14:52:20 - INFO - _load_source - Loading: outline/outline_draw.glsl
2026-03-21 14:52:20 - INFO - _load_source - Loading: visualize.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/sphere_instanced.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/sphere_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/outline/outline_prepare.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/sphere_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/shadow_mapping/depth_only.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/sphere_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/fragment_picking/frag_map.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lines_instanced.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lines_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/outline/outline_prepare.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lines_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/shadow_mapping/depth_only.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lines_instanced_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/fragment_picking/frag_map.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: chessboard.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: simple_unlit.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/lit_with_edges.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: directional_lights.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/mesh_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/shadow_mapping/depth_only.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/mesh_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/outline/outline_prepare.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/mesh_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/fragment_picking/frag_map.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: screen_texture.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/mesh_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/outline/outline_prepare.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/mesh_positions.vs.glsl
2026-03-21 14:52:22 - INFO - _load_shader - Loading: /usr/local/lib/python3.11/site-packages/aitviewer/shaders/fragment_picking/frag_map.fs.glsl
2026-03-21 14:52:22 - INFO - _load_source - Loading: clipping.glsl
Rendering frames: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 121/121 [00:27<00:00,  4.47it/s]
Video saved to /data01/ml/ml-comotion/results/teaser_5s.mp4
[INFO] Total time: 95 seconds.
[root@pm-d12e0109 ml-comotion]#

推理完成！生成结果位于：

渲染视频：results/teaser_5s.mp4
SMPL 姿态参数：results/*.pt
MOT 格式跟踪结果：results/*.txt

6. 常见问题

6.1 切换输入视频

修改 demo.sh 的 -i 参数：

bash demo.sh -i path/to/your_video.mp4 -o results

6.2 处理单张图片

bash demo.sh -i path/to/image.jpg -o results

6.3 跳过可视化（仅推理）

bash demo.sh -i path/to/video.mp4 -o results --skip-visualization

7. 致谢

本项目基于以下开源项目开发：

CoMotion - Apple Inc.，GitHub，论文：ICLR 2025
chumpy - Matthew Loper / Max-Planck-Gesellschaft，GitHub，MIT 许可证
aitviewer - ETH Zurich，GitHub，MIT 许可证

如果本工作对您有帮助，请引用原始论文：

@inproceedings{newell2025comotion,
  title      = {CoMotion: Concurrent Multi-person 3D Motion},
  author     = {Alejandro Newell and Peiyun Hu and Lahav Lipson and Stephan R. Richter and Vladlen Koltun},
  booktitle  = {International Conference on Learning Representations},
  year       = {2025},
  url        = {https://openreview.net/forum?id=qKu6KWPgxt},
}

8. 许可证

本项目的代码和模型受多个许可证约束，使用前请仔细阅读：

ml-comotion 代码：Apple 软件许可证 - 详见 LICENSE.md
ml-comotion 模型权重：Apple 机器学习研究模型许可证 - 仅限研究用途 - 详见 LICENSE_MODEL.md
chumpy：MIT 许可证 - 详见 LICENSE.txt
aitviewer：MIT 许可证 - 详见 LICENSE

Apple Machine Learning Research Model is licensed under the Apple Machine Learning Research Model License Agreement.