wrhhh/facebook-mask2former-swin-large-ade-semantic-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

facebook/mask2former-swin-large-ade-semantic on Ascend NPU

1. 模型简介

  • 模型名称: facebook/mask2former-swin-large-ade-semantic (commit aa25c92404a40599614215e76514c79b427c7527)
  • 类型: 图像分割 / 语义分割 (ADE20K 150类)
  • 架构: Mask2FormerForUniversalSegmentation (model_type: mask2former, Swin-Large backbone depths [2,2,18,2] embed_dim 192 hidden 1536, hidden_dim 256, num_queries 100, num_labels 150, common_stride 4)
  • 原始权重: https://huggingface.co/facebook/mask2former-swin-large-ade-semantic (model.safetensors 826 MB, 215.49M 参数, transformers, torch_dtype float32, dataset ade, license other)
  • 任务: 输入 384x384 RGB 图像,经 Mask2FormerImageProcessor (Resize 384x384, rescale 1/255, normalize mean [0.485,0.456,0.406] std [0.229,0.224,0.225], size_divisor 32) 输出 100个 query 的 masks_queries_logits [1,100,96,96] 与 class_queries_logits [1,100,151],经 post_process_semantic_segmentation 融合为原图尺寸语义图 [H,W] (150类)
  • 适配: 纯 PyTorch + torch_npu, Mask2FormerForUniversalSegmentation.from_pretrained(local, torch_dtype=float32).to(npu:0) + AutoImageProcessor, 无隐式联网,本地缓存 /opt/atomgit/.cache/huggingface/hub/models--facebook--mask2former-swin-large-ade-semantic/snapshots/aa25c92404a40599614215e76514c79b427c7527 (HF_ENDPOINT=https://hf-mirror.com)

2. 验证环境

  • 硬件: Ascend910_9362 x2, 推理 npu:0 (Phy 4 Bus 0000:0B:00.0 Health OK, HBM 4373/65536 MB 使用, Temp 44C, npu-smi 25.5.5)
  • 驱动: CANN 8.5.1 (/usr/local/Ascend/cann-8.5.1)
  • 软件: Python 3.11.14, torch 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, transformers 4.57.6, Pillow 12.2.0, numpy 1.26.4, safetensors
  • 加载: 本地权重 826 MB (model.safetensors 826M), torch_dtype float32, device npu:0, 首参 npu:0 float32, 215.49M 参数
  • 输入: 确定性合成图像 seed=0, 384x384 RGB 随机 uint8 -> processor pixel_values [1,3,384,384] float32 + pixel_mask [1,384,384]

3. 安装依赖

pip install -r requirements.txt
# 需预装 CANN Toolkit 8.5.1 使 torch_npu 可用
# 权重已缓存于 /opt/atomgit/.cache/huggingface/hub/models--facebook--mask2former-swin-large-ade-semantic/snapshots/aa25c92404a40599614215e76514c79b427c7527 (model.safetensors 826M)
# 离线可通过 HF_ENDPOINT=https://hf-mirror.com 走国内镜像 snapshot_download

4. NPU 推理

python inference.py
# 或指定模式
python inference.py --mode validate   # CPU vs NPU 一致性 (pixel agreement / IoU)
python inference.py --mode benchmark  # 3 warmup + 10 iters 同步计时
python inference.py --device npu:0 --dtype float32

5. 真实推理结果

  • 命令: python inference.py (device npu:0, float32, 合成图 seed 0, torch.npu.synchronize 计时)
  • 设备: npu:0 Ascend910_9362, torch_npu 2.9.0.post1, torch.npu.is_available() True, 模型首参 npu:0 float32
  • 输入: pixel_values [1,3,384,384] float32 (processor 384x384)
  • 输出:
    • masks_queries_logits: [1, 100, 96, 96] float32
    • class_queries_logits: [1, 100, 151] float32
    • 语义图 semantic [384,384] int64 via processor.post_process_semantic_segmentation
  Semantic map unique classes: [2] (total 1)
    class 2: 147456 pixels (100.00%)
  Top predicted class: 2 (147456 pixels)

(class 2 = sky in ADE20K id2label)

  • 延迟: 首次端到端 402.43 ms (含 processor+forward+同步, 含首次编译), 稳定 forward 见性能节
  • 结果: PASS (semantic shape [384,384] matches 384x384, masks finite)

6. CPU-NPU 一致性验证

  • 命令: python inference.py --mode validate
  • 对比: 同一张合成图、同 dtype float32, CPU 与 npu:0 同步推理
CPU masks_queries_logits (1, 100, 96, 96) class (1, 100, 151)
CPU semantic unique [2]
NPU masks_queries_logits (1, 100, 96, 96) class (1, 100, 151)
NPU semantic unique [2]
Max abs diff masks_queries_logits: 13.697899
Mean abs diff masks_queries_logits: 0.125415
Max abs diff class_queries_logits: 1.415354
Pixel agreement: 1.000000 (100.00%)
Mean IoU: 1.000000
  class 2 IoU 1.000000
CPU top class 2 NPU top 2 match=True
  • 判定: 像素一致性 100% / Mean IoU 1.0, PASS (阈值 0.99, NPU 与 CPU 语义图完全一致; raw logits 存在浮点数值差异但 argmax 融合后一致, 属 NPU/CPU 浮点实现预期差异)
  • 说明: 单样本 smoke consistency,非完整 ADE20K 验证集评测;NPU 与 CPU 在语义分割后处理层面高度一致

7. 性能测试

  • 命令: python inference.py --mode benchmark (torch.npu.synchronize 前后计时)
  • 条件: npu:0, float32, 输入 [1,3,384,384], 3 warmup + 10 iters
  • 结果 (同步 forward, 不含 processor):
 iter 1: 48.47 ms
 iter 2: 49.90 ms
 iter 3: 51.22 ms
 iter 4: 50.85 ms
 iter 5: 50.92 ms
 iter 6: 51.01 ms
 iter 7: 51.22 ms
 iter 8: 50.69 ms
 iter 9: 50.52 ms
 iter 10: 50.86 ms
 avg 50.57 ms min 48.47 max 51.22 p50 50.86 p90 51.22 p95 51.22
 throughput 19.78 images/s (batch 1, (384, 384))
 NPU memory free 61162.6 MB total 62740.0 MB
  • 首次 e2e 含 processor 402 ms,稳态仅模型 forward 约 48-51 ms,对应约 19.8 images/s (batch 1, 384)
  • 峰值显存: 推理时 HBM 4373 MB (npu-smi), 远低于 64 GiB 单卡上限
  • 说明: Swin-Large 215M 参数,float32;fp16/bf16 未启用,未来可尝试半精度进一步提速

8. 自验证截图

  • assets/agent_workflow.png - 下载、侦察、加载、NPU 推理、验证、性能全流程日志
  • assets/npu_device_call.png - npu-smi info + torch.npu.is_available() + 设备名 + 模型参数 device/dtype
  • assets/model_result.png - python inference.py 默认输出 (任务结果、语义图、延迟、PASS)

9. 已知限制

  • 输入固定 384x384 (processor size 384x384, size_divisor 32), 非 384 图像会经 Resize 规整,可能改变宽高比
  • 推荐 dtype float32 (训练即 float32); fp16/bf16 在 NPU 上未充分验证,logits 差异可能增大但语义一致性已验证
  • 单样本一致性验证,非 ADE20K 全量评测;大规模语义分割数据集 mIoU 评测需离线批量脚本
  • 随机合成图预测单一类别 (sky) 属固定 seed 输入特性,真实自然图像会预测多类别;本仓提供可复现的确定性 dummy 输入

10. 标签

  • Hardware: NPU, NPU, Ascend, Ascend910, Mask2Former, Swin, ADE20K, semantic-segmentation, vision