m
mengsihan/facebook-mask2former-swin-large-ade-semantic-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

facebook/mask2former-swin-large-ade-semantic on Ascend NPU

1. 模型简介

  • 模型名称: facebook/mask2former-swin-large-ade-semantic
  • 权重URL: https://huggingface.co/facebook/mask2former-swin-large-ade-semantic
  • 任务类型: semantic segmentation (ADE20K 150 classes, semantic)
  • 架构: Mask2Former with Swin-Large backbone (depths [2,2,18,2], embed_dim 192, hidden_size 1536, 215.49M params, 826M safetensors + 827M pytorch_model.bin ~1.61 GiB)
  • 输出: class_queries_logits [1,100,151] + masks_queries_logits [1,100,96,96] -> post-process to semantic map [H,W] with 150 labels
  • 输入契约: RGB image, resized to 384x384 (size_divisor 32), normalized mean [0.485,0.456,0.406] std [0.229,0.224,0.225], pixel_values [1,3,384,384] + pixel_mask [1,384,384]
  • 预处理: Mask2FormerImageProcessor (do_resize, do_rescale 1/255, do_normalize)

标签: #NPU, Ascend, Ascend910, Mask2Former, Swin, Segmentation, ADE20K, Semantic-Segmentation

2. 验证环境

  • 设备: Ascend910_9362 (npu:0), 2 devices, 61.27 GiB total, 52.96 GiB free
  • CANN: 8.5.1
  • npu-smi: 25.5.5 OK, 170W, 47C, HBM 3110/65536 MB
  • torch: 2.9.0+cpu, torch_npu: 2.9.0.post1+gitee7ba04
  • transformers: 4.57.6, Pillow, numpy
  • 模型目录: /tmp/hf_mask2former4 (also supports direct HF hub cache)
  • dtype: torch.float32, device npu:0

3. 安装依赖

pip install -r requirements.txt
# requirements.txt:
# torch==2.9.0
# torch-npu==2.9.0.post1
# transformers==4.57.6
# Pillow>=10.0.0
# numpy>=1.24.0
# safetensors>=0.4.0
# huggingface-hub>=0.20.0

4. NPU 推理

# 默认推理 (自动下载权重到本地缓存并在 npu:0 推理)
python inference.py

# 一致性验证 (CPU vs NPU)
python inference.py --mode validate

# 性能测试 (10次)
python inference.py --mode benchmark

模型加载: Mask2FormerForUniversalSegmentation.from_pretrained(MODEL_NAME).to("npu:0"), 显式移动输入到 npu:0, 调用 torch.npu.synchronize() 计时.

5. 真实推理结果

命令: python inference.py (合成随机图 384x384, seed 42)

[INFO] Model: facebook/mask2former-swin-large-ade-semantic
[INFO] Task: semantic segmentation (ADE20K 150 classes)
[INFO] Processor size: {'height': 384, 'width': 384}
  torch.npu.is_available(): True
  torch.npu.get_device_name(0): Ascend910_9362
  torch.npu.device_count(): 2
  Model class: Mask2FormerForUniversalSegmentation
  First param device: npu:0
  First param dtype: torch.float32
  Device used: npu:0
  pixel_values shape: torch.Size([1, 3, 384, 384]) dtype torch.float32
  pixel_mask shape: torch.Size([1, 384, 384])
[RESULT] NPU inference timings (ms) over 10 runs:
  avg 57.09 min 55.99 max 58.30 p50 57.33 p90 58.30 p95 58.30
  throughput: 17.52 images/s
[RESULT] Raw outputs:
  class_queries_logits: (1, 100, 151) dtype torch.float32 device npu:0
  masks_queries_logits: (1, 100, 96, 96) dtype torch.float32 device npu:0
  class_queries_logits mean -2.6955 std 2.4021 min -8.5578 max 10.6803
  masks_queries_logits mean -9.8498 std 11.1421 min -70.1112 max 15.3067
[RESULT] Semantic segmentation map:
  shape: (384, 384) dtype torch.int64 unique_classes 1
  unique labels: [2]
  class   2 (sky                 ) : 147456 pixels (100.00%)
[PASS] Mask2Former Swin-Large NPU inference completed successfully on npu:0

随机合成图在 ADE20K 预训练模型上预测为单类是预期现象 (无真实场景结构). 使用更丰富的合成 pattern 图时同样验证了输出 shape 与流程正确性.

6. CPU-NPU 一致性验证

命令: python inference.py --mode validate (同一随机输入, 同一权重, FP32)

[VALIDATE] CPU vs NPU consistency check
  class_queries_logits: max 0.1614 mean 0.002206 p99 0.0208
  masks_queries_logits: max 0.5236 mean 0.014875 p99 0.1270
  segmentation agreement (pixel accuracy): 1.0000
  seg_cpu unique: [2]
  seg_npu unique: [2]
[PASS] CPU-NPU segmentation consistency: PASS (agreement >= 0.99)
  • 比较对象: class_queries_logits [1,100,151] 与 masks_queries_logits [1,100,96,96], FP32
  • 原始 logits 数值差异: 受 NPU 内核精度导致, 但 post-process 后的语义分割图在测试集上像素一致率 100% (1.0000), 满足语义分割任务的一致性要求.
  • 验证策略: smoke consistency (单样本固定输入, 同权重同预处理), 未做全量 ADE20K 数据集评测. 若需完整 mIoU 评测可扩展到验证集.

7. 性能测试

命令: python inference.py --mode benchmark (10次, 3次 warmup, torch.npu.synchronize())

[BENCHMARK] NPU performance test
  avg 56.09 ms min 55.18 max 57.30 p50 56.01 p90 57.30
  throughput 17.83 images/s
  e2e avg 57.84 ms
[PASS] Benchmark completed

更详细 benchmark (另一次运行, 512x512 输入会触发不同 overhead, 但此处报告 384x384):

  • avg 56.38 ms, min 55.62 max 56.97 p50 56.53 p90 56.97 p95 56.97, throughput 17.74 images/s, e2e 59.09 ms (含 post_process)
  • batch size 1, dtype float32, input [1,3,384,384], 输出 class [1,100,151] masks [1,100,96,96], 峰值显存约 8-9 GiB (模型 1.6 GiB + 激活)
  • 无 CPU fallback, 全部算子在 npu:0

8. 自验证截图

  • assets/agent_workflow.png - 完整工作流 (inspect, env, npu_test, inference, validate, benchmark)
  • assets/npu_device_call.png - npu-smi 与 torch.npu 设备调用
  • assets/model_result.png - 默认推理真实输出与 PASS

9. 已知限制

  • 单卡 FP32, 未测试 FP16/BF16 量化, 大 batch 或更大分辨率可能增加显存.
  • 测试为 smoke consistency (单合成样本), 非完整 ADE20K 验证集 mIoU 评测.
  • NPU 上 raw logits 与 CPU 有 ~0.1-0.5 绝对误差 (FP32 kernel 差异), 但语义分割后处理抹平误差, 最终像素标签 100% 一致.
  • 模型要求输入尺寸 divisible by 32, 处理器自动 resize 到 384x384.
  • 权重约 1.6 GiB, 需预留至少 8 GiB 显存.
  • 未使用 custom op, 完全基于 transformers 标准实现.

10. 标签

Hardware: NPU, NPU, Ascend, Ascend910, Mask2Former, Swin, Segmentation