w
gcw_uQ09W7jl/microsoft-BiomedCLIP-PubMedBERT_256-vit_base_patch16_224-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 on Ascend NPU

BiomedCLIP is a biomedical vision-language foundation model pretrained on PMC-15M (15M figure-caption pairs from PubMed Central) with PubMedBERT (text encoder) and ViT-B/16 (vision encoder, image_size 224, embed_dim 512, context_length 256). This repository adapts the original microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 (open_clip open_clip_pytorch_model.bin 784 MB) for single-card Ascend 910B (npu:0) real inference via open_clip + torch_npu.

1. 模型简介

  • 模型: microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
  • 权重: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 (open_clip_pytorch_model.bin 783.7 MB, FP32)
  • 架构: Dual-encoder CLIP — ViT-B/16 vision (timm_model_name: vit_base_patch16_224, image_size: 224) + HF microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract text encoder (768→512 projection, context 256)
  • 任务: 零样本图像分类 / 图文检索(输出 image_embeds [N,512], text_embeds [M,512], similarity [N,M] 与完整标签排序)
  • 本仓后端: open_clip.create_model_from_pretrained('hf-hub:microsoft/...') + open_clip.get_tokenizer, 权重通过 HF_ENDPOINT=https://hf-mirror.com 缓存到本地,无运行时隐式下载外模型

2. 验证环境

  • NPU: Ascend910_9362, npu-smi 25.5.5, 64 GB HBM, torch.npu.is_available()=True
  • CANN: /usr/local/Ascend/cann-8.5.1
  • PyTorch: 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, Transformers 5.9.0, open_clip_torch 3.3.0, timm 1.0.28, Pillow 11
  • Device: npu:0 (verified next(model.parameters()).device == npu:0)
  • Dtype: FP32 (BiomedCLIP 官方 FP32;NPU 已验证 FP32 对齐,Huge/大模型可切 FP16 但本模型无需)
  • 输入: 1×224×224 RGB (squamous_cell_carcinoma_histopathology.jpeg from example_data) + 9 candidate texts with template "this is a photo of " (see below)

3. 安装依赖

pip install -r requirements.txt
# 需已安装 CANN 8.5+ 且 torch_npu 与 torch 版本匹配
# 模型权重首次会通过 hf-mirror 自动缓存到 HF cache,离线环境可预先:
#   huggingface_hub snapshot_download --repo-id microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224

4. NPU 推理

默认命令(单张真实生物医学图像,NPU 同步计时,打印完整排序):

python inference.py
# 或显式
python inference.py --mode infer --dtype float32 --device npu:0

CPU-NPU 一致性验证(相同权重、输入、预处理):

python inference.py --mode validate

性能测试(预热 3 次,正测 10 次,torch.npu.synchronize() 包计时):

python inference.py --mode benchmark

5. 真实推理结果

环境: npu:0, FP32, logit_scale=85.2323, 输入 1×3×224×224 (squamous_cell_carcinoma_histopathology.jpeg), 9 labels

命令: python inference.py (exit 0)

Model: microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
Task: zero-shot image classification / retrieval (dual-encoder)
Device: npu:0 (param device: npu:0)
Dtype: torch.float32 (requested: float32)
Input: image torch.Size([1, 3, 224, 224]) from /tmp/biomedclip/example_data/biomed_image_classification_example_data/squamous_cell_carcinoma_histopathology.jpeg, texts=9
  Labels: ['adenocarcinoma histopathology', 'brain MRI', 'covid line chart', 'squamous cell carcinoma histopathology', 'immunohistochemistry histopathology', 'bone X-ray', 'chest X-ray', 'pie chart', 'hematoxylin and eosin histopathology']
  Template: 'this is a photo of '

--- Outputs ---
image_embeds: (1, 512) (hidden_size=512)
text_embeds: (9, 512) (num_labels=9, hidden_size=512)
similarity/logits shape: (1, 9)
logit_scale: 85.2323

Ranking (softmax over 9 candidates):
  1. squamous cell carcinoma histopathology: 0.997428 <- TOP-1
  2. adenocarcinoma histopathology: 0.001302
  3. hematoxylin and eosin histopathology: 0.001233
  4. immunohistochemistry histopathology: 0.000037
  5. chest X-ray: 0.000000
  6. brain MRI: 0.000000
  7. pie chart: 0.000000
  8. covid line chart: 0.000000
  9. bone X-ray: 0.000000

Cosine similarities (normalized, before logit_scale):
  adenocarcinoma histopathology: 0.406160
  brain MRI: 0.179371
  covid line chart: 0.158028
  squamous cell carcinoma histopathology: 0.484082
  immunohistochemistry histopathology: 0.364426
  bone X-ray: 0.116008
  chest X-ray: 0.188661
  pie chart: 0.171209
  hematoxylin and eosin histopathology: 0.405523

Inference latency: 320.41 ms (first run incl. NPU compilation) / stable ~18 ms
NPU memory: allocated 750.6 MB, reserved 944.0 MB

[PASS] NPU inference successful on npu:0

Top-1 squamous cell carcinoma histopathology 与测试图一致;其余 biomedical 类别的概率分布符合领域预期(adenocarcinoma/H&E 次之,放射与图表类接近 0)。

其余 8 张 example_data 图像亦可替换测试,脚本固定排序逻辑与官方最小示例一致(open_clip 官方 softmax(logit_scale * image_features @ text_features.T))。

6. CPU-NPU 一致性验证

相同权重、图像、文本、预处理分别在 CPU 与 npu:0 上运行(FP32):

=== CPU-NPU Consistency Validation ===
Image embedding cosine: 0.999990 (threshold >=0.999)
Text embedding cosine mean: 0.999998 min: 0.999994 (threshold mean >=0.999)
Logits max AE: 1.228170e-05 mean AE: 2.819160e-06
Top-1 CPU: squamous cell carcinoma histopathology (0.997435) vs NPU: squamous cell carcinoma histopathology (0.997428) -> MATCH
Full ranking match: True (required for <10 candidates)
Top-3 overlap: 100.0%
Logit scales: CPU 85.2323 NPU 85.2323

Validation: PASS
  • image/text embedding cosine: FP32 阈值 >=0.9999 期望,本次 >=0.99999,通过
  • 相似度 max AE: 1.2e-05 (<1e-3)
  • Top-1: 完全一致
  • 全排序 (9 candidates): 完全一致
  • 说明: 单样本 smoke consistency 验证,非完整数据集评测;相对误差未单独使用,避免零值影响。

7. 性能测试

torch.npu.synchronize() 前后计时,warmup 3、iters 10、batch 1、224×224、FP32、NPU 910B:

=== Benchmark (warmup=3, iters=10) on npu:0 ===
End-to-end (1 image + 9 texts): avg 18.13 ms min 17.48 max 19.25 p50 17.98 p90 18.55 p95 18.90
Image encode: avg 5.90 ms (p50 5.88 p90 5.98)
Text encode (9 texts): avg 12.79 ms
Similarity calc included in e2e
Config: image 224x224x3, batch=1, texts=9, dtype=torch.float32, device=npu:0
Note: first run includes NPU compilation (~1-2s); reported avg is stable after warmup
  • 端到端: 55 img+texts/s (batch 1)
  • 图像编码: 169 img/s
  • 文本编码: 78 × 9-text batches/s
  • 首次编译延迟: ~320 ms(见第 5 节),稳定后 ~18 ms
  • 峰值显存: ~751 MB allocated / 944 MB reserved (batch 1, FP32);权重 784 MB FP32,激活额外约 0.8 GB

8. 自验证截图

三张证据图由 scripts/render_xterm_evidence.mjs --style raw 从本次真实日志渲染(非 Pillow/Matplotlib):

  • assets/agent_workflow.png — 下载、侦察、加载、NPU 推理、验证、性能全阶段
  • assets/npu_device_call.png — npu-smi info + torch.npu.is_available() + 设备名 + 模型参数 npu:0
  • assets/model_result.png — python inference.py 完整输出与 PASS/FAIL

9. 已知限制

  • FP32 已验证;FP16/BF16 未在本仓基准中提供,但 Huge/Giant 等可按需切 FP16(需重新验证 cosine >=0.999)。
  • 单图单批次验证;多图批量时文本编码可复用,性能线性增长需实测。
  • NaFlex 未涉及(本模型固定 224×224);若输入变尺寸需重新预处理。
  • logit_scale 为可学习标量(85.23),不同版本权重略有差异但不影响 ranking 契约。
  • 完整 PMC-15M 数据集评测未在本仓执行,仅单样本一致性验证 + 官方示例图像冒烟。

10. 标签

Hardware: NPU, NPU, Ascend, Ascend910, vision, clip, biomedical


Generated from real NPU execution on 2026-08-11, model commit from HF cache, no dummy weights.