waylong/microsoft-BiomedCLIP-PubMedBERT_256-vit_base_patch16_224-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 on Ascend NPU

1. 模型简介

BiomedCLIP 是基于 PMC-15M (1500万 图文对) 预训练的生物医学图文双塔基础模型,使用 PubMedBERT (microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract, 768 hidden, 12层) 作为文本编码器,Vision Transformer (vit_base_patch16_224, 12层, 768 hidden) 作为图像编码器,投影维度 512,上下文长度 256。支持跨模态检索、零样本分类、VQA 等。权重格式 open_clip_pytorch_model.bin (783.7 MB), 配置 open_clip_config.json (mean [0.48145,0.45782,0.40821] std [0.2686,0.2613,0.2757])。官方示例使用 open_clip.create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')。

  • 模型权重: https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
  • 配置: open_clip_config.json embed_dim=512 vision_cfg.timm_model_name=vit_base_patch16_224 text_cfg.hf_model_name=microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract
  • 类型: dual-encoder CLIP 路线: zero-shot-image-classification (embedding + similarity + ranking)

2. 验证环境

  • NPU: Ascend910_9362 CANN 8.5.1 npu-smi 25.5.5 2 devices (Chip 0 3112/65536 MB, Chip1 2869/65536 MB)
  • torch 2.9.0+cpu torch_npu 2.9.0.post1+gitee7ba04 torch.npu.is_available()=True device_name Ascend910_9362
  • Python 3.11 open_clip_torch 2.23.0 timm 0.9.10 transformers 4.35.2
  • 模型目录: /data/.cache/models/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 (ModelScope snapshot, 已通过 /tmp/biomed_full_real 完整 748M 权重校验 MD5 531cdee3c02f0e1ce22c474df82ad618)
  • 单图批处理 batch=1 image 224x224 FP32 参数设备 npu:0

3. 安装依赖

pip install -r requirements.txt
# 或
pip install torch==2.9.0 torch-npu open-clip-torch==2.23.0 timm==0.9.10 transformers==4.35.2 Pillow numpy

4. NPU 推理

# 默认零样本分类 (squamous cell carcinoma histopathology 示例)
python inference.py
# CPU-NPU 一致性验证
python inference.py --mode validate
# 性能测试 (10次同步计时)
python inference.py --mode benchmark

加载方式: 本地权重 + open_clip.factory._MODEL_CONFIGS 注册 biomedclip_local + create_model_and_transforms(pretrained=local.bin, image_mean/std) + model.to(npu:0, dtype=torch.float32),tokenizer 来自 get_tokenizer(biomedclip_local),context_length=256,不依赖运行时下载。

5. 真实推理结果

固定输入:

  • 模板: "this is a photo of "
  • 候选文本 (9): adenocarcinoma histopathology, brain MRI, covid line chart, squamous cell carcinoma histopathology, immunohistochemistry histopathology, bone X-ray, chest X-ray, pie chart, hematoxylin and eosin histopathology
  • 测试图: example_data/biomed_image_classification_example_data/squamous_cell_carcinoma_histopathology.jpeg (275x183 JPEG, NPU preprocess 后 224x224, mean/std 如上)
  • 设备: npu:0 dtype float32 logit_scale 85.2323

NPU 输出 (python inference.py, npu:0):

image_embeds: (1, 512) dtype=torch.float32
text_embeds: (9, 512) dtype=torch.float32
logit_scale: 85.2323
similarity/logits: (1, 9)
Ranking (candidate text sorted by probability):
  1. squamous cell carcinoma histopathology         prob=0.997428  logit=41.2594
  2. adenocarcinoma histopathology                  prob=0.001302  logit=34.6179
  3. hematoxylin and eosin histopathology           prob=0.001233  logit=34.5636
  4. immunohistochemistry histopathology            prob=0.000037  logit=31.0609
  5. chest X-ray                                    prob=0.000000  logit=16.0800
  6. brain MRI                                      prob=0.000000  logit=15.2882
  7. pie chart                                      prob=0.000000  logit=14.5926
  8. covid line chart                               prob=0.000000  logit=13.4691
  9. bone X-ray                                     prob=0.000000  logit=9.8876
Top-1: squamous cell carcinoma histopathology (prob=0.997428)  Expected: squamous cell carcinoma histopathology  Match: True
Result: PASS
Inference latency (e2e, batch=1): 17.11 ms (warmup 3, 同步后计时)

该结果与官方示例 9 类分类任务一致,正确区分鳞癌与腺癌病理。

6. CPU-NPU 一致性验证

同权重、同输入、同预处理,分别在 CPU 与 npu:0 运行,比较 embedding cosine 与 logits 误差:

python inference.py --mode validate
image_embeds cosine (NPU vs CPU): 0.999990
text_embeds cosine  (NPU vs CPU): 0.999998
logits max_abs_err: 1.220684e-05 mean_abs_err: 2.789616e-06
Top-1 CPU: squamous cell carcinoma histopathology  NPU: squamous cell carcinoma histopathology  Match: True
Top-3 overlap: 3/3
Thresholds: FP32 cosine>=0.9999, max_err <1e-3, Top-1 must match
Consistency: PASS
Overall: PASS

说明: 单样本 smoke consistency 验证,非完整数据集评测;FP32 下 cosine >=0.9999 且 Top-1 完全一致,证明 NPU 计算正确。

7. 性能测试

同步计时 torch.npu.synchronize() 前后,warmup 3 + formal 10 次,batch=1, 224x224, 9 texts, FP32, npu:0:

E2E latency ms: avg=16.19 min=16.12 max=16.42 p50=16.16 p90=16.28 p95=16.35
Image encode ms: avg=5.16 min=5.09 max=5.32 p50=5.15
Text encode ms (9 texts): avg=11.01 min=10.98 max=11.08 p50=11.00
NPU memory free=59.93 GiB total=61.27 GiB (推理峰值约 3112/65536 MB Chip0)

首次编译已包含在 warmup;稳定延迟约 16.2 ms/图 (~61.7 images/s e2e),图像塔 ~5.2 ms,文本塔 (9 texts) ~11 ms。

8. 自验证截图

  • assets/agent_workflow.png - 下载、侦察、加载、NPU推理、验证、性能全流程日志
  • assets/npu_device_call.png - npu-smi info、torch.npu.is_available、device_name、param device/dtype
  • assets/model_result.png - python inference.py 真实零样本分类排序与 PASS

9. 已知限制

  • 仅验证单图 9 标签零样本分类,非完整 PMC-15M 检索评测;多图 batch、其他模态图片 (MRI/X-ray/chart) 未在本次批量验证
  • FP32 精度,FP16/BF16 未启用,显存预估 0.73 GiB 权重 + 激活 <2 GiB,满足单卡 Ascend910B
  • 需本地权重目录 /data/.cache/models/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224,离线加载不依赖 HF Hub 运行时下载;首次下载需处理 ModelScope 分片 (748M bin)
  • open_clip 非 HF Transformers AutoModel,需通过 _MODEL_CONFIGS 注册;不支持 trust_remote_code

10. 标签

Hardware: NPU, NPU, Ascend, Ascend910, CLIP, BiomedCLIP, open_clip, vision-language, zero-shot