BiomedCLIP 是基于 PMC-15M (1500万 图文对) 预训练的生物医学图文双塔基础模型,使用 PubMedBERT (microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract, 768 hidden, 12层) 作为文本编码器,Vision Transformer (vit_base_patch16_224, 12层, 768 hidden) 作为图像编码器,投影维度 512,上下文长度 256。支持跨模态检索、零样本分类、VQA 等。权重格式 open_clip_pytorch_model.bin (783.7 MB), 配置 open_clip_config.json (mean [0.48145,0.45782,0.40821] std [0.2686,0.2613,0.2757])。官方示例使用 open_clip.create_model_from_pretrained('hf-hub:microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224')。
open_clip_config.json embed_dim=512 vision_cfg.timm_model_name=vit_base_patch16_224 text_cfg.hf_model_name=microsoft/BiomedNLP-BiomedBERT-base-uncased-abstractpip install -r requirements.txt
# 或
pip install torch==2.9.0 torch-npu open-clip-torch==2.23.0 timm==0.9.10 transformers==4.35.2 Pillow numpy# 默认零样本分类 (squamous cell carcinoma histopathology 示例)
python inference.py
# CPU-NPU 一致性验证
python inference.py --mode validate
# 性能测试 (10次同步计时)
python inference.py --mode benchmark加载方式: 本地权重 + open_clip.factory._MODEL_CONFIGS 注册 biomedclip_local + create_model_and_transforms(pretrained=local.bin, image_mean/std) + model.to(npu:0, dtype=torch.float32),tokenizer 来自 get_tokenizer(biomedclip_local),context_length=256,不依赖运行时下载。
固定输入:
"this is a photo of "adenocarcinoma histopathology, brain MRI, covid line chart, squamous cell carcinoma histopathology, immunohistochemistry histopathology, bone X-ray, chest X-ray, pie chart, hematoxylin and eosin histopathologyexample_data/biomed_image_classification_example_data/squamous_cell_carcinoma_histopathology.jpeg (275x183 JPEG, NPU preprocess 后 224x224, mean/std 如上)NPU 输出 (python inference.py, npu:0):
image_embeds: (1, 512) dtype=torch.float32
text_embeds: (9, 512) dtype=torch.float32
logit_scale: 85.2323
similarity/logits: (1, 9)
Ranking (candidate text sorted by probability):
1. squamous cell carcinoma histopathology prob=0.997428 logit=41.2594
2. adenocarcinoma histopathology prob=0.001302 logit=34.6179
3. hematoxylin and eosin histopathology prob=0.001233 logit=34.5636
4. immunohistochemistry histopathology prob=0.000037 logit=31.0609
5. chest X-ray prob=0.000000 logit=16.0800
6. brain MRI prob=0.000000 logit=15.2882
7. pie chart prob=0.000000 logit=14.5926
8. covid line chart prob=0.000000 logit=13.4691
9. bone X-ray prob=0.000000 logit=9.8876
Top-1: squamous cell carcinoma histopathology (prob=0.997428) Expected: squamous cell carcinoma histopathology Match: True
Result: PASS
Inference latency (e2e, batch=1): 17.11 ms (warmup 3, 同步后计时)该结果与官方示例 9 类分类任务一致,正确区分鳞癌与腺癌病理。
同权重、同输入、同预处理,分别在 CPU 与 npu:0 运行,比较 embedding cosine 与 logits 误差:
python inference.py --mode validate
image_embeds cosine (NPU vs CPU): 0.999990
text_embeds cosine (NPU vs CPU): 0.999998
logits max_abs_err: 1.220684e-05 mean_abs_err: 2.789616e-06
Top-1 CPU: squamous cell carcinoma histopathology NPU: squamous cell carcinoma histopathology Match: True
Top-3 overlap: 3/3
Thresholds: FP32 cosine>=0.9999, max_err <1e-3, Top-1 must match
Consistency: PASS
Overall: PASS说明: 单样本 smoke consistency 验证,非完整数据集评测;FP32 下 cosine >=0.9999 且 Top-1 完全一致,证明 NPU 计算正确。
同步计时 torch.npu.synchronize() 前后,warmup 3 + formal 10 次,batch=1, 224x224, 9 texts, FP32, npu:0:
E2E latency ms: avg=16.19 min=16.12 max=16.42 p50=16.16 p90=16.28 p95=16.35
Image encode ms: avg=5.16 min=5.09 max=5.32 p50=5.15
Text encode ms (9 texts): avg=11.01 min=10.98 max=11.08 p50=11.00
NPU memory free=59.93 GiB total=61.27 GiB (推理峰值约 3112/65536 MB Chip0)首次编译已包含在 warmup;稳定延迟约 16.2 ms/图 (~61.7 images/s e2e),图像塔 ~5.2 ms,文本塔 (9 texts) ~11 ms。
assets/agent_workflow.png - 下载、侦察、加载、NPU推理、验证、性能全流程日志assets/npu_device_call.png - npu-smi info、torch.npu.is_available、device_name、param device/dtypeassets/model_result.png - python inference.py 真实零样本分类排序与 PASStrust_remote_codeHardware: NPU, NPU, Ascend, Ascend910, CLIP, BiomedCLIP, open_clip, vision-language, zero-shot