BiomedCLIP is a biomedical vision-language foundation model pretrained on PMC-15M (15M figure-caption pairs from PubMed Central) with PubMedBERT (text encoder) and ViT-B/16 (vision encoder, image_size 224, embed_dim 512, context_length 256). This repository adapts the original microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 (open_clip open_clip_pytorch_model.bin 784 MB) for single-card Ascend 910B (npu:0) real inference via open_clip + torch_npu.
microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224https://huggingface.co/microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224 (open_clip_pytorch_model.bin 783.7 MB, FP32)timm_model_name: vit_base_patch16_224, image_size: 224) + HF microsoft/BiomedNLP-BiomedBERT-base-uncased-abstract text encoder (768→512 projection, context 256)image_embeds [N,512], text_embeds [M,512], similarity [N,M] 与完整标签排序)open_clip.create_model_from_pretrained('hf-hub:microsoft/...') + open_clip.get_tokenizer, 权重通过 HF_ENDPOINT=https://hf-mirror.com 缓存到本地,无运行时隐式下载外模型npu-smi 25.5.5, 64 GB HBM, torch.npu.is_available()=True/usr/local/Ascend/cann-8.5.12.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, Transformers 5.9.0, open_clip_torch 3.3.0, timm 1.0.28, Pillow 11npu:0 (verified next(model.parameters()).device == npu:0)squamous_cell_carcinoma_histopathology.jpeg from example_data) + 9 candidate texts with template "this is a photo of " (see below)pip install -r requirements.txt
# 需已安装 CANN 8.5+ 且 torch_npu 与 torch 版本匹配
# 模型权重首次会通过 hf-mirror 自动缓存到 HF cache,离线环境可预先:
# huggingface_hub snapshot_download --repo-id microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224默认命令(单张真实生物医学图像,NPU 同步计时,打印完整排序):
python inference.py
# 或显式
python inference.py --mode infer --dtype float32 --device npu:0CPU-NPU 一致性验证(相同权重、输入、预处理):
python inference.py --mode validate性能测试(预热 3 次,正测 10 次,torch.npu.synchronize() 包计时):
python inference.py --mode benchmark环境: npu:0, FP32, logit_scale=85.2323, 输入 1×3×224×224 (squamous_cell_carcinoma_histopathology.jpeg), 9 labels
命令: python inference.py (exit 0)
Model: microsoft/BiomedCLIP-PubMedBERT_256-vit_base_patch16_224
Task: zero-shot image classification / retrieval (dual-encoder)
Device: npu:0 (param device: npu:0)
Dtype: torch.float32 (requested: float32)
Input: image torch.Size([1, 3, 224, 224]) from /tmp/biomedclip/example_data/biomed_image_classification_example_data/squamous_cell_carcinoma_histopathology.jpeg, texts=9
Labels: ['adenocarcinoma histopathology', 'brain MRI', 'covid line chart', 'squamous cell carcinoma histopathology', 'immunohistochemistry histopathology', 'bone X-ray', 'chest X-ray', 'pie chart', 'hematoxylin and eosin histopathology']
Template: 'this is a photo of '
--- Outputs ---
image_embeds: (1, 512) (hidden_size=512)
text_embeds: (9, 512) (num_labels=9, hidden_size=512)
similarity/logits shape: (1, 9)
logit_scale: 85.2323
Ranking (softmax over 9 candidates):
1. squamous cell carcinoma histopathology: 0.997428 <- TOP-1
2. adenocarcinoma histopathology: 0.001302
3. hematoxylin and eosin histopathology: 0.001233
4. immunohistochemistry histopathology: 0.000037
5. chest X-ray: 0.000000
6. brain MRI: 0.000000
7. pie chart: 0.000000
8. covid line chart: 0.000000
9. bone X-ray: 0.000000
Cosine similarities (normalized, before logit_scale):
adenocarcinoma histopathology: 0.406160
brain MRI: 0.179371
covid line chart: 0.158028
squamous cell carcinoma histopathology: 0.484082
immunohistochemistry histopathology: 0.364426
bone X-ray: 0.116008
chest X-ray: 0.188661
pie chart: 0.171209
hematoxylin and eosin histopathology: 0.405523
Inference latency: 320.41 ms (first run incl. NPU compilation) / stable ~18 ms
NPU memory: allocated 750.6 MB, reserved 944.0 MB
[PASS] NPU inference successful on npu:0Top-1
squamous cell carcinoma histopathology与测试图一致;其余 biomedical 类别的概率分布符合领域预期(adenocarcinoma/H&E 次之,放射与图表类接近 0)。
其余 8 张 example_data 图像亦可替换测试,脚本固定排序逻辑与官方最小示例一致(open_clip 官方 softmax(logit_scale * image_features @ text_features.T))。
相同权重、图像、文本、预处理分别在 CPU 与 npu:0 上运行(FP32):
=== CPU-NPU Consistency Validation ===
Image embedding cosine: 0.999990 (threshold >=0.999)
Text embedding cosine mean: 0.999998 min: 0.999994 (threshold mean >=0.999)
Logits max AE: 1.228170e-05 mean AE: 2.819160e-06
Top-1 CPU: squamous cell carcinoma histopathology (0.997435) vs NPU: squamous cell carcinoma histopathology (0.997428) -> MATCH
Full ranking match: True (required for <10 candidates)
Top-3 overlap: 100.0%
Logit scales: CPU 85.2323 NPU 85.2323
Validation: PASS>=0.9999 期望,本次 >=0.99999,通过1.2e-05 (<1e-3)torch.npu.synchronize() 前后计时,warmup 3、iters 10、batch 1、224×224、FP32、NPU 910B:
=== Benchmark (warmup=3, iters=10) on npu:0 ===
End-to-end (1 image + 9 texts): avg 18.13 ms min 17.48 max 19.25 p50 17.98 p90 18.55 p95 18.90
Image encode: avg 5.90 ms (p50 5.88 p90 5.98)
Text encode (9 texts): avg 12.79 ms
Similarity calc included in e2e
Config: image 224x224x3, batch=1, texts=9, dtype=torch.float32, device=npu:0
Note: first run includes NPU compilation (~1-2s); reported avg is stable after warmup三张证据图由 scripts/render_xterm_evidence.mjs --style raw 从本次真实日志渲染(非 Pillow/Matplotlib):
assets/agent_workflow.png — 下载、侦察、加载、NPU 推理、验证、性能全阶段assets/npu_device_call.png — npu-smi info + torch.npu.is_available() + 设备名 + 模型参数 npu:0assets/model_result.png — python inference.py 完整输出与 PASS/FAILlogit_scale 为可学习标量(85.23),不同版本权重略有差异但不影响 ranking 契约。Hardware: NPU, NPU, Ascend, Ascend910, vision, clip, biomedical
Generated from real NPU execution on 2026-08-11, model commit from HF cache, no dummy weights.