w
gcw_uQ09W7jl/patrickjohncyh-fashion-clip-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

patrickjohncyh/fashion-clip on Ascend NPU

1. 模型简介

  • 模型名称: patrickjohncyh/fashion-clip
  • 模型类型: 计算机视觉 / 双塔图文检索 (CLIP)
  • 架构: CLIPModel (ViT-B/32 Vision + Transformer Text, fine-tuned on ~800k Farfetch fashion images)
  • 官方地址: https://huggingface.co/patrickjohncyh/fashion-clip
  • 配置: model_type=clip, architectures=["CLIPModel"], projection_dim=512, vision_hidden=768 (12层), text_hidden=512 (12层), patch_size=32, image_size=224
  • 权重格式: model.safetensors (605 MB) + pytorch_model.bin (605 MB), 1.13 GiB 合计, 实际加载 safetensors
  • 任务: 零样本图像分类 / 图文检索 (embedding + similarity ranking)
  • 适配精度: float32 (原始权重 float32, 单卡 Ascend910B 实测)
  • 加载后端: transformers CLIPProcessor + CLIPModel 从本地快照加载 (local_files_only=True), 不依赖运行时下载

2. 验证环境

  • 硬件: Ascend910_9362 (npu:0), 2× Ascend910, HBM 64 GB
  • NPU 驱动: npu-smi 25.5.5, Health OK, Power ~160W, Temp 44°C
  • 软件: torch 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, CANN 8.5.1, transformers 4.44.0, Pillow, numpy
  • 系统: Linux aarch64, Python 3.11.14
  • 检查命令:
    npu-smi info
    python -c "import torch; import torch_npu; print(torch.npu.is_available()); print(torch.npu.get_device_name(0))"
    输出: torch.npu.is_available()=True, device_name=Ascend910_9362, device_count=2, passed=true
  • 模型路径: /opt/atomgit/.cache/huggingface/hub/models--patrickjohncyh--fashion-clip/snapshots/7e3ba62ce16b379a1ab479346b66f192e76f51b7
  • 显存预估: 权重 605 MB (FP32) + 激活/编译 ~600 MB, 峰值 <2 GB, 实测 NPU 已用 1204.6 MB, 满足单卡 64GB

3. 安装依赖

pip install -r requirements.txt
# requirements 包含 torch, torch-npu, transformers, Pillow, numpy, huggingface_hub
# 已在环境中安装 torch-npu 与 CANN, 无需手动设置代理或 HF_ENDPOINT (已配置 hf-mirror)

4. NPU 推理

默认命令 (真实 NPU):

python inference.py
# 或显式指定本地权重目录
python inference.py --model-dir /opt/atomgit/.cache/huggingface/hub/models--patrickjohncyh--fashion-clip/snapshots/7e3ba62ce16b379a1ab479346b66f192e76f51b7

其他模式:

python inference.py --mode validate   # 完整验证 (含 CPU-NPU 一致性)
python inference.py --mode benchmark  # 性能测试 (预热3次, 测试10次, torch.npu.synchronize 计时)

输入契约 (固定可复现):

  • 图像: 1×3×224×224 RGB 合成时装图 (224x224, 淡色背景, 红色连衣裙轮廓: 上身体矩形 + 裙摆梯形 + 黑色腰带, 确定性 Pillow 生成)
  • 文本: 6 条候选, 按固定顺序:
    1. a photo of a dress
    2. a photo of a t-shirt
    3. a photo of jeans
    4. a photo of a sneaker
    5. a photo of a handbag
    6. a photo of a jacket
  • 预处理: CLIPProcessor (resize 224, center_crop, rescale 1/255, normalize mean [0.48145466,0.4578275,0.40821073] std [0.26862954,0.26130258,0.27577711]), tokenizer pad to 9 tokens
  • 设备: npu:0, dtype float32, batch 1

5. 真实推理结果

以下为本次在 npu:0 上 python inference.py 的真实输出 (首轮含编译):

Model: patrickjohncyh/fashion-clip
Task: zero-shot image classification / image-text retrieval (dual-encoder CLIP)
Device: npu:0  torch="2.9.0+cpu"  torch_npu="2.9.0.post1"  NPU name="Ascend910_9362"
Requested dtype: torch.float32  batch_size="1"  image_size="224"
Input: 1×3×224×224 synthetic dress image + 6 texts
Output contract:
  image_embeds: (1, 512)  dtype="torch.float32"  device="npu:0"
  text_embeds:  (6, 512)  dtype="torch.float32"  device="npu:0"
  logits_per_image: (1, 6)  dtype="torch.float32"
Inference latency (single, synchronized): 275.91 ms  (first run includes NPU compilation)
Similarity (logits_per_image) raw scores (CLIP):
  [0] a photo of a dress             logit="24.9999"  softmax="0.2563"
  [1] a photo of a t-shirt           logit="22.0199"  softmax="0.0130"
  [2] a photo of jeans               logit="22.5441"  softmax="0.0220"
  [3] a photo of a sneaker           logit="24.4217"  softmax="0.1437"
  [4] a photo of a handbag           logit="25.7809"  softmax="0.5596"
  [5] a photo of a jacket            logit="21.1410"  softmax="0.0054"
Ranking (desc by logit):
  Rank 1: [4] a photo of a handbag  logit="25.7809"  prob="0.5596"
  Rank 2: [0] a photo of a dress  logit="24.9999"  prob="0.2563"
  Rank 3: [3] a photo of a sneaker  logit="24.4217"  prob="0.1437"
  Rank 4: [2] a photo of jeans  logit="22.5441"  prob="0.0220"
  Rank 5: [1] a photo of a t-shirt  logit="22.0199"  prob="0.0130"
  Rank 6: [5] a photo of a jacket  logit="21.1410"  prob="0.0054"
Top-1: "a photo of a handbag"
Note: Fashion-CLIP uses CLIP softmax over candidates (not SigLIP sigmoid).

说明:

  • 采用模型原生 logits_per_image (image_embeds @ text_embeds.T * logit_scale.exp()) 并对候选集合做 softmax 得到分布
  • 合成图为连衣裙轮廓但受颜色/形状影响, 在 Fashion-CLIP 空间中手袋得分略高于连衣裙; 排序完整记录可复现, 不宣称 Top-1 必须为 dress
  • 所有数值来自本次 npu:0 真实执行, 设备与 dtype 已验证 (param device="npu:0)"

6. CPU-NPU 一致性验证

相同权重、相同输入、相同预处理分别在 CPU 与 NPU 上运行, 报告:

image_embeds cosine: 1.000000
text_embeds cosine mean: 1.000000 min: 1.000000 per-text: ['1.000000', '1.000000', '1.000000', '1.000000', '1.000000', '1.000000']
logits max_abs_err: 0.003529 mean_abs_err: 0.002101
Top-1 CPU: [4] a photo of a handbag  NPU: [4] a photo of a handbag  (一致)
Ranking CPU: [4, 0, 3, 2, 1, 5]  NPU: [4, 0, 3, 2, 1, 5]  Top3 overlap: 3/3

阈值与判定 (float32):

  • image/text embedding cosine >= 0.9999 (实测 1.0, 通过)
  • logits max_abs <= 0.01 (实测 0.0035, 通过)
  • Top-1 必须一致 (通过)
  • 候选数 6 < 10, 排序应全部一致 (实测全一致, 通过)

结论: [PASS] CPU-NPU consistency within tolerance

验证方法为单样本 smoke consistency (固定合成样例), 非完整数据集评测, 已在 inference.py 默认流程中自动执行。

7. 性能测试

计时区间前后均调用 torch.npu.synchronize(), 预热 3 次, 正式测试 10 次, batch 1, 224x224, 6 texts, float32, npu:0:

Full end-to-end (image+text+similarity): avg=13.90ms min=13.44ms max=14.38ms p50=13.89ms p90=14.36ms p95=14.37ms
Throughput: 71.93 inferences/s
Image encoder only: avg=6.18ms min=6.12ms max=6.49ms
Text encoder (6 texts): avg=7.21ms min=7.11ms max=7.29ms
NPU memory used: 1204.6 MB / 62740 MB
First run (cold, with compilation): 275.91 ms
Stable (warm): ~13.44ms (min) to 13.90ms (avg)

说明:

  • 端到端含图像编码、文本编码、相似度计算
  • 首次编译延迟单独标注, 稳定延迟为预热后均值
  • 图像/文本编码分别计时, 文本编码含 6 条文本的 batch 处理

复现:

python inference.py --mode benchmark

8. 自验证截图

  • assets/agent_workflow.png — 阶段日志 (下载、侦察、加载、NPU推理、验证、性能)
  • assets/npu_device_call.png — npu-smi info、torch.npu.is_available()、设备名、模型参数设备与 dtype
  • assets/model_result.png — python inference.py 完整输出、任务结果、计时与 PASS/FAIL

图片由 scripts/render_xterm_evidence.mjs 从本次真实日志渲染 (--style raw), 提示符为 atomgit@pod-a94f8701860f4700b161b00e290de466:~$, 不代表实际主机名。

9. 已知限制

  • 合成固定图非真实 Farfetch 时装照片, 仅用于可复现的 smoke 验证; 真实场景建议替换为业务图像并重新计时
  • 当前为 float32; Large/Huge 模型可切换 FP16/BF16 但本模型 (ViT-B/32, 605MB) 在 FP32 下已满足单卡显存与速度 (13.9ms)
  • 文本长度固定为 tokenizer pad 后 9 tokens; 超长文本会被截断至 77
  • 首次 NPU 编译约 276ms, 冷启动与热启动差异已在 README 明确区分
  • 未做完整数据集精度评测 (如 Fashion-MNIST/DeepFashion), 仅提供单样本 CPU-NPU 一致性; 如需数据集评测需额外脚本

10. 标签

Hardware: NPU, NPU, Ascend, Ascend910, CLIP, vision, fashion, dual-encoder, zero-shot

11. 目录结构

patrickjohncyh-fashion-clip-NPU/
├── inference.py
├── readme.md
├── requirements.txt
└── assets/
    ├── agent_workflow.png
    ├── npu_device_call.png
    └── model_result.png

12. 复现命令汇总

# 环境检查
npu-smi info
python -c "import torch; import torch_npu; print(torch.npu.is_available()); print(torch.npu.get_device_name(0))"

# 默认推理 (NPU)
python inference.py

# 验证 (含 CPU-NPU 一致性)
python inference.py --mode validate

# 性能
python inference.py --mode benchmark