冬
gcw_IDzXRVNw/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 昇腾 NPU 适配部署文档

1. 模型简介

CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 是 LAION 基于 OpenCLIP 训练的一个 CLIP 模型,使用 ConvNeXt-Base 作为视觉编码器。

本模型是 CLIP-convnext_base_w 系列之一,在 LAION-2B 数据集上训练,使用增强正则化 (AugReg)。

模型架构

属性值
模型类型CLIP (OpenCLIP)
视觉编码器ConvNeXt-Base (timm)
文本编码器Transformer (12 层, 10 头)
嵌入维度640
图像分辨率256x256
上下文长度77
词表大小49408
参数量~400M (估算)
权重文件大小~718 MB

模型特点

  • ConvNeXt 视觉编码器: 相比 ViT 更适合密集预测任务
  • OpenCLIP 训练: 在 LAION-2B (13B 样本) 上训练
  • 增强正则化 (AugReg): 使用 Random Resize Crop、Random Erasing、Stochastic Depth
  • Zero-shot 分类: ImageNet Zero-Shot top-1 达 71.5%
  • 图文匹配: 支持图像-文本检索和相似度计算

输入输出格式

  • 图像输入: RGB 图像 (256x256)
  • 文本输入: 文本字符串(最长 77 tokens)
  • 输出: 图像特征和文本特征的余弦相似度

2. 环境依赖清单

2.1 硬件要求

  • 华为昇腾系列 AI 处理器(Ascend 910B/310P 等)
  • 建议 ≥ 3 GB NPU 显存(CLIP-convnext_base 推理峰值约 2-3 GB)
  • 本次实测使用 910B2(单卡 64 GB)

2.2 软件依赖

依赖项版本要求说明
操作系统CentOS 7.6+ / Ubuntu 18.04+支持主流 Linux 发行版
CANN5.0.RC2+华为昇腾计算架构
Python3.10 / 3.12推荐 3.10+
PyTorch2.0.0+推荐 2.10
torch_npu与 PyTorch 版本匹配昇腾 PyTorch 扩展
open_clip_torch最新版OpenCLIP 模型加载
timm最新版ConvNeXt 视觉编码器
Pillow最新版图像处理
numpy最新版数值计算
ftfy最新版文本修复

2.3 安装方式

# 1) 激活已预装 torch / torch_npu 的 conda 环境
conda activate pt2100

# 2) 安装模型依赖(使用华为云源)
pip install --index-url https://repo.huaweicloud.com/repository/pypi/simple \
    open_clip_torch timm Pillow numpy ftfy

# 3) 验证环境
python -c "
import torch, torch_npu
print('torch:', torch.__version__)
print('torch_npu:', torch_npu.__version__)
print('NPU 可用:', torch.npu.is_available())
"

3. 分步推理操作流程

步骤 1:准备工作目录

# 进入适配目录
cd /workspace/agent1/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend

# 确认上游权重存在
ls -la /workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg/

# 期望输出:open_clip_config.json open_clip_model.safetensors (~718MB) ...

步骤 2:检查 NPU 环境

# 检查 NPU 设备状态
npu-smi info

# 验证 torch_npu 安装
python -c "import torch; import torch_npu; print('NPU 可用:', torch.npu.is_available())"

步骤 3:运行推理脚本

# 基本用法:默认参数
python inference.py --device npu:0

# 查看 NPU 显存使用
python inference.py --device npu:0 --show_npu_memory

# 指定其他 NPU 卡
python inference.py --device npu:1

4. 完整测试用例

测试用例 1:图文特征提取和相似度计算(NPU:0)

conda activate pt2100
cd /workspace/agent1/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend
python inference.py --device npu:0

实际输出:

[W822 08:39:21.308243390 FunctionLoader.cpp:48] Warning: LD_PRELOAD detected...
WARNING:root:No pretrained weights loaded for model 'convnext_base_w'. Model initialized randomly.
[W822 08:39:39.206082400 NPUCachingAllocator.cpp:201] Warning: The current CANN and Soc versions require processing for 32 padding size, with memory allocation. (function operator())
============================================================
CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 昇腾 NPU 推理
============================================================

[Device] npu:0
[Model Path] /workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg

[Loading] Model config...
  embed_dim: 640
  vision_cfg: {'timm_model_name': 'convnext_base', ...}
  text_cfg: {'context_length': 77, 'vocab_size': 49408, 'width': 640, 'heads': 10, 'layers': 12}

[Loading] OpenCLIP model architecture...
  Model architecture created

[Loading] Model weights from safetensors...
  Loaded 493 weights
  Missing keys: 0
  Unexpected keys: 0
  Model loaded successfully

[Model Loaded] Allocated: 0.67 GB, Reserved: 0.76 GB

[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2                   Version: 25.5.2                                               |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 104.0       41                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          4263 / 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 1237225       | python                   | 894                    |
+===========================+===============+====================================================+


[Creating] Dummy image and text...

[Input Image] Size: (256, 256)
[Input Text]: ['a photo of a cat', 'a photo of a dog', 'a photo of a bird']

[Preprocessing image]...

[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2                   Version: 25.5.2                                               |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 104.3       42                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          4263 / 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 1237225       | python                   | 894                    |
+===========================+===============+====================================================+


[Running Image Encoding]...

[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2                   Version: 25.5.2                                               |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 105.1       41                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          4299 / 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 1237225       | python                   | 930                    |
+===========================+===============+====================================================+


[Running Text Encoding]...

[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2                   Version: 25.5.2                                               |
+---------------------------+---------------+----------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip                      | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     910B2               | OK            | 105.1       42                0    / 0             |
| 0                         | 0000:C1:00.0  | 0           0    / 0          4309 / 65536         |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 1237225       | python                   | 940                    |
+===========================+===============+====================================================+


[Computing Similarity]...

[Results]
  Image features shape: torch.Size([1, 640])
  Text features shape: torch.Size([3, 640])
  Similarity matrix shape: torch.Size([1, 3])
  Similarity scores: [0.270, 0.275, 0.270]
[Memory After Inference] Allocated: 0.67 GB, Reserved: 0.79 GB

[Done]

✅ 验证要点:

  • 模型加载成功: OpenCLIP 模型架构创建 + 493 个权重加载(0 missing, 0 unexpected)
  • NPU 推理稳定: 图像编码和文本编码都在 NPU 上运行
  • NPU 显存峰值: Allocated 0.67 GB, Reserved 0.79 GB(约 940 MB 进程内存)
  • 输出形状: Image features (1, 640), Text features (3, 640)
  • 进程监控: npu-smi 显示 Python 进程 (PID 1237225) 在 NPU 上运行

5. 常见问题解答

Q1:提示 "NPU 不可用" 怎么办?

解决方案:

  1. 确认已安装 CANN 驱动并设置环境变量(source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh)
  2. 运行 npu-smi info 检查设备状态
  3. 确认 PyTorch 和 torch_npu 版本匹配

Q2:模型加载失败怎么办?

解释:OpenCLIP 模型需要正确加载权重。确认 open_clip_model.safetensors 文件存在且完整。

Q3:相似度分数异常低?

解释:使用随机图像和文本输入时,相似度分数是随机的。实际使用时,输入真实图像和匹配的文本描述时,相似度分数会更高。

6. 文件结构

CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend/
├── inference.py              # 推理脚本(torch_npu 推理)
├── README.md                 # 本文档
├── requirements.txt          # 依赖列表
└── assets/                   # 资源占位目录

上游权重目录(只读,未被修改):

/workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg/
├── open_clip_config.json     # OpenCLIP 配置
├── open_clip_model.safetensors  # 权重(约 718MB)
├── open_clip_pytorch_model.bin  # PyTorch 格式权重
├── tokenizer.json            # 分词器
├── vocab.json                # 词表
└── README.md                 # 原始文档

7. requirements.txt - 依赖列表

# CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 昇腾 NPU 推理环境依赖

# 核心依赖(必须)
torch>=2.0.0
torch_npu
open_clip_torch
timm
Pillow
numpy
ftfy

使用方式:

# 激活环境
conda activate pt2100

# 安装依赖(华为云源)
pip install --index-url https://repo.huaweicloud.com/repository/pypi/simple -r requirements.txt

8. 参考资源

  • OpenCLIP GitHub
  • HuggingFace 模型页
  • LAION 5B Blog
  • ConvNeXt 论文
  • 华为昇腾文档
  • torch_npu GitHub