CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 是 LAION 基于 OpenCLIP 训练的一个 CLIP 模型,使用 ConvNeXt-Base 作为视觉编码器。
本模型是 CLIP-convnext_base_w 系列之一,在 LAION-2B 数据集上训练,使用增强正则化 (AugReg)。
| 属性 | 值 |
|---|---|
| 模型类型 | CLIP (OpenCLIP) |
| 视觉编码器 | ConvNeXt-Base (timm) |
| 文本编码器 | Transformer (12 层, 10 头) |
| 嵌入维度 | 640 |
| 图像分辨率 | 256x256 |
| 上下文长度 | 77 |
| 词表大小 | 49408 |
| 参数量 | ~400M (估算) |
| 权重文件大小 | ~718 MB |
| 依赖项 | 版本要求 | 说明 |
|---|---|---|
| 操作系统 | CentOS 7.6+ / Ubuntu 18.04+ | 支持主流 Linux 发行版 |
| CANN | 5.0.RC2+ | 华为昇腾计算架构 |
| Python | 3.10 / 3.12 | 推荐 3.10+ |
| PyTorch | 2.0.0+ | 推荐 2.10 |
| torch_npu | 与 PyTorch 版本匹配 | 昇腾 PyTorch 扩展 |
| open_clip_torch | 最新版 | OpenCLIP 模型加载 |
| timm | 最新版 | ConvNeXt 视觉编码器 |
| Pillow | 最新版 | 图像处理 |
| numpy | 最新版 | 数值计算 |
| ftfy | 最新版 | 文本修复 |
# 1) 激活已预装 torch / torch_npu 的 conda 环境
conda activate pt2100
# 2) 安装模型依赖(使用华为云源)
pip install --index-url https://repo.huaweicloud.com/repository/pypi/simple \
open_clip_torch timm Pillow numpy ftfy
# 3) 验证环境
python -c "
import torch, torch_npu
print('torch:', torch.__version__)
print('torch_npu:', torch_npu.__version__)
print('NPU 可用:', torch.npu.is_available())
"# 进入适配目录
cd /workspace/agent1/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend
# 确认上游权重存在
ls -la /workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg/
# 期望输出:open_clip_config.json open_clip_model.safetensors (~718MB) ...# 检查 NPU 设备状态
npu-smi info
# 验证 torch_npu 安装
python -c "import torch; import torch_npu; print('NPU 可用:', torch.npu.is_available())"# 基本用法:默认参数
python inference.py --device npu:0
# 查看 NPU 显存使用
python inference.py --device npu:0 --show_npu_memory
# 指定其他 NPU 卡
python inference.py --device npu:1conda activate pt2100
cd /workspace/agent1/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend
python inference.py --device npu:0实际输出:
[W822 08:39:21.308243390 FunctionLoader.cpp:48] Warning: LD_PRELOAD detected...
WARNING:root:No pretrained weights loaded for model 'convnext_base_w'. Model initialized randomly.
[W822 08:39:39.206082400 NPUCachingAllocator.cpp:201] Warning: The current CANN and Soc versions require processing for 32 padding size, with memory allocation. (function operator())
============================================================
CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 昇腾 NPU 推理
============================================================
[Device] npu:0
[Model Path] /workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg
[Loading] Model config...
embed_dim: 640
vision_cfg: {'timm_model_name': 'convnext_base', ...}
text_cfg: {'context_length': 77, 'vocab_size': 49408, 'width': 640, 'heads': 10, 'layers': 12}
[Loading] OpenCLIP model architecture...
Model architecture created
[Loading] Model weights from safetensors...
Loaded 493 weights
Missing keys: 0
Unexpected keys: 0
Model loaded successfully
[Model Loaded] Allocated: 0.67 GB, Reserved: 0.76 GB
[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 104.0 41 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 4263 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1237225 | python | 894 |
+===========================+===============+====================================================+
[Creating] Dummy image and text...
[Input Image] Size: (256, 256)
[Input Text]: ['a photo of a cat', 'a photo of a dog', 'a photo of a bird']
[Preprocessing image]...
[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 104.3 42 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 4263 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1237225 | python | 894 |
+===========================+===============+====================================================+
[Running Image Encoding]...
[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 105.1 41 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 4299 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1237225 | python | 930 |
+===========================+===============+====================================================+
[Running Text Encoding]...
[npu-smi info]
+------------------------------------------------------------------------------------------------+
| npu-smi 25.5.2 Version: 25.5.2 |
+---------------------------+---------------+----------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page)|
| Chip | Bus-Id | AICore(%) Memory-Usage(MB) HBM-Usage(MB) |
+===========================+===============+====================================================+
| 0 910B2 | OK | 105.1 42 0 / 0 |
| 0 | 0000:C1:00.0 | 0 0 / 0 4309 / 65536 |
+===========================+===============+====================================================+
+---------------------------+---------------+----------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===========================+===============+====================================================+
| 0 0 | 1237225 | python | 940 |
+===========================+===============+====================================================+
[Computing Similarity]...
[Results]
Image features shape: torch.Size([1, 640])
Text features shape: torch.Size([3, 640])
Similarity matrix shape: torch.Size([1, 3])
Similarity scores: [0.270, 0.275, 0.270]
[Memory After Inference] Allocated: 0.67 GB, Reserved: 0.79 GB
[Done]✅ 验证要点:
(1, 640), Text features (3, 640)解决方案:
source /usr/local/Ascend/ascend-toolkit/latest/set_env.sh)npu-smi info 检查设备状态解释:OpenCLIP 模型需要正确加载权重。确认 open_clip_model.safetensors 文件存在且完整。
解释:使用随机图像和文本输入时,相似度分数是随机的。实际使用时,输入真实图像和匹配的文本描述时,相似度分数会更高。
CLIP-convnext_base_w-laion2B-s13B-b82K-augreg-ascend/
├── inference.py # 推理脚本(torch_npu 推理)
├── README.md # 本文档
├── requirements.txt # 依赖列表
└── assets/ # 资源占位目录上游权重目录(只读,未被修改):
/workspace/agent2/laion/CLIP-convnext_base_w-laion2B-s13B-b82K-augreg/
├── open_clip_config.json # OpenCLIP 配置
├── open_clip_model.safetensors # 权重(约 718MB)
├── open_clip_pytorch_model.bin # PyTorch 格式权重
├── tokenizer.json # 分词器
├── vocab.json # 词表
└── README.md # 原始文档# CLIP-convnext_base_w-laion2B-s13B-b82K-augreg 昇腾 NPU 推理环境依赖
# 核心依赖(必须)
torch>=2.0.0
torch_npu
open_clip_torch
timm
Pillow
numpy
ftfy使用方式:
# 激活环境
conda activate pt2100
# 安装依赖(华为云源)
pip install --index-url https://repo.huaweicloud.com/repository/pypi/simple -r requirements.txt