BertModel (无分类头, sentence-embedding)sentence_transformers 三段流水 Transformer(0) + Pooling(1, cls) + Normalize(2)model.safetensors + 418M pytorch_model.bin, 0.408 GiB, sha256_1mb a8c112cf0acfc31a)AutoTokenizer BertTokenizer (vocab 30522) 经 padding=True truncation=True max_length=512 → input_ids [batch, seq_len] + attention_mask + token_type_ids (int64)last_hidden_state [batch, seq_len, 768] → cls = hidden[:,0,:] → F.normalize(cls, p=2) → [batch,768] float32 归一化向量 (norm=1.0), 相似度 cosine 相近句 > 无关句model.safetensors (437955512 bytes, 0.408 GiB) 完全一致, 通过 AutoModel.from_pretrained(..., trust_remote_code=False) 本地加载, 不替换为随机权重torch.npu.is_available()=True, device_count=2, device_name=Ascend910_9362, npu:0 可用, 显存 free 62373.2 MB / total 62740.0 MB (检查时)npu:0 推理, 非 CPU 降级, 同权重同输入对比 CPU, 同 dtype float32 同 pre-processingpip install -r requirements.txt
# requirements.txt:
# torch==2.9.0
# torch-npu==2.9.0.post1
# transformers>=4.44.0
# numpy<2.0
# huggingface_hub>=0.20.0
# tokenizers权重获取 (已在 /tmp/bge-base-en-v1.5 本地缓存, 亦可自动下载):
# 方式1: HF (走代理, 已验证 418M)
export HF_ENDPOINT=https://hf-mirror.com
export https_proxy=http://127.0.0.1:7890
huggingface-cli download BAAI/bge-base-en-v1.5 --local-dir /tmp/bge-base-en-v1.5
# 方式2: 通用 huggingface_hub
python -c "from huggingface_hub import snapshot_download; snapshot_download('BAAI/bge-base-en-v1.5', local_dir='/tmp/bge-base-en-v1.5')"
# 验证
ls -lh /tmp/bge-base-en-v1.5/model.safetensors
# -rw-r--r-- 418M /tmp/bge-base-en-v1.5/model.safetensors (437955512 bytes)
# config.json, tokenizer.json, vocab.txt, 1_Pooling/config.json默认命令 (真实 NPU, 自动解析 /tmp/bge-base-en-v1.5):
python inference.py
# 或显式指定:
python inference.py --mode all # 推理 + 一致性 + 性能 (默认)
python inference.py --mode validate # 仅一致性验证
python inference.py --mode benchmark # 仅性能测试
python inference.py --model-dir /tmp/bge-base-en-v1.5AutoModel.from_pretrained(..., torch_dtype=float32) → 搬运至 npu:0, 验证 first_param.device == npu:0BertTokenizer(padding=True truncation=True max_length=512) → [4,11] input_ids (int64)model(input_ids, attention_mask) → last_hidden_state [4,11,768] → cls pooling → F.normalize → [4,768] 归一化向量torch.npu.synchronize() 计时以下为本次 python inference.py --model-dir /tmp/bge-base-en-v1.5 在 npu:0 上的真实输出 (非模拟, 截取关键):
Model: BAAI/bge-base-en-v1.5
Task: embedding (sentence-embedding, retrieval)
Architecture: BertModel (bert, 12 layers, hidden 768, cls pooling)
Loading backend: transformers AutoModel.from_pretrained (BertModel)
Requested dtype: float32 (torch.float32)
Weight URL: https://huggingface.co/BAAI/bge-base-en-v1.5
Model dir resolved: /tmp/bge-base-en-v1.5
Weight file: /tmp/bge-base-en-v1.5/model.safetensors bytes=437955512 (0.408 GiB)
Weight sha256_1mb: a8c112cf0acfc31a
Device: npu:0 (torch.npu.is_available=True)
NPU device name: Ascend910_9362
NPU device count: 2
NPU memory free 62373.2 MB / total 62740.0 MB
Tokenizer: BertTokenizerFast vocab_size=30522
Model config: model_type=bert hidden_size=768 layers=12
Model total params: 109.48 M
Model first param device: npu:0 dtype: torch.float32
Input sentences (4):
- A man is eating food.
- A man is eating a piece of bread.
- The girl is carrying a baby.
- The weather is sunny with clear skies.
Tokenized input_ids shape: (4, 11) dtype=torch.int64 device=cpu
...
[Inference] running on npu:0 ...
Inference done: time 253.63 ms
last_hidden_state: (4, 11, 768) dtype=torch.float32 device=npu:0 min -5.1369 max 2.5017 mean -0.0187
cls pooled: (4, 768) device=npu:0
normalized embeddings: (4, 768) dtype=torch.float32 device=npu:0 norm mean 1.0000
embedding[0][:8]: ['0.0121', '-0.0214', '0.0056', ...]
NPU semantic cosine sim(0,1)=0.8269 dis(0,3)=0.4895 -> PASSnpu:0 上得到 768维归一化句向量 (norm 1.0), 相近句 cosine 0.8269 显著 > 无关句 0.4895, 符合检索语义, 且落在 npu:0 上非 dummynpu:0modules.json 显示流水为 Transformer + Pooling(cls) + Normalize, 与本推理一致相同权重 (/tmp/bge-base-en-v1.5 418M)、输入、预处理、dtype float32 分别在 CPU 与 NPU 上运行, 对比 F.normalize(cls) 后向量:
[CPU-NPU Consistency] running ...
sentence 0 cosine 0.999995 max_diff 0.000385
sentence 1 cosine 0.999995 max_diff 0.000422
sentence 2 cosine 0.999995 max_diff 0.000380
sentence 3 cosine 0.999996 max_diff 0.000337
avg_cos 0.999995 flat_cos 0.999995 mae 0.000090 max_diff_all 0.000422
semantic sim(0,1)=0.8269 dis(0,3)=0.4895
Consistency: PASS (thresholds: cosine>=0.999 max_diff<=1e-3)计时均在 torch.npu.synchronize() 前后取 perf_counter, 预热 3 次, 正式 10 次, batch=4, seq_len=11, float32:
[Benchmark] warmup 3, runs 10 (with torch.npu.synchronize)...
Full forward: avg 8.71ms min 8.44ms max 9.43ms p50 8.55ms p90 9.34ms p95 9.39ms
Peak NPU memory: 420.54 MB
Input shape: [4, 11] dtype=float32 batch=4
Throughput: 459.17 sentences/s (batch=4, seq_len≈11)
Note: first inference 253.63ms includes compilation; stable avg 8.71msassets/agent_workflow.png - 下载、侦察、加载、NPU推理、验证、性能全流程日志assets/npu_device_call.png - npu-smi info + torch.npu.is_available() + 设备名 + 参数 deviceassets/model_result.png - python inference.py 完整输出与 PASSHardware: NPU, NPU, Ascend, Ascend910, BGE, embedding, BertModel, sentence-embedding, retrieval