multi-qa-MiniLM-L6-cos-v1 是基于 MiniLM-L6-H384-uncased 的句子嵌入模型，专为 QA 检索任务优化。该模型将文本句子映射到 384 维稠密向量空间，可用于语义搜索、问答匹配、信息检索等任务。
multi-qa-MiniLM-L6-cos-v1-ascend/
├── inference.py # 推理测试脚本
├── log.txt # 测试日志
├── README.md # 本文档
├── test_sentences.txt # 测试句子
├── inference_result.json # 推理结果
└── precision_result.json # 精度测试结果docker exec -it test-modelagent bashsource /usr/local/Ascend/ascend-toolkit/set_env.sh模型文件位于 /data/ysws/agentsp/5-16/multi-qa-MiniLM-L6-cos-v1/ 目录下：
pip install transformers torch_npu -i https://pypi.huaweicloud.com/repository/pypi/simple/Run the inference script to extract sentence embeddings:
cd /data/ysws/agentsp/5-16/multi-qa-MiniLM-L6-cos-v1-ascend/
python3 inference.py
python3 inference.py --mode inference运行精度对比测试：
cd /data/ysws/agentsp/5-16/multi-qa-MiniLM-L6-cos-v1-ascend/
python3 inference.py --mode precision_test| 参数 | 说明 | 默认值 |
|---|---|---|
--mode | 测试模式: all, inference 或 precision_test | all |
| 指标 | 实测值 | 阈值 | 状态 |
|---|---|---|---|
| 最大相对误差 | 0.0910% | < 1.00% | PASS |
| CPU 推理时间 | 0.074s | - | - |
| NPU 推理时间 | 0.010s | - | - |
| 加速比 | 7.66x | > 1x | PASS |
| Cosine 相似度差 | 0.0000 | < 0.01 | PASS |
输入句子:
输出:
multi-qa-MiniLM-L6-cos-v1 NPU Test
Model: nreimers/MiniLM-L6-H384-uncased (sentence embeddings)
Output: /data/ysws/agentsp/5-16/multi-qa-MiniLM-L6-cos-v1-ascend
============================================================
Inference Test (NPU)
============================================================
Device: npu:0
Loading model and tokenizer...
Model loaded successfully
Input sentences: 3
Input shape: torch.Size([3, 12])
Inference time: 0.243s
Embeddings shape: torch.Size([3, 384])
Sample embedding (first 5 dims): [1.0559076070785522, 0.5140956044197083, -0.17752766609191895, 0.284322589635849, -0.2778976857662201]
============================================================
Precision Test (CPU vs NPU)
============================================================
NPU Device: npu:0
Loading model...
Input sentences: 3
Running on CPU...
Running on NPU...
CPU inference time: 0.074s
NPU inference time: 0.010s
Speedup: 7.66x
Max absolute error: 1.085460e-03
Max relative error: 0.0910% (threshold: 1.0%)
Cosine similarity (CPU): 0.5041
Cosine similarity (NPU): 0.5041
Status: PASS
============================================================
Precision Test Result: PASS
============================================================
============================================================
Test Complete!
============================================================import torch
from transformers import AutoTokenizer, AutoModel
MODEL_DIR = "/data/ysws/agentsp/5-16/multi-qa-MiniLM-L6-cos-v1"
tokenizer = AutoTokenizer.from_pretrained(MODEL_DIR)
model = AutoModel.from_pretrained(MODEL_DIR)
model = model.to("npu:0").eval()
sentences = ["Query sentence", "Passage to match"]
inputs = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt')
inputs = {k: v.to("npu:0") for k, v in inputs.items()}
def mean_pooling(model_output, attention_mask):
token_embeddings = model_output.last_hidden_state
input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)
with torch.no_grad():
outputs = model(**inputs)
embeddings = mean_pooling(outputs, inputs['attention_mask'])
print(f"Embeddings shape: {embeddings.shape}")from sklearn.metrics.pairwise import cosine_similarity
emb1 = embeddings[0].cpu().numpy()
emb2 = embeddings[1].cpu().numpy()
similarity = cosine_similarity([emb1], [emb2])[0][0]
print(f"Cosine similarity: {similarity:.4f}")| 组件 | 说明 |
|---|---|
| embeddings | BERT 词嵌入 (vocab_size=30522) |
| encoder | 6 层 Transformer 编码器 |
| 1_Pooling | 基于令牌嵌入的 Mean Pooling |
从 config.json 提取的关键参数:
{
"hidden_size": 384,
"intermediate_size": 1536,
"num_attention_heads": 12,
"num_hidden_layers": 6,
"vocab_size": 30522
}A: 检查 NPU 驱动是否正确安装。MiniLM 模型在 CPU 和 NPU 上的数值误差极小（< 0.1%），远低于 1% 阈值。
A: 使用批处理可以显著提高吞吐量。NPU 相比 CPU 有显著加速（7.66x）。
A: 可用于：
本项目遵循 Apache-2.0 许可证