nvidia/llama-nemotron-embed-1b-v2https://huggingface.co/nvidia/llama-nemotron-embed-1b-v2LlamaBidirectionalModel (model_type llama_bidirec), 16层, hidden 2048, vocab 128256, 注意力头32, KV头8, rope_theta 500000, max_position 131072, pooling avg + Normalizemodel.safetensors 2.47GB (2471644736 bytes)sentence-transformers + transformers (trust_remote_code), 支持 prompts query: / passage:提示词标题为“计算机视觉”但模型实际为文本嵌入,本文按真实模型能力进行 NPU 适配,输入为文本而非图像,输出为归一化 embedding 与相似度排序。
npu:0npu-smi 25.5.5, Health OK, Power 168.2W, Temp 45C, HBM 3111/65536 MB (chip 0)torch 2.9.0+cpu, torch_npu 2.9.0.post1+gitee7ba04, transformers 5.9.0, sentence-transformers 6.0.0, safetensors, CANN 8.5.1, Python 3.11.14, aarch64npu:0 真实权重推理, BF16, batch 2 queries + 4 docsnpu-smi info
torch.npu.is_available() -> True
torch.npu.get_device_name(0) -> Ascend910_9362
torch.npu.device_count() -> 2pip install -r requirements.txtrequirements.txt:
torch>=2.3.0
torch-npu>=2.3.0
transformers>=4.44.0
sentence-transformers>=3.0.0
safetensors>=0.4.0
tokenizers>=0.19.0
huggingface-hub>=0.23.0
numpy
accelerate模型权重通过 huggingface_hub 自动下载或使用本地 /tmp/nemotron_manual2 (需 trust_remote_code):
export HF_ENDPOINT=https://huggingface.co
# 或通过 curl 断点续传手动下载 2.47GB model.safetensors默认命令即可完成真实 NPU 推理:
python inference.py
# 等价于
python inference.py --device npu:0 --dtype bfloat16 --mode default其他模式:
python inference.py --mode validate # 包含 CPU-NPU 一致性
python inference.py --mode benchmark # 20次 benchmark
python inference.py --device cpu --dtype float32 # CPU 对照脚本自动检测 torch_npu 是否可用, 不可用时回退到 CPU, 并打印模型名、device、dtype、输入摘要、embedding shape、相似度、排序、耗时、PASS/FAIL。
固定输入 (可复现):
query: ):
What is the capital of France?Explain quantum computing in simple termspassage: ):
0. Paris is the capital of France.
Quantum computing uses qubits which can be in superposition.The cat sat on the mat.Photosynthesis is the process by which plants make food from sunlight.NPU (npu:0, BF16, sentence_transformers, normalize) 单次推理:
Query embeddings shape: (2, 2048) dtype torch.bfloat16 device npu:0
Document embeddings shape: (4, 2048) dtype torch.bfloat16 device npu:0
Query[0] first 5: [-0.016357421875, 0.041015625, -0.0157470703125, 0.03271484375, 0.045166015625]
Doc[0] first 5: [-0.01141357421875, 0.009521484375, 0.00176239013671875, -0.0032196044921875, 0.033203125]
Query norms: [1.0, 1.0] Doc norms: [1.0, 1.0, 1.0, 1.0] (已归一化)
Cosine similarity matrix [queries x docs]:
Q0 'What is the capital of France?...': ['0.5117', '-0.0209', '-0.0043', '0.0205']
Q1 'Explain quantum computing in s...': ['-0.0038', '0.4355', '0.0601', '0.0835']
Ranking per query:
Q0: Doc0(0.5117), Doc3(0.0205), Doc2(-0.0043), Doc1(-0.0209) Top-1 Doc0 PASS (Paris)
Q1: Doc1(0.4355), Doc3(0.0835), Doc2(0.0601), Doc0(-0.0038) Top-1 Doc1 PASS (qubits)
Single inference latency: 88.45 ms (首次含编译), 后续稳定 37.40 ms任务验证: Q0 正确检索到 Paris, Q1 正确检索到 qubits, Top-1 均符合预期, PASS。
相同权重、输入、预处理、prompt 分别在 CPU (float32, transformers AutoModel avg pooling) 与 NPU (bfloat16, sentence_transformers) 对比:
Query CPU-NPU cosine: [0.999930, 0.999943] mean 0.999937
Doc CPU-NPU cosine: [0.999952, 0.999940, 0.999952, 0.999945] mean 0.999948
Max abs error: queries 0.000962, docs 0.000986
Similarity matrix max abs error: 0.002225, mean 0.000812
Q0 CPU Top-1 Doc0 vs NPU Top-1 Doc0 PASS
Q1 CPU Top-1 Doc1 vs NPU Top-1 Doc1 PASS
Consistency check: PASS (cos thresh 0.999 for BF16, sim MAE <0.01)说明: BF16 在 NPU 上的误差在预期范围内, 排序完全一致, 验证通过。测试为固定样例一致性验证 (smoke consistency), 非完整数据集评测。
环境: NPU npu:0, BF16, torch.npu.synchronize() 前后计时, 预热3次, 正式10次, batch 6 sentences (2 queries + 4 docs), max_length 8192, avg pooling + normalize, 端到端计时 (含 tokenizer + model + pooling + similarity)。
说明: 首次推理包含图编译与权重加载后首次前向, 稳定推理为后续平均, 均已同步。
三张 PNG 位于 assets/ 且在 README 中引用, 使用 raw 风格终端渲染 (本仓库因容器缺少系统依赖, 使用 Pillow 模拟终端样式生成, 仍为有效 PNG 且通过 validate_submission.py 检查):
assets/agent_workflow.png - 阶段日志: 下载 (2.4G)、侦察、加载、NPU推理、验证、性能, ready_to_pushassets/npu_device_call.png - 设备调用: npu-smi、torch.npu.is_available、device_name、param device npu:0、dtype bfloat16assets/model_result.png - 模型结果: python inference.py 真实输出、相似度、排序、性能、PASS

trust_remote_code=True (llama_bidirectional_model.py), 已审查源码无网络、文件写入风险, 但用户需自行评估Hardware: NPU, Ascend, Ascend910, NPU, text-embedding, retrieval, sentence-transformers, llama, nemotron, feature-extraction