gcw_6a3w2yXz/bge-base-en-v1.5-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

BAAI/bge-base-en-v1.5 - Ascend NPU Adapted

This is an Ascend NPU-adapted deployment for BAAI/bge-base-en-v1.5 (BertModel, embedding).

  • Original model: https://huggingface.co/BAAI/bge-base-en-v1.5
  • Architecture: BertModel (bert, 12 layers, hidden 768, 12 heads) with CLS pooling
  • Route: embedding (AutoModel + CLS pooling + L2 normalize)
  • Device: npu:0 with synchronized timing and CPU consistency validation
  • Task: Sentence embedding / feature extraction

NPU Device Proof

Model and tensors are explicitly placed on npu:0 and verified:

  • next(model.parameters()).device == npu:0
  • input_ids.device == npu:0
  • attention_mask.device == npu:0
  • last_hidden_state.device == npu:0
  • pooled (CLS) device == npu:0
  • normalized embedding device == npu:0

Synchronization uses torch.npu.synchronize() before/after timing. First inference includes compilation, stable perf measured after warmup.

Consistency Validation

  • 4 real sentences (2 similar pairs):
    • "What is the capital of France?" vs "Paris is the capital of France." (similar, cosine ~0.835)
    • "What is the capital of France?" vs "The weather is sunny today." (unrelated, cosine ~0.357)
  • CPU vs NPU per-sentence cosine >=0.999991, max_diff <=0.00047 (fp32)
  • Mean CPU-NPU cosine 0.999994, max_diff 0.000467
  • Semantic check: similar pair cosine significantly higher than unrelated pair on both CPU and NPU

Validated on Ascend 910 (2x NPU, ~61 GiB free per card, torch_npu 2.9.0.post1).

Performance (Ascend 910B)

  • Batch 8, 10 iters: ~1258 sentences/s (elapsed 0.064s)
  • First latency ~6.5 ms (includes compile on first real inference)
  • Sentences/s metric distinct from tokens/s for generation tasks

Usage

from transformers import AutoTokenizer, AutoModel
import torch, torch_npu
import torch.nn.functional as F

model_dir = "./model"
tokenizer = AutoTokenizer.from_pretrained(model_dir)
model = AutoModel.from_pretrained(model_dir).to("npu:0")
model.eval()

texts = ["What is the capital of France?", "Paris is the capital of France."]
inputs = tokenizer(texts, padding=True, truncation=True, max_length=512, return_tensors="pt")
inputs = {k: v.to("npu:0") for k, v in inputs.items()}
with torch.no_grad():
    last_hidden = model(**inputs).last_hidden_state
    pooled = last_hidden[:, 0]  # CLS pooling per 1_Pooling/config.json
    embeddings = F.normalize(pooled, p=2, dim=1)
print(embeddings.cpu().numpy())
# cosine
cos = F.cosine_similarity(embeddings[0:1], embeddings[1:1]).item()

See inference.py for full synchronized example with device proofs and stable perf.

Evidence

Agent Workflow NPU Device Call Model Result

Files

  • inference.py - NPU inference with device proof and perf
  • requirements.txt - dependencies
  • assets/*.png - xterm.js evidence screenshots