SmolLM-135M 是基于 Llama 架构的小型语言模型，拥有 135M 参数。该模型经过指令微调，适合在资源受限的环境中运行，支持文本生成任务。
SmolLM-135M-ascend/
├── inference.py # 推理测试脚本
├── log.txt # 测试日志
├── README.md # 本文档
└── test_prompts.txt # 测试提示词source /usr/local/Ascend/ascend-toolkit/set_env.sh模型文件位于 /data/ysws/agentsp/5-15/SmolLM-135M/ 目录下：
pip install transformers torch_npu运行推理脚本进行文本生成：
cd /data/ysws/agentsp/5-15/SmolLM-135M-ascend/
# 使用默认测试提示词
python3 inference.py
# 使用指定设备
python3 inference.py --device npu:0运行精度对比测试，验证 NPU 计算结果与 CPU 一致性：
cd /data/ysws/agentsp/5-15/SmolLM-135M-ascend/
# 运行完整精度测试
python3 inference.py --mode precision_test| 参数 | 说明 | 默认值 |
|---|---|---|
--mode | 测试模式: inference 或 precision_test | inference |
--device | 运行设备 | npu:0 (自动检测) |
| 指标 | 实测值 | 阈值 | 状态 |
|---|---|---|---|
| 相对误差 | 0.0011% | < 1.00% | PASS |
| 操作 | 耗时 |
|---|---|
| CPU 推理时间 | 0.530s |
| NPU 推理时间 | 0.420s |
输入提示词:
Once upon a time生成的文本:
Once upon a time. That is, they were not yet grown to size. At one point, their mothers were about to stop...============================================================
SmolLM-135M NPU 推理测试
============================================================
Model dir: /data/ysws/agentsp/5-15/SmolLM-135M
Output dir: /data/ysws/agentsp/5-15/SmolLM-135M-ascend
NPU available: True
NPU device count: 8
NPU 0: Ascend910B3, total_memory=61.0GB
============================================================
Precision Test: CPU vs NPU
============================================================
Loading tokenizer...
Loading model for CPU...
Loading model for NPU...
Running inference on CPU...
Running inference on NPU...
CPU inference time: 0.530s
NPU inference time: 0.420s
Max absolute error: 1.654625e-04
Max relative error: 1.104891e-05 (0.0011%)
PASS: True (threshold: 1.0%)
============================================================
PRECISION TEST RESULT
============================================================
Relative error: 1.104891e-05
CPU time: 0.530s
NPU time: 0.420s
PASS: True
============================================================
Test Complete!
============================================================import torch
from transformers import LlamaForCausalLM, LlamaTokenizer
MODEL_DIR = "/data/ysws/agentsp/5-15/SmolLM-135M"
tokenizer = LlamaTokenizer.from_pretrained(MODEL_DIR)
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
model = LlamaForCausalLM.from_pretrained(MODEL_DIR, torch_dtype=torch.float32)
model = model.to("npu:0")
model.eval()
prompt = "Once upon a time"
inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=128)
inputs = {k: v.to("npu:0") for k, v in inputs.items()}
with torch.no_grad():
outputs = model.generate(
input_ids=inputs["input_ids"],
attention_mask=inputs["attention_mask"],
max_new_tokens=50,
temperature=1.0,
top_p=0.95,
do_sample=True
)
generated_text = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(f"Generated: {generated_text}")prompt = "Hello, how are you?"
inputs = tokenizer(prompt, return_tensors="pt", padding=True, truncation=True, max_length=128)
inputs = {k: v.to("npu:0") for k, v in inputs.items()}
with torch.no_grad():
outputs = model(**inputs)
logits = outputs.logits
predictions = torch.argmax(logits, dim=-1)
print(f"Next token predictions: {predictions}")| 组件 | 说明 |
|---|---|
| embeddings | 词嵌入层 |
| layers | 30 层 Transformer 解码器 |
| norm | RMS 归一化 |
| lm_head | 语言模型头部 |
从 config.json 提取的关键参数:
{
"hidden_size": 576,
"intermediate_size": 1536,
"num_attention_heads": 9,
"num_hidden_layers": 30,
"vocab_size": 49152,
"max_position_embeddings": 2048,
"rope_theta": 10000.0
}A: 正常现象。NPU 首次运行需要进行模型编译和算子加载，后续推理会快很多。
A: 检查 NPU 驱动是否正确安装，确保 CANN 环境变量已 source。0.1-0.2% 的数值误差是正常的。
A: 可以在 generate 时调整 max_new_tokens 参数。
A: 可以调整 temperature、top_p、top_k 等采样参数。
本项目遵循相关许可证