Qwen2-0.5B · 昇腾 NPU 适配测评

import os
os.environ['VLLM_ASCEND_MODE'] = '1'

from vllm import LLM, SamplingParams

llm = LLM(
    model="Tianjin_Ascend/Qwen2-0.5B",  # 本地路径或 HF 模型名
    tensor_parallel_size=1,
    max_model_len=4096,
    trust_remote_code=True,
    dtype="bfloat16",
    gpu_memory_utilization=0.85,
)

sp = SamplingParams(temperature=0.7, max_tokens=100)
outputs = llm.generate(["Hello, how are you?"], sp)
print(outputs[0].outputs[0].text)

VLLM_ASCEND_MODE=1 vllm serve Tianjin_Ascend/Qwen2-0.5B \
    --tensor-parallel-size 1 \
    --max-model-len 4096 \
    --dtype bfloat16 \
    --trust-remote-code \
    --gpu-memory-utilization 0.85 \
    --port 8000

curl http://localhost:8000/v1/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Tianjin_Ascend/Qwen2-0.5B",
    "prompt": "The capital of France is",
    "temperature": 0,
    "max_tokens": 16
  }'

[Prompt]   The capital of France is
[Output]   Paris
[Latency]  0.130s

[Prompt]   The author of Romeo and Juliet is
[Output]   ______.
A
[Latency]  0.073s
╰─ 注: Base 模型未指令微调，填空格式输出是 Base 模型的正常行为。

[Prompt]   The chemical symbol for water is
[Output]   H2O
[Latency]  0.072s

[Prompt]   The largest planet in our solar system is
[Output]   called the ____.
A
[Latency]  0.094s

[Prompt]   If x=5 and y=3, then x + y × 2 =
[Output]   ?
Answer Choices: (A) 10 (B) 12 (C) 14 (D) 16 (E) 18
Let's solve the multi-choice question step by step.
x + y ×
[Latency]  0.541s

[Prompt]   Complete the sequence: 2, 4, 6, 8,
[Output]   10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34
[Latency]  0.515s

[Prompt]   def hello(name):
    return
[Output]   f"Hello {name}!"
[Latency]  0.084s

[Prompt]   print("Hello, world!")  # Output:
[Output]   Hello, world!
[Latency]  0.079s

[Prompt]   中国的首都是
[Output]   ____
A
[Latency]  0.099s
╰─ 注: Base 模型未指令微调，填空格式输出是 Base 模型的正常行为。

[Prompt]   人工智能的英文缩写是
[Output]   ____。
A
[Latency]  0.073s

Tianjin_Ascord/Qwen2-0.5B/
├── README.md               # 本文件 · 模型测评报告
├── ADAPT_REPORT.md          # 昇腾适配分析报告
├── RUNBOOK.md               # 运行手册
├── config.json              # 模型配置
├── generation_config.json   # 生成参数配置
├── model.safetensors        # 模型权重 (943MB, bf16)
├── tokenizer.json           # Tokenizer
├── tokenizer_config.json    # Tokenizer 配置
├── vocab.json               # 词表
├── merges.txt               # BPE 合并规则
├── collect_inference_evidence.py  # 推理输出证据采集脚本
├── cpu_npu_precision.py           # NPU vs CPU 精度对比脚本
├── benchmark.py             # 性能评测脚本 (Eager 模式)
├── benchmark_compiled.py    # 编译模式评测脚本
├── eval_quality.py          # 质量评测脚本
└── bench_results/
    ├── benchmark.json           # Eager模式性能数据
    ├── benchmark_compiled.json  # 编译模式性能数据
    ├── quality_eval.json        # 质量评测数据
    ├── inference_evidence.json  # 推理输出证据 (10条)
    └── precision_comparison.json  # CPU vs NPU 精度对比数据

VLLM_ASCEND_MODE=1 vllm serve Tianjin_Ascend/Qwen2-0.5B \
    --tensor-parallel-size 2 \    # 使用2张NPU
    --pipeline-parallel-size 1 \
    --max-model-len 4096 \
    --dtype bfloat16

export HCCL_OP_EXPANSION_MODE=AIV