Qwen3-4B-FP8 Ascend NPU 适配文档

RuntimeError: fp8 quantization is currently not supported in npu

Explain the concept of derivatives in simple terms

..., and then use it to find the derivative of the function f(x) = 3x^2 + 5x - 7.
Also, provide a real-world example where derivatives are used, and explain how it relates to the function.

### Step 1: Explain Derivatives in Simple Terms
Derivatives are a way to measure how a function changes as its input changes. In simpler terms, they tell us the slope of a function at any given point. This slope represents the instantaneous rate of change of the function with respect to the variable.

weight_bf16 = weight_fp8.to(BF16) × expanded_scale_inv

# 单卡部署
vllm serve /tmp/Qwen3-4B-FP8-Ascend/model_files \
  --dtype bfloat16 \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --port 8000 \
  --trust-remote-code

# 多卡部署 (TP=4)
vllm serve /tmp/Qwen3-4B-FP8-Ascend/model_files \
  --dtype bfloat16 \
  --tensor-parallel-size 4 \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --port 8000 \
  --trust-remote-code

# 使用推理脚本
python /tmp/Qwen3-4B-FP8-Ascend/inference.py \
  --model_path /tmp/Qwen3-4B-FP8-Ascend/model_files \
  --tp 1 \
  --max_model_len 8192 \
  --prompt "Explain the concept of derivatives in simple terms" \
  --max_tokens 128

# 服务模式验证
curl -sf http://127.0.0.1:8000/v1/models

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "Qwen3-4B-FP8",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
    "max_tokens": 128
  }'

# 1. 克隆仓库
git clone https://gitcode.com/qionner/Qwen3-4B-FP8-Ascend.git
cd Qwen3-4B-FP8-Ascend

# 2. 设置环境
source scripts/setup_env.sh

# 3. 运行推理
python inference.py \
  --model_path model_files \
  --prompt "Hello, how are you?" \
  --max_tokens 128

RuntimeError: fp8 quantization is currently not supported in npu

OutOfMemoryError: CUDA out of memory

# 减小最大序列长度
--max-model-len 4096

# 减小批处理大小
--max-num-seqs 8

# 启用内存分配器优化
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

RuntimeError: EZ9999: [PID:xxx] 507903... ACL Graph capture error

# 启用 AIV 模式
export HCCL_OP_EXPANSION_MODE="AIV"

# 或使用 eager 模式
--enforce-eager

Qwen3-4B-FP8-Ascend/
├── model_files/              # 适配后的模型文件
│   ├── config.json           # 模型配置 (quantization_config 已移除)
│   ├── model-00001-bf16.safetensors  # 主权重 (7.5GB)
│   ├── model-00002-bf16.safetensors  # lm_head 权重 (742MB)
│   └── tokenizer_*           # 分词器文件
├── inference.py              # 推理脚本
├── prompts.jsonl             # 测试 prompts
├── benchmark/                # 基准测试
│   ├── precision_verify.py   # 精度验证
│   └── perf_benchmark.py     # 性能测试
├── scripts/
│   └── setup_env.sh          # 环境设置
├── docs/
│   ├── README.md             # 本文档
│   ├── 昇腾适配测评报告.md    # 完整测评报告
│   └── logs/                 # 日志目录
└── VALIDATION_REPORT.md      # 验证报告