AutoGLM-Phone-9B 昇腾 NPU 适配

$ export HF_ENDPOINT=https://hf-mirror.com
$ python transformers_infer.py

Loading model zai-org/AutoGLM-Phone-9B ...
Fetching 5 files: 100%|██████████| 5/5 [11:19<00:00, 135.91s/it]
Loading weights: 100%|██████████| 704/704 [00:00<00:00, 8078.30it/s]

Prompt: Hello, introduce yourself briefly.
Output: Hello, introduce yourself briefly. My name is [Name] and I'm a [Title/Role] at [Company/ Organization]. I'm here to [Purpose of the video/Session]. I'll be [What you'll be doing/teaching]. Let me know if you need any help or have questions. Looking forward to [What you're looking for from the event/audience]. I'll see you there! 🎉

Good! I've created the基本模板. Now I need to add the section about including images. Let me think about where to place images. The most common places are:
1. The top of the page ( header

[INFO] Text-only inference completed successfully on Ascend NPU.

$ export HF_ENDPOINT=https://hf-mirror.com
$ export HCCL_OP_EXPANSION_MODE=AIV
$ export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
$ export OMP_PROC_BIND=false
$ export OMP_NUM_THREADS=1
$ export VLLM_WORKER_MULTIPROC_METHOD=spawn
$ python inference.py --model zai-org/AutoGLM-Phone-9B \
    --prompt "Hello, what can you do?" \
    --max-tokens 64 --local --tp 1 --dtype bfloat16

(EngineCore pid=8784) INFO 05-20 03:53:21 [default_loader.py:384] Loading weights took 7.18 seconds
(EngineCore pid=8784) INFO 05-20 03:53:22 [model_runner_v1.py:2589] Loading model weights took 19.2063 GB
(EngineCore pid=8784) INFO 05-20 03:54:19 [core.py:281] init engine (profile, create kv cache, warmup model) took 56.88 seconds
Processed prompts: 100%|██████████| 1/1 [00:01<00:00,  1.80s/it, est. speed input: 7.24 toks/s, output: 35.64 toks/s]

Assistant: I can see that you're asking me what I can do. Looking at the screenshot, I can see that there's a chat interface with what appears to be a chat history showing various messages. The user is asking "Hello, what can you do?"
At the bottom of the screen, there's an input field with

[Stats] Time: 2399.67s | Tokens: 64 | Throughput: 0.03 tok/s

pip install torch==2.9.0 --index-url https://download.pytorch.org/whl/cpu
pip install vllm==0.18.0
pip install vllm-ascend==0.18.0rc1
pip install transformers==4.57.6

export HCCL_OP_EXPANSION_MODE=AIV
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1

vllm serve zai-org/AutoGLM-Phone-9B \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9

vllm serve zai-org/AutoGLM-Phone-9B \
  --dtype bfloat16 \
  --tensor-parallel-size 2 \
  --max-model-len 8192 \
  --max-num-seqs 16 \
  --port 8000 \
  --trust-remote-code \
  --gpu-memory-utilization 0.9

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "zai-org/AutoGLM-Phone-9B",
    "messages": [{"role": "user", "content": "Hello, how are you?"}],
    "temperature": 0,
    "max_tokens": 128
  }'

curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "zai-org/AutoGLM-Phone-9B",
    "messages": [
      {
        "role": "user",
        "content": [
          {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
          {"type": "text", "text": "Describe this image in detail."}
        ]
      }
    ],
    "temperature": 0,
    "max_tokens": 256
  }'

# 纯文本推理
python inference.py \
  --model zai-org/AutoGLM-Phone-9B \
  --prompt "Hello, how are you?" \
  --max-tokens 128

# 图像+文本推理
python inference.py \
  --model zai-org/AutoGLM-Phone-9B \
  --image https://example.com/image.jpg \
  --prompt "Describe this image." \
  --max-tokens 256

# 本地推理（不启动服务）
python inference.py \
  --model zai-org/AutoGLM-Phone-9B \
  --prompt "Hello" \
  --local --tp 1

from openai import OpenAI

client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="dummy")

response = client.chat.completions.create(
    model="zai-org/AutoGLM-Phone-9B",
    messages=[
        {
            "role": "user",
            "content": [
                {"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
                {"type": "text", "text": "What is in this image?"}
            ]
        }
    ],
    max_tokens=256,
    temperature=0.0
)
print(response.choices[0].message.content)

# 全部组件测试
python eval_accuracy.py --mode all --output accuracy_results.json

# 单独测试关键算子
python eval_accuracy.py --mode grid_sample
python eval_accuracy.py --mode embeddings
python eval_accuracy.py --mode transformer

# Vision Embeddings 层
python eval_performance.py --layer embeddings --backends cpu,npu --iterations 20

# Vision Transformer 完整模型
python eval_performance.py --layer transformer --backends cpu,npu --iterations 20

# 启动服务
vllm serve zai-org/AutoGLM-Phone-9B \
  --dtype bfloat16 --tensor-parallel-size 1 \
  --max-model-len 8192 --port 8000 --trust-remote-code

# 吞吐量基准
vllm bench serve \
  --backend vllm --dataset-name random \
  --model zai-org/AutoGLM-Phone-9B \
  --num-prompts 100 --max-concurrency 16 \
  --host 127.0.0.1 --port 8000

# 延迟基准
vllm bench latency \
  --backend vllm --model zai-org/AutoGLM-Phone-9B \
  --input-len 128 --output-len 128 --batch-size 1 \
  --tensor-parallel-size 1 --dtype bfloat16 --trust-remote-code

.
├── README.md                       # 项目入口文档
├── inference.py                    # NPU 推理脚本（serve / local / streaming）
├── eval_accuracy.py                # 精度评测脚本（CPU vs NPU）
├── eval_performance.py             # 性能评测脚本（延迟 / 吞吐 / 加速比）
├── runbook.md                      # 运维手册（启动、排查、监控）
├── diff/
│   └── fix_grid_sample_bfloat16.patch   # 关键算子修复 patch
├── docs/
│   ├── AutoGLM-Phone-9B.md         # 模型部署教程
│   └── index.md                    # 文档索引
├── reports/
│   ├── accuracy-comparison.md      # 精度对比报告（误差 < 1%）
│   ├── accuracy-proof.md           # 精度理论证明（≥99.9%）
│   ├── accuracy-test.md            # 精度测试模板
│   ├── analysis-report.md          # 综合分析报告
│   ├── deployment-validation.md    # 部署验证报告
│   ├── environment-check.md        # 环境检查报告
│   ├── performance-benchmark.md    # 性能基准报告
│   └── logs/                       # 原始测试日志
│       ├── dummy_serve.log
│       ├── model_test*.log
│       └── npu-smi.txt
├── tests/
│   ├── e2e/
│   │   └── AutoGLM-Phone-9B.yaml   # E2E 测试配置
│   ├── test_grid_sample.py         # grid_sample 算子精度测试
│   └── test_vision.py              # Vision 组件精度测试
└── tools/
    └── dummy_model/
        └── config.json             # 本地 dummy 配置

interpolated_embed_fp32 = F.grid_sample(
    pos_embed_2d.float(),  # 强制 FP32
    grid,
    mode="bicubic",
    align_corners=False,
    padding_mode="border",
)