Ascend-SACT/GLM-5.3-Flash-w8a8
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

GLM-5.3-Flash-w8a8 双机PD分离部署指南

1 概述

本文介绍 GLM-5.3-Flash-w8a8 模型在双 A2 节点上的 1P1D(Prefill-Decode,PD)分离部署方案。

2 准备环境

2.1 下载模型权重

modelscope download --model Eco-Tech/GLM-5.3-Flash-w8a8 --local_dir /tmp/Eco-Tech/GLM-5.3-Flash-w8a8

2.2 加载镜像

docker pull quay.io/ascend/vllm-ascend:v0.30.0rc2

2.3 创建容器

分别在Prefill节点和Decode节点创建容器,脚本如下:

#!/bin/bash
IMAGES_ID=quay.io/ascend/vllm-ascend:v0.30.0rc2
NAME=glm53flashw8a8
SHM_SIZE=500g

docker run --name "${NAME}" -d --net=host --shm-size="${SHM_SIZE}" \
    --privileged=true \
    --device=/dev/davinci_manager \
    --device=/dev/hisi_hdc \
    --device=/dev/devmm_svm \
    -v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
    -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
    -v /usr/local/sbin:/usr/local/sbin \
    -v /etc/hccn.conf:/etc/hccn.conf \
    -v /home:/home \
    -v /root:/root \
    -v /tmp/Eco-Tech/GLM-5.3-Flash-w8a8: /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
    "${IMAGES_ID}" tail -f /dev/null

3 部署实施

3.1 准备P/D端部署脚本

Prefill节点脚本run_dp_template.sh如下所示,其中IP需要替换为对应服务器的IP地址,IP需要替换为对应服务器的IP地址,IP需要替换为对应服务器的IP地址,eth需要替换为绑定该IP地址的网卡名。

export HCCL_IF_IP=$IP
export GLOO_SOCKET_IFNAME=$eth
export TP_SOCKET_IFNAME=$eth
export HCCL_SOCKET_IFNAME=$eth
export VLLM_LOGGING_LEVEL="info"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_DETERMINISTIC=True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export HCCL_INTRA_ROCE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=0
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export VLLM_ENABLE_PER_REQUEST_METRICS=1

export ASCEND_RT_VISIBLE_DEVICES=$1

nohup vllm serve /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
   --host 0.0.0.0 \
   --port $2 \
   --data-parallel-size $3 \
   --data-parallel-address $5 \
   --data-parallel-rpc-port $6 \
   --tensor-parallel-size $7 \
   --seed 1024 \
   --served-model-name GLM-5.3-Flash-W8A8 \
   --max-model-len 131072 \
   --max-num-batched-tokens 4096 \
   --max-num-seqs 40 \
   --trust-remote-code \
   --gpu-memory-utilization 0.92 \
   --quantization ascend \
   --enable-auto-tool-choice \
   --enable-chunked-prefill \
   --enable-prefix-caching \
   --enforce-eager \
   --async-scheduling \
   --tool-call-parser glm47 \
   --reasoning-parser glm45 \
   --speculative-config '{"num_speculative_tokens":3,"method":"deepseek_mtp"}' \
   --additional-config '{"multistream_overlap_shared_expert":true, "ascend_compilation_config":{"fuse_qknorm_rope":false,"fuse_muls_add":true,"enable_npugraph_ex":true}}' \
   --kv-transfer-config '{ "kv_connector": "MooncakeConnectorV2", "kv_role": "kv_producer", "kv_port": "30000", "engine_id":"0", "kv_connector_extra_config": {"use_ascend_direct":true, "prefill":{"dp_size":1,"tp_size":8}, "decode":{"dp_size":4,"tp_size":2}} }' \
   --api-key 123456aA &

Decode节点脚本:run_dp_template.sh如下所示,其中IP需要替换为对应服务器的IP地址,IP需要替换为对应服务器的IP地址,IP需要替换为对应服务器的IP地址,eth需要替换为绑定该IP地址的网卡名。

export HCCL_IF_IP=$IP
export GLOO_SOCKET_IFNAME=$eth
export TP_SOCKET_IFNAME=$eth
export HCCL_SOCKET_IFNAME=$eth
export VLLM_LOGGING_LEVEL="info"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,pin_memory_expandable_segments:True
export HCCL_DETERMINISTIC=True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export HCCL_INTRA_ROCE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=0
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export VLLM_ENABLE_PER_REQUEST_METRICS=1

export ASCEND_RT_VISIBLE_DEVICES=$1

nohup vllm serve /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
   --host 0.0.0.0 \
   --port $2 \
   --data-parallel-size $3 \
   --data-parallel-rank "$4" \
   --data-parallel-address $5 \
   --data-parallel-rpc-port $6 \
   --tensor-parallel-size $7 \
   --seed 1024 \
   --served-model-name GLM-5.3-Flash-W8A8 \
   --max-model-len 131072 \
   --max-num-batched-tokens 4096 \
   --max-num-seqs 40 \
   --trust-remote-code \
   --gpu-memory-utilization 0.92 \
   --quantization ascend \
   --enable-auto-tool-choice \
   --enable-expert-parallel \
   --enable-chunked-prefill \
   --enable-prefix-caching \
   --async-scheduling \
   --tool-call-parser glm47 \
   --reasoning-parser glm45 \
   --compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96],"cudagraph_mode": "FULL_DECODE_ONLY"}' \
   --speculative-config '{"num_speculative_tokens":3,"method":"deepseek_mtp"}' \
   --additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert":true, "ascend_compilation_config":{"fuse_qknorm_rope":false,"fuse_muls_add":true,"enable_npugraph_ex":true}}' \
   --kv-transfer-config '{ "kv_connector": "MooncakeConnectorV2", "kv_role": "kv_consumer", "kv_port": "30100", "engine_id":"1", "kv_connector_extra_config": {"use_ascend_direct":true, "prefill":{"dp_size":1,"tp_size":8}, "decode":{"dp_size":4,"tp_size":2}} }' \
   --api-key 123456aA &

3.2 准备DP启动器脚本

针对Prefill节点和Decode节点分别准备launch_online_dp.py 脚本:

import argparse
import multiprocessing
import os
import subprocess
import sys


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--dp-size", type=int, required=True, help="Data parallel size.")
    parser.add_argument("--tp-size", type=int, default=1, help="Tensor parallel size.")
    parser.add_argument("--dp-size-local", type=int, default=-1, help="Local data parallel size.")
    parser.add_argument("--dp-rank-start", type=int, default=0, help="Starting rank for data parallel.")
    parser.add_argument("--dp-address", type=str, required=True, help="IP address for data parallel master node.")
    parser.add_argument("--dp-rpc-port", type=str, default=12345, help="Port for data parallel master node.")
    parser.add_argument("--vllm-start-port", type=int, default=9000, help="Starting port for the engine.")
    return parser.parse_args()


args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
    dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port


def run_command(visible_devices, dp_rank, vllm_engine_port):
    command = [
        "bash",
        "./run_dp_template.sh",
        visible_devices,
        str(vllm_engine_port),
        str(dp_size),
        str(dp_rank),
        dp_address,
        dp_rpc_port,
        str(tp_size),
    ]
    subprocess.run(command, check=True)


if __name__ == "__main__":
    template_path = "run_dp_template.sh"
    if not os.path.exists(template_path):
        print(f"Template file {template_path} does not exist.")
        sys.exit(1)

    processes = []
    num_cards = dp_size_local * tp_size
    for i in range(dp_size_local):
        dp_rank = dp_rank_start + i
        vllm_engine_port = vllm_start_port + i
        visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
        process = multiprocessing.Process(target=run_command, args=(visible_devices, dp_rank, vllm_engine_port))
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

启动器通过启动参数向3.1节中定义的脚本依次传入可见设备、引擎端口、全局 DP 大小、DP rank、DP 地址、DP RPC 端口和 TP 大小。

3.3 准备服务启动脚本

Prefill节点脚本:server.sh,其中$P0需要替换为Prefill节点的IP地址。

python launch_online_dp.py --dp-size 1 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address $P0 --dp-rpc-port 12321 --vllm-start-port 8000

Decode节点脚本:server.sh,$D0需要替换为Decode节点的IP地址。

python launch_online_dp.py --dp-size 4 --tp-size 2 --dp-size-local 4 --dp-rank-start 0 --dp-address $D0 --dp-rpc-port 12321 --vllm-start-port 8000

3.4 准备PD代理脚本

下述脚本部署在Prefill节点上,命名为proxy.sh。其中P0需要替换为Prefill节点的IP地址,P0需要替换为Prefill节点的IP地址,P0需要替换为Prefill节点的IP地址,D0需要替换为Decode节点的IP地址。

export OPENAI_API_KEY=123456aA

PROXY=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py
nohup python "$PROXY" \
 --port 8082 \
 --host $P0 \
 --prefiller-hosts $P0 \
 --prefiller-ports 8000 \
 --decoder-hosts $D0 $D0 $D0 $D0 \
 --decoder-ports 8000 8001 8002 8003 \
 > proxy.log 2>&1 &

3.5 启动服务

分别执行Prefill节点和Decode节点的server.sh脚本,然后在Prefill节点启动proxy.sh脚本。后续通过proxy配置的IP和端口访问模型服务。

4 推理验证

4.1 示例一

curl http://0.0.0.0:8082/v1/chat/completions \
-H "Authorization: Bearer 123456aA" \
-H "Content-Type: application/json" \
-d '{
  "model": "GLM-5.3-Flash-W8A8",
  "max_tokens": 1024,
  "stream":false,
  "temperature":1.0,
   "top_p":0.95,
  "messages": [
    {
      "role": "user",
      "content": [{"type": "text", "text": "9.11和9.8谁大?"}]
    }
  ]
}'

执行结果如下:

{
  "id": "chatcmpl-502a66c7-1af7-45dd-b77a-b59c4d4dd327",
  "object": "chat.completion",
  "created": 1791969215,
  "model": "GLM-5.3-Flash-W8A8",
  "choices": [
    {
      "index": 0,
      "message": {
        "role": "assistant",
        "content": "**9.8 更大。**\n\n比较小数时要把位数对齐,补零后比较:\n\n- 9.11 → 9.11\n- 9.8 → 9.80\n\n比较十分位:1 < 8,所以 9.8 > 9.11。\n\n也可以用差值验证:9.8 − 9.11 = 0.69 > 0。\n\n常见误区是觉得\"11 比 8 大,所以 9.11 大\",但小数点后的数字不是按整数比的。比如 9.8 元是 9 元 8 角,而 9.11 元是 9 元 1 角 1 分,显然 9.8 元更多。\n\n(顺带一提:如果是软件版本号,如 Python 3.11 和 3.8,那 3.11 确实比 3.8 \"新\",但那是版本编号规则,不是小数比较。)",
        "refusal": null,
        "annotations": null,
        "audio": null,
        "function_call": null,
        "reasoning": "The user is asking in Chinese: \"Which is bigger, 9.11 or 9.8?\"\n\nThis is a classic comparison question that often trips people up. Let me think carefully.\n\n9.11 vs 9.8\n\nComparing decimals:\n- 9.11 = 9 + 0.11 (9.110)\n- 9.8 = 9 + 0.8 (9.800)\n\n0.8 > 0.11, so 9.8 > 9.11.\n\nThe confusion arises because people sometimes think 11 > 8, so 9.11 > 9.8. But that's incorrect for decimal comparison. When comparing decimals, you compare digit by digit after aligning place values:\n- 9.11\n- 9.80\n\nTenths place: 1 vs 8. 8 > 1, so 9.8 > 9.11.\n\nAlternatively, think of it as money: 9.8 yuan = 9 yuan 8 jiao, 9.11 yuan = 9 yuan 1 jiao 1 fen. 9.8 yuan is more.\n\nOr fractions: 9.8 = 9 + 8/10, 9.11 = 9 + 11/100. 8/10 = 80/100 > 11/100.\n\nAnother way: 9.8 - 9.11 = 0.69 > 0.\n\nOne caveat: if these were version numbers (like Python 3.11 vs 3.8), 9.11 could be \"later\" than 9.8 in versioning semantics. But the question as asked is about decimal numbers, so the answer is 9.8 is bigger. I could briefly mention the version number context as a fun aside, but the main answer should be clear: 9.8 > 9.11.\n\nThe user wrote in Chinese, so I should respond in Chinese.\n\nFormat: Keep it concise. Give the answer clearly, explain why briefly (pad with zeros method), maybe mention the common misconception and possibly the version number caveat. Short response, maybe a few lines. No need for heavy markdown, but a small comparison could help. Keep it light.",
        "logprobs": null
      },
      "finish_reason": "stop"
    }
  ],
  "service_tier": null,
  "system_fingerprint": "vllm-0.30.0-tp8-ep-ef39caaf",
  "usage": {
    "prompt_tokens": 22,
    "total_tokens": 701,
    "completion_tokens": 679
  }
}

4.2 示例二

curl  http:// 0.0.0.0:8082/v1/chat/completions \
-H "Authorization: Bearer 123456aA" \
-H "Content-Type: application/json" \
-d '{
"model": "GLM-5.3-Flash-W8A8",
"max_token": 128,
"stream": true,
    "messages": [
        {
            "role": "user",
            "content": [{"type": "text", "text": "9.11和9.8谁大?"}]
        }
    ]
}'

执行结果如下:

stream_out.txt