GLM-5.2 采用混合专家（Mixture-of-Experts, MoE）架构，主要面向复杂系统工程和长周期智能体任务。
本文档将介绍该模型的主要验证步骤，包括支持特性、特性配置、环境准备、单节点与多节点部署、精度及性能评估。
请参考支持特性获取模型支持特性矩阵。
请参考特性指南获取特性配置方法。
GLM-5.2（BF16版本）需要2个Atlas 800 A3（128G × 8）节点或4个Atlas 800 A2（64G × 8）节点：下载模型权重。GLM-5.2-w8a8：需要1个Atlas 800 A3（128G × 8）节点或2个Atlas 800 A2（64G × 8）节点：下载模型权重。建议将模型权重下载至多节点共享目录，例如 /root/.cache/。
您可以使用我们的官方Docker镜像直接运行GLM-5。
:::::{tab-set} :sync-group: install
::::{tab-item} A3系列 :sync: A3
在每个节点上启动Docker镜像。
:substitutions:
export IMAGE=quay.io/ascend/vllm-ascend:glm5.2-a3
export NAME=vllm-ascend
# Run the container using the defined variables
# Note: If you are running bridge network with docker, please expose available ports for multiple nodes communication in advance
docker run --rm \
--name $NAME \
--net=host \
--shm-size=1g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci8 \
--device /dev/davinci9 \
--device /dev/davinci10 \
--device /dev/davinci11 \
--device /dev/davinci12 \
--device /dev/davinci13 \
--device /dev/davinci14 \
--device /dev/davinci15 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash:::: ::::{tab-item} A2 series :sync: A2
在每个节点上启动 docker 镜像。
:substitutions:
export IMAGE=quay.io/ascend/vllm-ascend:glm5.2
docker run --rm \
--name vllm-ascend \
--shm-size=1g \
--net=host \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /root/.cache:/root/.cache \
-it $IMAGE bash:::: :::::
如果您想部署多节点环境，需要在每个节点上配置环境。
glm-5.2-w8a8 可在 1 台 Atlas 800 A3（64G × 16）上部署。运行以下脚本执行在线推理。
:substitutions:
export HCCL_OP_EXPANSION_MODE="AIV"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_VERSION=0.21.0
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port 8077 \
--data-parallel-size 2 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-52 \
--max-num-seqs 48 \
--max-model-len 20480 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--gpu-memory-utilization 0.95 \
--quantization ascend \
--async-scheduling \
--additional-config '{"enable_npugraph_ex": true,"fuse_muls_add":true,"multistream_overlap_shared_expert":true}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp"}'
注意： 参数说明如下：
dp2tp8 并关闭专家并行。如果需要部署多节点环境，需根据验证多节点通信环境验证多节点通信。
:::::{tab-set} :sync-group: install
::::{tab-item} A3 系列 :sync: A3
glm-5.2-w8a8：可部署在 2 台 Atlas 800 A3（64G × 16）上。在两个节点上分别运行以下脚本。
节点 0
:substitutions:
# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"
export VLLM_VERSION=0.21.0
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ASCEND_BALANCE_SCHEDULING=0
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=400
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export ASCEND_LAUNCH_BLOCKING=0
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port 8077 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 12980 \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name glm-5 \
--max-num-seqs 48 \
--max-model-len 64000 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--gpu-memory-utilization 0.93 \
--quantization ascend \
--enable-prefix-caching \
--async-scheduling \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--additional-config '{"enable_npugraph_ex": true,"fuse_muls_add":true,"multistream_overlap_shared_expert":true}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}'节点 1
:substitutions:
# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxxx"
export VLLM_VERSION=0.21.0
export HCCL_OP_EXPANSION_MODE="AIV"
export VLLM_ASCEND_BALANCE_SCHEDULING=0
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=400
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export ASCEND_LAUNCH_BLOCKING=0
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port 8077 \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-rpc-port 12980 \
--data-parallel-address $node0_ip \
--tensor-parallel-size 16 \
--seed 1024 \
--served-model-name glm-5 \
--max-num-seqs 48 \
--max-model-len 64000 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--gpu-memory-utilization 0.93 \
--quantization ascend \
--enable-prefix-caching \
--async-scheduling \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--additional-config '{"enable_npugraph_ex": true,"fuse_muls_add":true,"multistream_overlap_shared_expert":true}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}':::: ::::{tab-item} A2 series :sync: A2
glm-5.2-w8a8：可部署在 2 台 Atlas 800 A2（64G × 32）上。节点 0
:substitutions:
# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxx"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
#export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export USE_MULTI_GROUPS_KV_CACHE=1
#export USE_MULTI_BLOCK_POOL=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200
export VLLM_VERSION=0.21.0
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--max_model_len 40000 \
--max-num-batched-tokens 4096 \
--served-model-name glm-52 \
--seed 1024 \
--gpu-memory-utilization 0.95 \
--api-server-count 1 \
--max-num-seqs 16 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--quantization ascend \
--port 7000 \
--safetensors-load-strategy 'prefetch' \
--block-size 128 \
--async-scheduling \
--additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}'节点 1
:substitutions:
# this obtained through ifconfig
# nic_name is the network interface name corresponding to local_ip of the current node
nic_name="xxx"
local_ip="xxx"
# The value of node0_ip must be consistent with the value of local_ip set in node0 (master node)
node0_ip="xxx"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
#export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export USE_MULTI_GROUPS_KV_CACHE=1
#export USE_MULTI_BLOCK_POOL=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200
export VLLM_VERSION=0.21.0
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--max_model_len 40000 \
--max-num-batched-tokens 4096 \
--served-model-name glm-52 \
--seed 1024 \
--gpu-memory-utilization 0.95 \
--api-server-count 1 \
--max-num-seqs 16 \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--quantization ascend \
--port 7000 \
--safetensors-load-strategy 'prefetch' \
--block-size 128 \
--async-scheduling \
--additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}':::: :::::
在共置（混合）部署中，prefill（预填充）和decode（解码）在同一节点上协同运行，这与下文的分离式部署形成对比。以下模板将GLM-5.2部署在4个节点上，采用DP4 TP8（每节点data-parallel-size-local=1），上下文窗口为200k，并启用MTP（num_speculative_tokens=5）。节点0承载API服务器，同时作为DP主节点；节点1至节点3以--headless模式运行。此配置中禁用前缀缓存（--no-enable-prefix-caching）。所有IP、NIC名称、端口和权重路径均为占位符。
节点0（API服务器/DP主节点）：
#!/usr/bin/bash
nic_name="<NIC_NAME>"
local_ip=$(hostname -I | awk -F " " '{print $1}')
echo "$local_ip"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200
export VLLM_VERSION=0.21.0
vllm serve <MODEL_PATH> \
--max_model_len 200000 \
--max-num-batched-tokens 4096 \
--served-model-name glm \
--seed 1024 \
--api-server-count 1 \
--gpu-memory-utilization 0.95 \
--max-num-seqs 32 \
--data-parallel-size 4 \
--data-parallel-size-local 1 \
--data-parallel-address $local_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--quantization ascend \
--port 7000 \
--safetensors-load-strategy 'prefetch' \
--block-size 128 \
--enable-chunked-prefill \
--no-enable-prefix-caching \
--async-scheduling \
--additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}'节点 1（无头模式，--data-parallel-start-rank 1）：
#!/usr/bin/bash
nic_name="<NIC_NAME>"
local_ip=$(hostname -I | awk -F " " '{print $1}')
node0_ip="<NODE0_IP>"
echo "$local_ip"
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_RPC_TIMEOUT=360000
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=200
export HCCL_CONNECT_TIMEOUT=120
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ACL_OP_INIT_MODE=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ENGINE_READY_TIMEOUT_S=1200
export VLLM_VERSION=0.21.0
vllm serve <MODEL_PATH> \
--max_model_len 200000 \
--max-num-batched-tokens 4096 \
--headless \
--served-model-name glm \
--seed 1024 \
--gpu-memory-utilization 0.95 \
--max-num-seqs 32 \
--safetensors-load-strategy 'prefetch' \
--data-parallel-size 4 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--quantization ascend \
--port 7000 \
--block-size 128 \
--enable-chunked-prefill \
--no-enable-prefix-caching \
--async-scheduling \
--additional-config '{"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp"}'节点 2 和节点 3 使用与节点 1 相同的脚本，分别将 --data-parallel-start-rank 设置为 2 和 3（且 node0_ip 指向节点 0）。
我们将展示在多节点环境下采用 1P1D（1 预填充节点 + 1 解码节点）部署 GLM-5 以获得更优性能的指南。
预填充-解码分离部署可在 4 台 Atlas 800 A3（64G × 32）服务器上进行。
开始之前，请按以下步骤操作：
在每个节点上准备脚本 launch_online_dp.py：
import argparse
import multiprocessing
import os
import subprocess
import sys
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument(
"--dp-size",
type=int,
required=True,
help="数据并行大小。"
)
parser.add_argument(
"--tp-size",
type=int,
default=1,
help="张量并行大小。"
)
parser.add_argument(
"--dp-size-local",
type=int,
default=-1,
help="本地数据并行大小。"
)
parser.add_argument(
"--dp-rank-start",
type=int,
default=0,
help="数据并行起始序号。"
)
parser.add_argument(
"--dp-address",
type=str,
required=True,
help="数据并行主节点的 IP 地址。"
)
parser.add_argument(
"--dp-rpc-port",
type=str,
default=12345,
help="数据并行主节点的端口。"
)
parser.add_argument(
"--vllm-start-port",
type=int,
default=9000,
help="引擎的起始端口。"
)
return parser.parse_args()
args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port
def run_command(visible_devices, dp_rank, vllm_engine_port):
command = [
"bash",
"./run_dp_template.sh",
visible_devices,
str(vllm_engine_port),
str(dp_size),
str(dp_rank),
dp_address,
dp_rpc_port,
str(tp_size),
]
subprocess.run(command, check=True)
if __name__ == "__main__":
template_path = "./run_dp_template.sh"
if not os.path.exists(template_path):
print(f"模板文件 {template_path} 不存在。")
sys.exit(1)
processes = []
num_cards = dp_size_local * tp_size
for i in range(dp_size_local):
dp_rank = dp_rank_start + i
vllm_engine_port = vllm_start_port + i
visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
process = multiprocessing.Process(target=run_command,
args=(visible_devices, dp_rank,
vllm_engine_port))
processes.append(process)
process.start()
for process in processes:
process.join()
在每个节点上准备脚本 run_dp_template.sh。
为了在预填充阶段支持 200k 上下文窗口，需要在每个预填充节点的 --additional_config 中添加参数 "layer_sharding": ["q_b_proj"]。
预填充节点 0
nic_name="xxxx" # 请替换为您自己的网卡名称
local_ip="xxxx" # 请替换为您自己的 IP 地址
export VLLM_VERSION=0.21.0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=256
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-52 \
--max-model-len 135000 \
--speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \
--additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "ascend_compilation_config": {"enable_npugraph_ex": true},"enable_dsa_cp": true}' \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--max-num-seqs 64 \
--async-scheduling \
--quantization ascend \
--gpu-memory-utilization 0.95 \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{"kv_connector": "MooncakeConnectorV1",
"kv_role": "kv_producer",
"kv_port": "30000",
"engine_id": "0",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}'
预填充节点 1
nic_name="xxxx" # 请替换为您自己的网卡名称
local_ip="xxxx" # 请替换为您自己的 IP 地址
export VLLM_VERSION=0.21.0
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=256
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-52 \
--max-model-len 135000 \
--speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \
--additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "ascend_compilation_config": {"enable_npugraph_ex": true},"enable_dsa_cp": true}' \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--max-num-seqs 64 \
--async-scheduling \
--quantization ascend \
--gpu-memory-utilization 0.95 \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{"kv_connector": "MooncakeConnectorV1",
"kv_role": "kv_producer",
"kv_port": "30000",
"engine_id": "0",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}'解码节点 0
nic_name="xxxx" # 请替换为您自己的网卡名称
local_ip="xxxx" # 请替换为您自己的 IP 地址
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=500
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export VLLM_VERSION=0.21.0
export TASK_QUEUE_ENABLE=1
export ASCEND_RT_VISIBLE_DEVICES=$1
export DYNAMIC_EPLB=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_MLAPO=1
vllm serve /root/.cache/modelscope/hub/models/vllm-ascend/GLM5.2-w8a8 \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-52 \
--max-model-len 135000 \
--max-num-batched-tokens 164 \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \
--additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--trust-remote-code \
--max-num-seqs 48 \
--gpu-memory-utilization 0.92 \
--async-scheduling \
--quantization ascend \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{"kv_connector": "MooncakeConnectorV1",
"kv_role": "kv_consumer",
"kv_port": "30100",
"engine_id": "1",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}'解码节点 1
nic_name="xxxx" # 请替换为您自己的网卡名称
local_ip="xxxx" # 请替换为您自己的 IP 地址
export HCCL_OP_EXPANSION_MODE="AIV"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_BUFFSIZE=500
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export ASCEND_A3_ENABLE=1
export TASK_QUEUE_ENABLE=1
export VLLM_VERSION=0.21.0
export ASCEND_RT_VISIBLE_DEVICES=$1
export DYNAMIC_EPLB=1
export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export VLLM_ASCEND_ENABLE_MLAPO=1
vllm serve /mnt/share/weight/GLM-5.2-0610-Provider-w8a8/ \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm-52 \
--max-model-len 135000 \
--max-num-batched-tokens 164 \
--speculative-config '{"num_speculative_tokens": 5, "method":"deepseek_mtp"}' \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
--additional-config '{"enable_sparse_c8":false,"fuse_muls_add": true, "multistream_overlap_shared_expert": true, "recompute_scheduler_enable": true, "ascend_compilation_config": {"enable_npugraph_ex": true}}' \
--trust-remote-code \
--max-num-seqs 48 \
--gpu-memory-utilization 0.92 \
--async-scheduling \
--quantization ascend \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{"kv_connector": "MooncakeConnectorV1",
"kv_role": "kv_consumer",
"kv_port": "30100",
"engine_id": "1",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}'准备工作完成后，可在每个节点上通过以下命令启动服务器：
预填充节点 0
# 请将 IP 替换为您自己的 IP
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 2 --dp-rank-start 0 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081预填充节点 1
# 请将 IP 替换为您自己的 IP
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 2 --dp-rank-start 2 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081解码节点 0
# 请将 IP 替换为您自己的 IP
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 4 --dp-rank-start 0 --dp-address $node_p0_ip --dp-rpc-port 16600 --vllm-start-port 9900解码节点 1
# 请将 IP 替换为您自己的 IP
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 4 --dp-rank-start 4 --dp-address $node_p0_ip --dp-rpc-port 16600 --vllm-start-port 9900在 Atlas 800 A2 服务器（每节点配备 8 张卡）上，相同的全局 P/D 拓扑结构（预填充 DP4 TP8，解码 DP8 TP4）将分布在 8 个节点上：4 个预填充节点各承载 1 个 DP 序号（每个序号 8 张卡），4 个解码节点各承载 2 个 DP 序号（每个序号 4 张卡）。可直接复用上述 launch_online_dp.py 脚本。预填充端启用 FlashComm1 和 DSA CP；解码端启用 MLAPO 和 DYNAMIC_EPLB，并使用 FULL_DECODE_ONLY 图。两端均启用前缀缓存和 MTP（num_speculative_tokens=3）。以下所有 IP、网卡名称、端口和权重路径均为占位符。
预填充节点的 run_dp_template.sh 脚本：
#!/usr/bin/bash
nic_name="<NIC_NAME>"
local_ip="<CURRENT_NODE_IP>"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_HOST_IP=$local_ip
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export HCCL_OP_EXPANSION_MODE="AIV"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=256
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export VLLM_VERSION=0.21.0
export ASCEND_RT_VISIBLE_DEVICES=$1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
vllm serve <MODEL_PATH> \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm5.2 \
--max-model-len 115168 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--max-num-seqs 64 \
--gpu-memory-utilization 0.95 \
--quantization ascend \
--async-scheduling \
--enable-chunked-prefill \
--enable-prefix-caching \
--enforce-eager \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_producer",
"kv_port": "30000",
"engine_id": "0",
"kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}' \
--additional-config \
'{
"enable_sparse_c8": false,
"fuse_muls_add": true,
"multistream_overlap_shared_expert": true,
"recompute_scheduler_enable": true,
"ascend_compilation_config": {
"enable_npugraph_ex": true
},
"enable_dsa_cp": true
}' \
--speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}'run_dp_template.sh 用于解码节点：
#!/usr/bin/bash
nic_name="<NIC_NAME>"
local_ip="<CURRENT_NODE_IP>"
export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export VLLM_HOST_IP=$local_ip
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_ASCEND_ENABLE_MLAPO=1
export HCCL_OP_EXPANSION_MODE="AIV"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=1
export HCCL_BUFFSIZE=500
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export TASK_QUEUE_ENABLE=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export VLLM_VERSION=0.21.0
export DYNAMIC_EPLB=1
export ASCEND_RT_VISIBLE_DEVICES=$1
vllm serve <MODEL_PATH> \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank $4 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--enable-expert-parallel \
--seed 1024 \
--served-model-name glm5.2 \
--max-model-len 135168 \
--max-num-batched-tokens 164 \
--trust-remote-code \
--max-num-seqs 48 \
--gpu-memory-utilization 0.92 \
--async-scheduling \
--quantization ascend \
--enable-prefix-caching \
--enable-auto-tool-choice \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--kv-transfer-config \
'{
"kv_connector": "MooncakeConnector",
"kv_role": "kv_consumer",
"kv_port": "30100",
"engine_id": "1",
"kv_connector_module_path": "vllm_ascend.distributed.kv_transfer.kv_p2p.mooncake_connector",
"kv_connector_extra_config": {
"use_ascend_direct": true,
"prefill": {
"dp_size": 4,
"tp_size": 8
},
"decode": {
"dp_size": 8,
"tp_size": 4
}
}
}' \
--compilation-config \
'{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--additional-config \
'{
"enable_sparse_c8": false,
"fuse_muls_add": true,
"multistream_overlap_shared_expert": true,
"recompute_scheduler_enable": true,
"ascend_compilation_config": {
"enable_npugraph_ex": true
}
}' \
--speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}'准备工作完成后，使用以下命令启动服务器：
预填充节点 — 在 $node_p0_ip、$node_p1_ip、$node_p2_ip、$node_p3_ip 上运行，--dp-rank-start 参数分别为 0/1/2/3：
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 1 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 2 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 3 --dp-address $node_p0_ip --dp-rpc-port 16591 --vllm-start-port 9081解码节点 — 在 $node_d0_ip、$node_d1_ip、$node_d2_ip、$node_d3_ip 上运行，--dp-rank-start 参数分别为 0/2/4/6：
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 0 --dp-address $node_d0_ip --dp-rpc-port 16600 --vllm-start-port 9900
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 2 --dp-address $node_d0_ip --dp-rpc-port 16600 --vllm-start-port 9900
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 4 --dp-address $node_d0_ip --dp-rpc-port 16600 --vllm-start-port 9900
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 6 --dp-address $node_d0_ip --dp-rpc-port 16600 --vllm-start-port 9900对于这种 8 节点 A2 布局的请求转发，在以下请求转发命令中使用 4 个预填充主机（每个 1 个端点）和 4 个解码主机（每个 2 个端点）。
要设置请求转发，请在任意机器上运行以下脚本。您可以在仓库的示例中获取代理程序：load_balance_proxy_server_example.py
unset http_proxy
unset https_proxy
python load_balance_proxy_server_example.py \
--port 8000 \
--host 0.0.0.0 \
--prefiller-hosts \
$node_p0_ip \
$node_p0_ip \
$node_p1_ip \
$node_p1_ip \
--prefiller-ports \
6700 6701 \
6700 6701 \
--decoder-hosts \
$node_d0_ip \
$node_d0_ip \
$node_d0_ip \
$node_d0_ip \
$node_d1_ip \
$node_d1_ip \
$node_d1_ip \
$node_d1_ip \
--decoder-ports \
6800 6801 6802 6803 \
6800 6801 6802 6803 \ 注意：
以下为部分优化配置说明：
VLLM_ASCEND_ENABLE_FLASHCOMM1：启用 FlashComm 优化以减少预填充节点的通信和计算开销。启用 FlashComm 后，layer_sharding 列表中不能包含 o_proj 元素。VLLM_ASCEND_ENABLE_FUSED_MC2：启用以下融合算子：dispatch_gmm_combine_decode 和 dispatch_ffn_combine 算子。有关上述环境变量的进一步解释和限制，请参考以下 Python 文件：envs.py
服务器启动后，您可以使用输入提示词对模型进行查询：
curl http://<node0_ip>:<port>/v1/completions \
-H "Content-Type: application/json" \
-d '{
"model": "glm-52",
"prompt": "The future of AI is",
"max_completion_tokens": 50,
"temperature": 0
}'以下是两种准确性评估方法。
详情请参考使用 AISBench。
执行后，您可以获取结果。
暂未测试。
详情请参考使用 AISBench 进行性能评估。
更多详情请参考vllm benchmark。
注意：
max-model-len 和 max-num-seqs 需要根据实际使用场景进行设置。其他设置请参考**部署** 章节。
问：如何为 GLM-5.2 启用函数调用？
答：请在 vLLM 启动命令中添加以下配置
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--enable-auto-tool-choice \