本文介绍 GLM-5.3-Flash-w8a8 模型在双 A2 节点上的 1P1D(Prefill-Decode,PD)分离部署方案。
modelscope download --model Eco-Tech/GLM-5.3-Flash-w8a8 --local_dir /tmp/Eco-Tech/GLM-5.3-Flash-w8a8
docker pull quay.io/ascend/vllm-ascend:v0.30.0rc2
分别在Prefill节点和Decode节点创建容器,脚本如下:
#!/bin/bash
IMAGES_ID=quay.io/ascend/vllm-ascend:v0.30.0rc2
NAME=glm53flashw8a8
SHM_SIZE=500g
docker run --name "${NAME}" -d --net=host --shm-size="${SHM_SIZE}" \
--privileged=true \
--device=/dev/davinci_manager \
--device=/dev/hisi_hdc \
--device=/dev/devmm_svm \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /usr/local/sbin:/usr/local/sbin \
-v /etc/hccn.conf:/etc/hccn.conf \
-v /home:/home \
-v /root:/root \
-v /tmp/Eco-Tech/GLM-5.3-Flash-w8a8: /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
"${IMAGES_ID}" tail -f /dev/nullPrefill节点脚本run_dp_template.sh如下所示,其中eth需要替换为绑定该IP地址的网卡名。
export HCCL_IF_IP=$IP
export GLOO_SOCKET_IFNAME=$eth
export TP_SOCKET_IFNAME=$eth
export HCCL_SOCKET_IFNAME=$eth
export VLLM_LOGGING_LEVEL="info"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_DETERMINISTIC=True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export HCCL_INTRA_ROCE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=0
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export VLLM_ENABLE_PER_REQUEST_METRICS=1
export ASCEND_RT_VISIBLE_DEVICES=$1
nohup vllm serve /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--seed 1024 \
--served-model-name GLM-5.3-Flash-W8A8 \
--max-model-len 131072 \
--max-num-batched-tokens 4096 \
--max-num-seqs 40 \
--trust-remote-code \
--gpu-memory-utilization 0.92 \
--quantization ascend \
--enable-auto-tool-choice \
--enable-chunked-prefill \
--enable-prefix-caching \
--enforce-eager \
--async-scheduling \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--speculative-config '{"num_speculative_tokens":3,"method":"deepseek_mtp"}' \
--additional-config '{"multistream_overlap_shared_expert":true, "ascend_compilation_config":{"fuse_qknorm_rope":false,"fuse_muls_add":true,"enable_npugraph_ex":true}}' \
--kv-transfer-config '{ "kv_connector": "MooncakeConnectorV2", "kv_role": "kv_producer", "kv_port": "30000", "engine_id":"0", "kv_connector_extra_config": {"use_ascend_direct":true, "prefill":{"dp_size":1,"tp_size":8}, "decode":{"dp_size":4,"tp_size":2}} }' \
--api-key 123456aA &Decode节点脚本:run_dp_template.sh如下所示,其中eth需要替换为绑定该IP地址的网卡名。
export HCCL_IF_IP=$IP
export GLOO_SOCKET_IFNAME=$eth
export TP_SOCKET_IFNAME=$eth
export HCCL_SOCKET_IFNAME=$eth
export VLLM_LOGGING_LEVEL="info"
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,pin_memory_expandable_segments:True
export HCCL_DETERMINISTIC=True
export HCCL_BUFFSIZE=1024
export TASK_QUEUE_ENABLE=1
export HCCL_OP_EXPANSION_MODE="AIV"
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
export HCCL_INTRA_ROCE_ENABLE=1
export CPU_AFFINITY_CONF=1
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export VLLM_ASCEND_ENABLE_FLASHCOMM1=0
export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export VLLM_ENABLE_PER_REQUEST_METRICS=1
export ASCEND_RT_VISIBLE_DEVICES=$1
nohup vllm serve /tmp/Eco-Tech/GLM-5.3-Flash-w8a8 \
--host 0.0.0.0 \
--port $2 \
--data-parallel-size $3 \
--data-parallel-rank "$4" \
--data-parallel-address $5 \
--data-parallel-rpc-port $6 \
--tensor-parallel-size $7 \
--seed 1024 \
--served-model-name GLM-5.3-Flash-W8A8 \
--max-model-len 131072 \
--max-num-batched-tokens 4096 \
--max-num-seqs 40 \
--trust-remote-code \
--gpu-memory-utilization 0.92 \
--quantization ascend \
--enable-auto-tool-choice \
--enable-expert-parallel \
--enable-chunked-prefill \
--enable-prefix-caching \
--async-scheduling \
--tool-call-parser glm47 \
--reasoning-parser glm45 \
--compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96],"cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens":3,"method":"deepseek_mtp"}' \
--additional-config '{"recompute_scheduler_enable":true,"multistream_overlap_shared_expert":true, "ascend_compilation_config":{"fuse_qknorm_rope":false,"fuse_muls_add":true,"enable_npugraph_ex":true}}' \
--kv-transfer-config '{ "kv_connector": "MooncakeConnectorV2", "kv_role": "kv_consumer", "kv_port": "30100", "engine_id":"1", "kv_connector_extra_config": {"use_ascend_direct":true, "prefill":{"dp_size":1,"tp_size":8}, "decode":{"dp_size":4,"tp_size":2}} }' \
--api-key 123456aA &针对Prefill节点和Decode节点分别准备launch_online_dp.py 脚本:
import argparse
import multiprocessing
import os
import subprocess
import sys
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--dp-size", type=int, required=True, help="Data parallel size.")
parser.add_argument("--tp-size", type=int, default=1, help="Tensor parallel size.")
parser.add_argument("--dp-size-local", type=int, default=-1, help="Local data parallel size.")
parser.add_argument("--dp-rank-start", type=int, default=0, help="Starting rank for data parallel.")
parser.add_argument("--dp-address", type=str, required=True, help="IP address for data parallel master node.")
parser.add_argument("--dp-rpc-port", type=str, default=12345, help="Port for data parallel master node.")
parser.add_argument("--vllm-start-port", type=int, default=9000, help="Starting port for the engine.")
return parser.parse_args()
args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port
def run_command(visible_devices, dp_rank, vllm_engine_port):
command = [
"bash",
"./run_dp_template.sh",
visible_devices,
str(vllm_engine_port),
str(dp_size),
str(dp_rank),
dp_address,
dp_rpc_port,
str(tp_size),
]
subprocess.run(command, check=True)
if __name__ == "__main__":
template_path = "run_dp_template.sh"
if not os.path.exists(template_path):
print(f"Template file {template_path} does not exist.")
sys.exit(1)
processes = []
num_cards = dp_size_local * tp_size
for i in range(dp_size_local):
dp_rank = dp_rank_start + i
vllm_engine_port = vllm_start_port + i
visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
process = multiprocessing.Process(target=run_command, args=(visible_devices, dp_rank, vllm_engine_port))
processes.append(process)
process.start()
for process in processes:
process.join()
启动器通过启动参数向3.1节中定义的脚本依次传入可见设备、引擎端口、全局 DP 大小、DP rank、DP 地址、DP RPC 端口和 TP 大小。
Prefill节点脚本:server.sh,其中$P0需要替换为Prefill节点的IP地址。
python launch_online_dp.py --dp-size 1 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address $P0 --dp-rpc-port 12321 --vllm-start-port 8000Decode节点脚本:server.sh,$D0需要替换为Decode节点的IP地址。
python launch_online_dp.py --dp-size 4 --tp-size 2 --dp-size-local 4 --dp-rank-start 0 --dp-address $D0 --dp-rpc-port 12321 --vllm-start-port 8000下述脚本部署在Prefill节点上,命名为proxy.sh。其中D0需要替换为Decode节点的IP地址。
export OPENAI_API_KEY=123456aA
PROXY=/vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py
nohup python "$PROXY" \
--port 8082 \
--host $P0 \
--prefiller-hosts $P0 \
--prefiller-ports 8000 \
--decoder-hosts $D0 $D0 $D0 $D0 \
--decoder-ports 8000 8001 8002 8003 \
> proxy.log 2>&1 &分别执行Prefill节点和Decode节点的server.sh脚本,然后在Prefill节点启动proxy.sh脚本。后续通过proxy配置的IP和端口访问模型服务。
curl http://0.0.0.0:8082/v1/chat/completions \
-H "Authorization: Bearer 123456aA" \
-H "Content-Type: application/json" \
-d '{
"model": "GLM-5.3-Flash-W8A8",
"max_tokens": 1024,
"stream":false,
"temperature":1.0,
"top_p":0.95,
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": "9.11和9.8谁大?"}]
}
]
}'执行结果如下:
{
"id": "chatcmpl-502a66c7-1af7-45dd-b77a-b59c4d4dd327",
"object": "chat.completion",
"created": 1791969215,
"model": "GLM-5.3-Flash-W8A8",
"choices": [
{
"index": 0,
"message": {
"role": "assistant",
"content": "**9.8 更大。**\n\n比较小数时要把位数对齐,补零后比较:\n\n- 9.11 → 9.11\n- 9.8 → 9.80\n\n比较十分位:1 < 8,所以 9.8 > 9.11。\n\n也可以用差值验证:9.8 − 9.11 = 0.69 > 0。\n\n常见误区是觉得\"11 比 8 大,所以 9.11 大\",但小数点后的数字不是按整数比的。比如 9.8 元是 9 元 8 角,而 9.11 元是 9 元 1 角 1 分,显然 9.8 元更多。\n\n(顺带一提:如果是软件版本号,如 Python 3.11 和 3.8,那 3.11 确实比 3.8 \"新\",但那是版本编号规则,不是小数比较。)",
"refusal": null,
"annotations": null,
"audio": null,
"function_call": null,
"reasoning": "The user is asking in Chinese: \"Which is bigger, 9.11 or 9.8?\"\n\nThis is a classic comparison question that often trips people up. Let me think carefully.\n\n9.11 vs 9.8\n\nComparing decimals:\n- 9.11 = 9 + 0.11 (9.110)\n- 9.8 = 9 + 0.8 (9.800)\n\n0.8 > 0.11, so 9.8 > 9.11.\n\nThe confusion arises because people sometimes think 11 > 8, so 9.11 > 9.8. But that's incorrect for decimal comparison. When comparing decimals, you compare digit by digit after aligning place values:\n- 9.11\n- 9.80\n\nTenths place: 1 vs 8. 8 > 1, so 9.8 > 9.11.\n\nAlternatively, think of it as money: 9.8 yuan = 9 yuan 8 jiao, 9.11 yuan = 9 yuan 1 jiao 1 fen. 9.8 yuan is more.\n\nOr fractions: 9.8 = 9 + 8/10, 9.11 = 9 + 11/100. 8/10 = 80/100 > 11/100.\n\nAnother way: 9.8 - 9.11 = 0.69 > 0.\n\nOne caveat: if these were version numbers (like Python 3.11 vs 3.8), 9.11 could be \"later\" than 9.8 in versioning semantics. But the question as asked is about decimal numbers, so the answer is 9.8 is bigger. I could briefly mention the version number context as a fun aside, but the main answer should be clear: 9.8 > 9.11.\n\nThe user wrote in Chinese, so I should respond in Chinese.\n\nFormat: Keep it concise. Give the answer clearly, explain why briefly (pad with zeros method), maybe mention the common misconception and possibly the version number caveat. Short response, maybe a few lines. No need for heavy markdown, but a small comparison could help. Keep it light.",
"logprobs": null
},
"finish_reason": "stop"
}
],
"service_tier": null,
"system_fingerprint": "vllm-0.30.0-tp8-ep-ef39caaf",
"usage": {
"prompt_tokens": 22,
"total_tokens": 701,
"completion_tokens": 679
}
}curl http:// 0.0.0.0:8082/v1/chat/completions \
-H "Authorization: Bearer 123456aA" \
-H "Content-Type: application/json" \
-d '{
"model": "GLM-5.3-Flash-W8A8",
"max_token": 128,
"stream": true,
"messages": [
{
"role": "user",
"content": [{"type": "text", "text": "9.11和9.8谁大?"}]
}
]
}'执行结果如下: