Ascend-SACT/GLM-5.3-Flash-w8a8-A2
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

GLM-5.3-Flash-w8a8 带MoonCake模型适配文档

硬件类型
hardwareNPU

1. 概述

GLM-5.3-Flash-w8a8 是智谱 AI(Z.ai)GLM-5 系列首个原生多模态模型 GLM-5.3-Flash 的 W8A8 量化部署版本。

部署方式:

  • 支持1M上下文长度的4机PD分离部署模式。 本文主要介绍4机PD分离部署模式

2. 底座与硬件信息

2.1 硬件配置

项目规格
NPU 型号Ascend 910B3
NPU 数量32(单机 8 卡)
单卡 HBM64 GB
单机 HBM 总量(8 卡)512 GB
架构aarch64
操作系统Ubuntu 22.04.5 LTS (aarch64)

2.2 软件栈

组件版本
npu-smi25.5.1
CANN Toolkit9.2.0
torch2.10.0+cpu
torch_npu2.10.0.post7
vLLM0.30.0
vllm-ascend0.30.0rc2
Python3.12.13 aarch64 Linux

3. 模型与部署配置

3.1 模型信息

项目取值
模型名称GLM-5.3-w8a8
模型获取链接https://modelscope.cn/models/Eco-Tech/GLM-5.3-Flash-w8a8

模型权重下载脚本参考:

nohup modelscope download --model Eco-Tech/GLM-5.3-Flash-w8a8 --local_dir /mnt/sfs_turbo/GLM-5.3-w8a8/GLM-5.3-Flash-w8a8-1013  > down_glm53-flash-w8a8.log 2>&1 &

容器部署脚本参考:

export IMAGE=quay.io/ascend/vllm-ascend:v0.30.0rc2
export NAME=glm53-flash-030rc2

docker run  \
--name $NAME \
--net=host \
--shm-size=500g \
--privileged \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/Ascend/driver:/usr/local/Ascend/driver \
-v /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \
-v /usr/local/sbin/npu-smi:/usr/local/sbin/npu-smi \
-v /usr/local/sbin:/usr/local/sbin \
-v /etc/hccn.conf:/etc/hccn.conf:ro \
-itd $IMAGE bash

3.2 1M长上下文4机PD分离部署

MoonCake master进程启动脚本参考(在P主节点启动):

#!/usr/bin/env bash
mooncake_master \
  --port 50088 \
  --eviction_high_watermark_ratio 0.95 \
  --eviction_ratio 0.1 \
  --default_kv_lease_ttl 120s \
  --enable_offload=false \
  --client_ttl=600 \
  --alsologtostderr > start_mooncake-master.fix.log 2>&1 &

各节点的mooncake.json配置文件配置参考:

{
  "metadata_server": "P2PHANDSHAKE",
  "protocol": "ascend",
  "device_name": "",
  "master_server_address": "${Mooncake master的IP}:50088",
  "global_segment_size": "64GB",
  "local_buffer_size": "16GB",
  "preferred_segment": true,
  "prefer_alloc_in_same_node": true,
  "enable_ssd_offload": false,
  "tenant_id": "default"
}

部署方案:使用prefill:dp_size: 2,tp_size: 8,dp-size-local 1,decode: dp_size: 8,tp_size: 2,dp-size-local 4,为1P4D部署方案。

3.2.1 P+D节点统一的launch_online_dp.py文件配置:

import argparse
import multiprocessing
import subprocess


def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--template", default="./run_p.sh")
    parser.add_argument("--dp-size", type=int, required=True)
    parser.add_argument("--tp-size", type=int, default=1)
    parser.add_argument("--dp-size-local", type=int, default=-1)
    parser.add_argument("--dp-rank-start", type=int, default=0)
    parser.add_argument("--dp-address", required=True)
    parser.add_argument("--dp-rpc-port", default="12325")
    parser.add_argument("--vllm-start-port", type=int, default=8000)
    args = parser.parse_args()
    if args.dp_size_local == -1:
        args.dp_size_local = args.dp_size
    return args


def run(args, devices, port, dp_rank):
    subprocess.run(
        [
            "bash",
            args.template,
            devices,
            str(port),
            str(args.dp_size),
            str(dp_rank),
            args.dp_address,
            args.dp_rpc_port,
            str(args.tp_size),
        ],
        check=True,
    )


if __name__ == "__main__":
    args = parse_args()
    processes = []
    for i in range(args.dp_size_local):
        devices = ",".join(
            str(device)
            for device in range(i * args.tp_size, (i + 1) * args.tp_size)
        )
        process = multiprocessing.Process(
            target=run,
            args=(
                args,
                devices,
                args.vllm_start_port + i,
                args.dp_rank_start + i,
            ),
        )
        processes.append(process)
        process.start()
    for process in processes:
        process.join()

3.2.2 Prefill节点run_p.sh配置:

A2 系列节点不具备 A3 的 HCCS 网络,缓存池通过 A2 RoCE 网络运行,在配置中增加 export HCCL_INTRA_ROCE_ENABLE=1配置:

#!/usr/bin/env bash
# Usage: bash run_p.sh <visible_devices> <http_port> <dp_size> \
#   <dp_rank> <dp_address> <rpc_port> <tp_size>

: "${LOCAL_IP:?Set LOCAL_IP in the P0 or P1 wrapper}"
: "${NIC_NAME:?Set NIC_NAME in the P0 or P1 wrapper}"
: "${MODEL_PATH:?Set MODEL_PATH in the P0 or P1 wrapper}"

export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export VLLM_HOST_IP="$LOCAL_IP"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_SERVER_DEV_MODE=1
export VLLM_USE_V2_MODEL_RUNNER=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_BUFFSIZE=1024
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=3600
export HCCL_CONNECT_TIMEOUT=1200
export VLLM_RPC_TIMEOUT=3600000
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export ASCEND_RT_VISIBLE_DEVICES="$1"
export TASK_QUEUE_ENABLE=2
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_TRANSFER_TIMEOUT=120000
export VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT="${VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT:-300}"
export PYTHONHASHSEED=0
export MOONCAKE_CONFIG_PATH="${本地mooncake.json路径}/mooncake.json"

exec vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port "$2" \
  --data-parallel-size "$3" \
  --data-parallel-size-local 1 \
  --data-parallel-address "$5" \
  --data-parallel-rpc-port "$6" \
  --tensor-parallel-size "$7" \
  --enable-expert-parallel \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --seed 1024 \
  --served-model-name glm53-flash \
  --safetensors-load-strategy prefetch \
  --max-model-len 1000000 \
  --max-num-seqs 64 \
  --max-num-batched-tokens 8192 \
  --trust-remote-code \
  --quantization ascend \
  --kv-cache-dtype int8 \
  --attention_config.indexer_kv_dtype int8 \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --gpu-memory-utilization 0.9 \
  --async-scheduling \
  --enforce-eager \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp", "enforce_eager": true}' \
  ${SERVER_ROLE_ARGS:-} \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --additional-config '{"enable_flashcomm1": true,"mc2_comm_alg": "hierarchy","enable_dsa_cp": true,"enable_fused_mc2": 0, "ascend_compilation_config": {"enable_npugraph_ex": true, "enable_static_kernel": false}, "multistream_overlap_shared_expert": true, "enable_mc2_hierarchy_comm": false,"enable_sparse_sfa_c8": true, "enable_sparse_li_c8": true, "enable_cpu_binding": true, "recompute_scheduler_enable": false}' \
  --kv-transfer-config \
  '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "engine_id": "0",
    "kv_connector_extra_config": {
      "connectors": [
        {
          "kv_connector": "MooncakeConnectorV2",
          "kv_role": "kv_producer",
          "kv_port": "30000",
          "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 2, "tp_size": 8},
            "decode": {"dp_size": 8, "tp_size": 2}
          }
        },
        {
          "kv_connector": "AscendStoreConnector",
          "kv_role": "kv_producer",
          "kv_connector_extra_config": {
            "backend": "mooncake",
            "use_layerwise": false,
            "lookup_rpc_port": '"$((37000 + $4))"'
          }
        }
      ]
    }
  }'
  --prefix-cache-retention-interval None

3.2.3 decode节点run_d.sh配置:

A2 系列节点不具备 A3 的 HCCS 网络,缓存池通过 A2 RoCE 网络运行,在配置中增加 export HCCL_INTRA_ROCE_ENABLE=1配置:

#!/usr/bin/env bash
# Usage: bash run_d.sh <visible_devices> <http_port> <dp_size> \
#   <dp_rank> <dp_address> <rpc_port> <tp_size>

: "${LOCAL_IP:?Set LOCAL_IP in the D0 or D1 wrapper}"
: "${NIC_NAME:?Set NIC_NAME in the D0 or D1 wrapper}"
: "${MODEL_PATH:?Set MODEL_PATH in the D0 or D1 wrapper}"

export HCCL_IF_IP="$LOCAL_IP"
export GLOO_SOCKET_IFNAME="$NIC_NAME"
export TP_SOCKET_IFNAME="$NIC_NAME"
export HCCL_SOCKET_IFNAME="$NIC_NAME"
export VLLM_HOST_IP="$LOCAL_IP"
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True,pin_memory_expandable_segments:True
export VLLM_SERVER_DEV_MODE=1
export VLLM_USE_V2_MODEL_RUNNER=0
export HCCL_OP_EXPANSION_MODE=AIV
export HCCL_BUFFSIZE=1024
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=3000
export HCCL_EXEC_TIMEOUT=3600
export HCCL_CONNECT_TIMEOUT=1200
export VLLM_RPC_TIMEOUT=3600000
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
# Reserve disjoint HCCL port ranges per DP rank so multiple engines on
# one node never share host/NPU socket ports.
export HCCL_HOST_SOCKET_PORT_RANGE="$((60000 + $4 * 100))-$((60099 + $4 * 100))"
export HCCL_NPU_SOCKET_PORT_RANGE="$((60000 + $4 * 100))-$((60099 + $4 * 100))"
export ASCEND_RT_VISIBLE_DEVICES="$1"
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_TRANSFER_TIMEOUT=120000
export VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT="${VLLM_MOONCAKE_ABORT_REQUEST_TIMEOUT:-300}"
export PYTHONHASHSEED=0
export MOONCAKE_CONFIG_PATH="${本地mooncake.json路径}/mooncake.json"

exec vllm serve "$MODEL_PATH" \
  --host 0.0.0.0 \
  --port "$2" \
  --data-parallel-size "$3" \
  --data-parallel-rank "$4" \
  --data-parallel-address "$5" \
  --data-parallel-rpc-port "$6" \
  --tensor-parallel-size "$7" \
  --enable-expert-parallel \
  --enable-chunked-prefill \
  --enable-prefix-caching \
  --seed 1024 \
  --served-model-name glm53-flash \
  --safetensors-load-strategy prefetch \
  --max-model-len 1000000 \
  --max-num-seqs 32 \
  --max-num-batched-tokens 1024 \
  --trust-remote-code \
  --quantization ascend \
  --kv-cache-dtype int8 \
  --attention_config.indexer_kv_dtype int8 \
  --gpu-memory-utilization 0.85 \
  --skip-mm-profiling \
  --limit-mm-per-prompt '{"image":1,"video":0}' \
  --async-scheduling \
  --tool-call-parser glm47 \
  --reasoning-parser glm47 \
  --enable-auto-tool-choice \
  --compilation-config '{"cudagraph_mode": "FULL_DECODE_ONLY"}' \
  --additional-config '{"enable_flashcomm1": false, "mc2_comm_alg": "hierarchy","enable_dsa_cp": false, "ascend_compilation_config": {"enable_npugraph_ex": true, "enable_static_kernel": false}, "multistream_overlap_shared_expert": true, "enable_mc2_hierarchy_comm": false, "enable_sparse_sfa_c8": true, "enable_sparse_li_c8": true, "enable_cpu_binding": true, "recompute_scheduler_enable": true}' \
  --speculative-config '{"num_speculative_tokens": 5, "method": "deepseek_mtp", "enforce_eager": true}' \
  --kv-transfer-config \
  '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_consumer",
    "engine_id": "1",
    "kv_connector_extra_config": {
      "connectors": [
        {
          "kv_connector": "MooncakeConnectorV2",
          "kv_role": "kv_consumer",
          "kv_port": "30100",
          "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {"dp_size": 2, "tp_size": 8},
            "decode": {"dp_size": 8, "tp_size": 2}
          }
        },
        {
          "kv_connector": "AscendStoreConnector",
          "kv_role": "kv_consumer",
          "kv_connector_extra_config": {
            "backend": "mooncake",
            "use_layerwise": false,
            "consumer_is_to_load": false,
            "consumer_is_to_put": false,
            "lookup_rpc_port": '"$((37100 + $4))"'
          }
        }
      ]
    }
  }'
  --prefix-cache-retention-interval None

3.2.4 prefill0节点的传参脚本run_p0.sh:

prefill0节点需要配置"--api-server-count 1",IP,eth分别为prefill0节点的IP和网卡名,还需要正确配置权重路径:

#!/usr/bin/env bash
export LOCAL_IP="${IP}"
export NIC_NAME="${eth}"
export MODEL_PATH="${权重目录}/GLM-5.3-Flash-w8a8-1013"
export SERVER_ROLE_ARGS="--api-server-count 1"
exec bash ./run_p.sh "$@"

3.2.5 prefill1节点的传参脚本run_p1.sh:

prefill1节点需要配置"--headless --data-parallel-start-rank 1",IP,eth分别为prefill1节点的IP和网卡名,还需要正确配置权重路径:

#!/usr/bin/env bash
export LOCAL_IP="${IP}"
export NIC_NAME="${eth}"
export MODEL_PATH="${权重目录}/GLM-5.3-Flash-w8a8-1013"
export SERVER_ROLE_ARGS="--headless --data-parallel-start-rank 1"
exec bash ./run_p.sh "$@"

3.2.6 prefill0节点的启动脚本start_prefill_node.sh:

IP,eth分别为prefill0节点的IP

cd "$(dirname "$0")"
LOG_DIR=/mnt/modeltest-glm53-flash-pd-mem
mkdir -p $LOG_DIR
LAUNCHER_LOG=$LOG_DIR/node0_p_launcher.log
nohup python launch_online_dp.py \
  --template ./run_p0.sh \
  --dp-size 2 \
  --tp-size 8 \
  --dp-size-local 1 \
  --dp-rank-start 0 \
  --dp-address ${IP} \
  --dp-rpc-port 12321 \
  --vllm-start-port 9081 > $LAUNCHER_LOG 2>&1 &

3.2.7 prefill1节点的启动脚本start_prefill_node.sh:

IP,eth分别为prefill0节点的IP

cd "$(dirname "$0")"
LOG_DIR=/mnt/modeltest-glm53-flash-pd-mem
mkdir -p $LOG_DIR
LAUNCHER_LOG=$LOG_DIR/node1_p_launcher.log
nohup python launch_online_dp.py \
  --template ./run_p1.sh \
  --dp-size 2 \
  --tp-size 8 \
  --dp-size-local 1 \
  --dp-rank-start 1 \
  --dp-address ${IP} \
  --dp-rpc-port 12321 \
  --vllm-start-port 9082 > $LAUNCHER_LOG 2>&1 &

3.2.8 decode0节点的传参脚本run_d0.sh:

IP,eth分别为decode0节点的IP

#!/usr/bin/env bash
export LOCAL_IP="${IP}"
export NIC_NAME="${eth}"
export MODEL_PATH="${模型权重地址}/GLM-5.3-Flash-w8a8-1013"
exec bash ./run_d.sh "$@"

3.2.9 decode1节点的传参脚本run_d1.sh:

IP,eth分别为decode1节点的IP

#!/usr/bin/env bash
export LOCAL_IP="${IP}"
export NIC_NAME="${eth}"
export MODEL_PATH="${模型权重地址}/GLM-5.3-Flash-w8a8-1013"
exec bash ./run_d.sh "$@"

3.2.10 decode0节点的启动脚本参考start_decode_node.sh:

其中decode0_IP需要配置为decode0节点的IP:

cd "$(dirname "$0")"

LOG_DIR=/mnt/modeltest-glm53-flash-pd-mem
mkdir -p $LOG_DIR
LAUNCHER_LOG=$LOG_DIR/node2_d_launcher.log

nohup python launch_online_dp.py \
  --template ./run_d0.sh \
  --dp-size 8 \
  --tp-size 2 \
  --dp-size-local 4 \
  --dp-rank-start 0 \
  --dp-address ${decode0_IP} \
  --dp-rpc-port 12321 \
  --vllm-start-port 9900 > $LAUNCHER_LOG 2>&1 &

3.2.11 decode1节点的启动脚本参考start_decode_node.sh:

其中decode0_IP需要配置为decode0节点的IP:

cd "$(dirname "$0")"

LOG_DIR=/mnt/modeltest-glm53-flash-pd-mem
mkdir -p $LOG_DIR
LAUNCHER_LOG=$LOG_DIR/node3_d_launcher.log

nohup python launch_online_dp.py \
  --template ./run_d1.sh \
  --dp-size 8 \
  --tp-size 2 \
  --dp-size-local 4 \
  --dp-rank-start 4 \
  --dp-address ${decode0_IP} \
  --dp-rpc-port 12321 \
  --vllm-start-port 9900 > $LAUNCHER_LOG 2>&1 &

3.2.12 proxy启动脚本run_proxy.sh:

cd "$(dirname "$0")"

unset ftp_proxy
unset https_proxy
unset http_proxy

LOG_DIR=/mnt/modeltest-glm53-flash-pd-mem
mkdir -p $LOG_DIR
PROXY_LOG=$LOG_DIR/proxy_18000.log

nohup python /vllm-workspace/vllm-ascend/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py \
    --port 18000 \
    --host 0.0.0.0 \
    --prefiller-hosts \
      ${prefill0_IP} \
    --prefiller-ports \
      9081 \
    --decoder-hosts \
      ${decode0_IP} \
      ${decode0_IP} \
      ${decode0_IP} \
      ${decode0_IP} \
      ${decode1_IP} \
      ${decode1_IP} \
      ${decode1_IP} \
      ${decode1_IP} \
    --decoder-ports \
      9900 9901 9902 9903 \
      9900 9901 9902 9903 > $PROXY_LOG 2>&1 &

5. 推理验证

推理请求及报文:

curl http://{prefill0_IP}:18000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "glm53-flash",
    "messages": [{"role": "user", "content": "Who are you?"}],
    "max_tokens": 256,
    "temperature": 0
  }'