Ascend-SACT/GLM-5-W8A8-Rot
模型介绍文件和版本Pull Requests讨论分析
下载使用量0

同样适用于GLM-5.1

环境介绍

【硬件配置】Atlas 800T A2（优化措施同样适用于A3，可参考环境部署指导进行环境配置）
【NPU驱动固件】25.2.0
【CANN软件】CANN 8.5
【推理框架】vLLM 0.17.0.rc1 & 0.18.0.rc1

transformers                      5.2.0
huggingface_hub				1.5.0

部署指南

1、下载权重：

git lfs install
git clone https://modelers.cn/Eco-Tech/GLM-5-w8a8.git

2、加载镜像：

docker pull quay.io/ascend/vllm-ascend:v0.17.0rc1       # checked #7139 PR merged in v0.17.0rc1（https://github.com/vllm-project/vllm-ascend/pull/7139）

3、创建容器：

（1）创建容器脚本：Benchmark-docker_run.sh

# Update the vllm-ascend image
IMAGE=$1
NAME=$2

# Run the container using the defined variables
# Note: If you are running bridge network with docker, please expose available ports for multiple nodes communication in advance
docker run -itd -u 0  --ipc=host  --privileged \
-e VLLM_USE_MODELSCOPE=True -e PYTORCH_NPU_ALLOC_CONF=max_split_size_mb:256 \
--name $NAME \
--net=host \
--shm-size=256g \
--device /dev/davinci0 \
--device /dev/davinci1 \
--device /dev/davinci2 \
--device /dev/davinci3 \
--device /dev/davinci4 \
--device /dev/davinci5 \
--device /dev/davinci6 \
--device /dev/davinci7 \
--device /dev/davinci_manager \
--device /dev/devmm_svm \
--device /dev/hisi_hdc \
-v /usr/local/dcmi:/usr/local/dcmi \
-v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \
-v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
-v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
-v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
-v /etc/ascend_install.info:/etc/ascend_install.info \
-v /etc/hccn.conf:/etc/hccn.conf \
-v /root/.cache:/root/.cache \
-v /home/:/home/ \
-v /opt/data/:/opt/data/ \
-p 8000:8000 \
-it $IMAGE bash

（2）创建容器 执行如下指令

bash Benchmark-docker_run.sh IMAGE_ID glm-5_vllm-ascend
docker exec -it glm-5_vllm-ascend /bin/bash

# check更新transformers与huggingface_hub版本
pip install transformers==5.2.0 --no-deps --force-reinstall
pip install huggingface_hub==1.5.0 --no-deps --force-reinstall

4、A2双机PD混部部署指导：

（1）部署脚本：
--Node0：Benchmark-glm5-infer-node0.sh

nic_name="xxx"
local_ip="xx.xx.xx.xx"

export VLLM_ASCEND_ENABLE_MLAPO=1 # 开启MLAPO
export VLLM_ASCEND_ENABLE_NZ=1	  # 开启量化

export HCCL_OP_EXPANSION_MODE="AIV"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export HCCL_INTRA_ROCE_ENABLE=1

# optim
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

vllm serve /data/liaobiting/models/umiiiiii/GLM-W8A8/ \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--quantization ascend \
--seed 1024 \
--served-model-name glm-5 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 46080 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--gpu-memory-utilization 0.95 \
--enable-chunked-prefill \
--enable-prefix-caching \
--async-scheduling \
--additional-config '{"multistream_overlap_shared_expert":true, "fuse_qknorm_rope": false, "fuse_muls_add": true, "enable_npugraph_ex": true}' \
--compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96], "cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp"}' \
--profiler-config '{"profiler": "torch", "torch_profiler_dir": "/data/liaobiting/glm5/w8a8_pfm/profiling/0309/", "torch_profiler_with_stack": false}'
>output.log 2>&1 &

--节点1：Benchmark-glm5-infer-node1.sh

nic_name="xxx"
local_ip="xx.xx.xx.xx"
node0_ip="xx.xx.xx.xx"

export VLLM_ASCEND_ENABLE_MLAPO=1 # 开启MLAPO
export VLLM_ASCEND_ENABLE_NZ=1	  # 开启量化

export HCCL_OP_EXPANSION_MODE="AIV"

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export VLLM_ASCEND_BALANCE_SCHEDULING=1
export HCCL_INTRA_ROCE_ENABLE=1

# optim
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

vllm serve /data/liaobiting/models/umiiiiii/GLM-W8A8/ \
--host 0.0.0.0 \
--port 8000 \
--headless \
--data-parallel-size 2 \
--data-parallel-size-local 1 \
--data-parallel-start-rank 1 \
--data-parallel-address $node0_ip \
--data-parallel-rpc-port 13389 \
--tensor-parallel-size 8 \
--quantization ascend \
--seed 1024 \
--served-model-name glm-5 \
--enable-expert-parallel \
--max-num-seqs 16 \
--max-model-len 46080 \
--max-num-batched-tokens 4096 \
--trust-remote-code \
--gpu-memory-utilization 0.95 \
--enable-chunked-prefill \
--enable-prefix-caching \
--async-scheduling \
--additional-config '{"multistream_overlap_shared_expert":true, "fuse_qknorm_rope": false, "fuse_muls_add": true, "enable_npugraph_ex": true}' \
--compilation-config '{"cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96], "cudagraph_mode": "FULL_DECODE_ONLY"}' \
--speculative-config '{"num_speculative_tokens": 3, "method": "deepseek_mtp"}' \
--profiler-config '{"profiler": "torch", "torch_profiler_dir": "/data/liaobiting/glm5/w8a8_pfm/profiling/0309/", "torch_profiler_with_stack": false}'
>output.log 2>&1 &

（2）执行脚本
注：参考 https://vllm-ascend.readthedocs.io/zh-cn/latest/tutorials/multi_node_kimi.html 中的 referring to multi_node.md 完成双机环境检查

#节点0，参考注释与环境情况适配修改
bash Benchmark-glm5-infer-node0.sh
tail -f output.log
#节点1，参考注释与环境情况适配修改
bash Benchmark-glm5-infer-node1.sh
tail -f output.log

5、A2 8机PD分离部署指导：

【部署方案】P: DP4 TP8 D: DP8 TP4 （1）部署脚本：
--P实例：run_dp_template.sh

rm -rf ~/ascend
#export VLLM_VERSION=0.16.1

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
#export PYTHONPATH=/workspace/vllm-ascend_3_5/vllm-ascend:${PYTHONPATH}

export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1

export HCCL_OP_EXPANSION_MODE="AIV"

nic_name="enp67s0f0np0"
local_ip=71.10.29.138

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

#Mooncake
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
#export ASCEND_CONNECT_TIMEOUT=300000
#export ASCEND_TRANSFER_TIMEOUT=300000
#export ASCEND_BUFFER_POOL=4:8
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True

#export VLLM_ASCEND_BALANCE_SCHEDULING=1

# optim
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export HCCL_INTRA_ROCE_ENABLE=1

export ASCEND_RT_VISIBLE_DEVICES=$1

vllm serve /opt/data/weights/GLM-W8A8 \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 135168 \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --max-num-seqs 48 \
    --gpu-memory-utilization 0.95 \
    --quantization ascend \
    --async-scheduling \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_producer",
        "kv_port": "30000",
        "engine_id": "0",
        "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
        "kv_connector_extra_config": {
            "use_ascend_direct": true,
            "prefill": {
                "dp_size": 4,
                "tp_size": 8
            },
            "decode": {
                "dp_size": 8,
                "tp_size": 4
            }
        }
    }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true
    }' \
    --speculative-config '{"num_speculative_tokens": 3, "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/home/z00500064/glm5-base/profiling",
        "torch_profiler_with_stack": false
    }' \
    2>&1 | tee glm.log

--D实例：run_dp_template.sh

rm -rf ~/ascend
#export VLLM_VERSION=0.16.1

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
#export PYTHONPATH=/workspace/vllm-ascend_3_5/vllm-ascend:${PYTHONPATH}

export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1

export HCCL_OP_EXPANSION_MODE="AIV"

nic_name="enp67s0f0np0"
local_ip=71.10.29.124

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

#Mooncake
export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
#export ASCEND_CONNECT_TIMEOUT=300000
#export ASCEND_TRANSFER_TIMEOUT=300000
#export ASCEND_BUFFER_POOL=4:8
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH

export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200

export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
#export VLLM_ASCEND_BALANCE_SCHEDULING=1

# optim
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

#export VLLM_ASCEND_ENABLE_FUSED_MC2=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export HCCL_INTRA_ROCE_ENABLE=1

export ASCEND_RT_VISIBLE_DEVICES=$1


vllm serve /opt/data/verification/models/GLM-W8A8 \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 135168 \
    --max-num-batched-tokens 64 \
    --trust-remote-code \
    --max-num-seqs 48 \
    --gpu-memory-utilization 0.95 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
        "kv_connector": "MooncakeConnectorV1",
        "kv_role": "kv_consumer",
        "kv_port": "30100",
        "engine_id": "1",
        "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
        "kv_connector_extra_config": {
                "use_ascend_direct": true,
                "prefill": {
                        "dp_size": 4,
                        "tp_size": 8
                },
                "decode": {
                        "dp_size": 8,
                        "tp_size": 4
                }
        }
    }' \
    --compilation-config \
    '{
        "cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96],
        "cudagraph_mode": "FULL_DECODE_ONLY"
     }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true
    }' \
    --speculative-config '{"num_speculative_tokens": 3,  "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/home/z00500064/glm5-base/profiling",
        "torch_profiler_with_stack": false
     }' \
    2>&1 | tee glm.log

（2）launch_dp 注：参考 https://vllm-ascend.readthedocs.io/zh-cn/latest/tutorials/multi_node_kimi.html 中的 referring to multi_node.md 完成双机环境检查
P0P3、D0D3 依次执行如下指令，修改环境IP端口号等 launch_online_dp.py脚本见最后附录

# P实例
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 0 --dp-address 71.10.29.138 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 1 --dp-address 71.10.29.138 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 2 --dp-address 71.10.29.138 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 4 --tp-size 8 --dp-size-local 1 --dp-rank-start 3 --dp-address 71.10.29.138 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
# D实例
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 0 --dp-address 71.10.29.124 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 2 --dp-address 71.10.29.124 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 4 --dp-address 71.10.29.124 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &
python launch_online_dp.py --dp-size 8 --tp-size 4 --dp-size-local 2 --dp-rank-start 6 --dp-address 71.10.29.124 --dp-rpc-port 10521 --vllm-start-port 6600 >server.log 2>&1 &

（3）任一环境启动代理 load_balance_proxy_server_example.py脚本见最后附录 proxy.sh：注意修改适配IP与端口号

unset http_proxy
unset https_proxy

python load_balance_proxy_server_example.py \
    --port 8000 \
    --host 0.0.0.0 \
    --prefiller-hosts \
       71.10.29.138 \
       71.10.29.141 \
       71.10.29.128 \
       71.10.29.125 \
    --prefiller-ports \
       6600 \
       6600 \
       6600 \
       6600 \
    --decoder-hosts \
       71.10.29.124 \
       71.10.29.124 \
       71.10.29.142 \
       71.10.29.142 \
       71.10.29.139 \
       71.10.29.139 \
       71.10.29.123 \
       71.10.29.123 \
    --decoder-ports \
       6600 6601 \
       6600 6601 \
       6600 6601 \
       6600 6601 \
    >proxy.log 2>&1 &

6、A2 8机PD分离 叠加kvstore部署指导：

【环境】 1、建议使用0.18.0rc1版本：docker pull quay.io/ascend/vllm-ascend:v0.18.0rc1 2、vllm已知问题请叠加PR#34079：https://github.com/vllm-project/vllm/pull/34079

配置环境变量 MOONCAKE_CONFIG_PATH 为mooncake.json所在路径的完整路径

export MOONCAKE_CONFIG_PATH="/xxxxx/mooncake.json"

mooncake.json

{
    "metadata_server": "P2PHANDSHAKE",
    "protocol": "ascend",
    "device_name": "",
    "master_server_address": "xx.xx.xx.xx:50088",
    "global_segment_size": "1GB",
    "default_kv_lease_ttl": 11000
}

【注】
（1）"global_segment_size"为每张卡分配的内存大小，可通过free -h查看当前环境剩余内存，然后按需合理分配，注意是每张卡
（2）"default_kv_lease_ttl"约束：default_kv_lease_ttl＞传输时间ASCEND_TRANSFER_TIMEOUT> RDMA_TIMEOUT*重传次数
（3）所有Node的json配置一致，"master_server_address"指向masterIP

cd /vllm-workspace/Mooncake
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages/mooncake:$LD_LIBRARY_PATH
mooncake_master --port 50088 --eviction_high_watermark_ratio 0.9 --eviction_ratio 0.1

【注】 default_kv_lease_ttl＞传输时间ASCEND_TRANSFER_TIMEOUT> RDMA_TIMEOUT*重传次数

P实例部署脚本

# P实例
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export HCCL_OP_EXPANSION_MODE="AIV"

#debug
#export ASCEND_GLOBAL_LOG_LEVEL=3
#export ASCEND_HOST_LOG_FILE_NUM=100

nic_name="bond0"
local_ip=$(ifconfig $nic_name 2>/dev/null | grep 'inet ' | awk '{print $2}' | head -n 1)

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10

export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
#export VLLM_ASCEND_BALANCE_SCHEDULING=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
#export ASCEND_AGGREGATE_ENABLE=1
#export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1

#kvstore
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONHASHSEED=0 
#export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export MOONCAKE_CONFIG_PATH="/data/yxliu64/kvstore_script/mooncake.json"
#export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export ACL_OP_INIT_MODE=1
#A3
#export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
export HCCL_INTRA_ROCE_ENABLE=1
# Unit: ms. The timeout for one-sided communication connection establishment is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039). Users can adjust this value based on their specific setup.
# The recommended formula is: ASCEND_CONNECT_TIMEOUT = connection_time_per_card (typically within 500ms) × total_number_of_Decode_cards.
# This ensures that even in the worst-case scenario—where all Decode cards simultaneously attempt to connect to the same Prefill card the connection will not time out.
export ASCEND_CONNECT_TIMEOUT=10000
# Unit: ms. The timeout for one-sided communication transfer is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039).
export ASCEND_TRANSFER_TIMEOUT=10000
export ASCEND_BUFFER_POOL=4:8
export HCCL_RDMA_TIMEOUT=16

#debug
#echo "/tmp/core.%u.%e.%p" > /proc/sys/kernel/core_pattern
#ulimit -c unlimited

vllm serve /data/shiqiu/Eco-Tech/GLM-5-w8a8/ \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --no-enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 73728 \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --max-num-seqs 512 \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --async-scheduling \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
        "connectors": [
            {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_producer",
                "kv_port": "30000",
                "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {
                        "dp_size": 4,
                        "tp_size": 8
                    },
                    "decode": {
                        "dp_size": 8,
                        "tp_size": 4
                    }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_producer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "mooncake"
                }
            }
        ]
    }
    }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/data/yxliu64/profiling",
        "torch_profiler_with_stack": false
    }' \
    2>&1 | tee glm.log

【注】
（1）环境变量：参数取值约束 default_kv_lease_ttl＞传输时间ASCEND_TRANSFER_TIMEOUT> RDMA_TIMEOUT*重传次数（默认7次）
（2）环境变量：HCCL_RDMA_TIMEOUT=16对应0.268s
（3）配置：原"kv_connector": "MooncakeConnectorV1"中取消"engine_id"配置，在"MultiConnector"下无法解析
（4）配置："lookup_rpc_port"实例间差异配置，实例内多个节点配置相同，如1P1D，P的4个Node配置“0”，D的4个Node配置“1”

D实例部署脚本

# D实例
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export HCCL_OP_EXPANSION_MODE="AIV"

#debug
#export ASCEND_GLOBAL_LOG_LEVEL=3
#export ASCEND_HOST_LOG_FILE_NUM=100

nic_name="bond0"
local_ip=$(ifconfig $nic_name 2>/dev/null | grep 'inet ' | awk '{print $2}' | head -n 1)

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
#export VLLM_ASCEND_BALANCE_SCHEDULING=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1

#kvstore
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
#export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export PYTHONHASHSEED=0 
export MOONCAKE_CONFIG_PATH="/data/yxliu64/kvstore_script/mooncake.json"
#export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
export ACL_OP_INIT_MODE=1
#A3
#export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export ASCEND_BUFFER_POOL=4:8
export HCCL_RDMA_TIMEOUT=16

#debug
#echo "/tmp/core.%u.%e.%p" > /proc/sys/kernel/core_pattern
#ulimit -c unlimited

vllm serve /data/shiqiu/Eco-Tech/GLM-5-w8a8/ \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --no-enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 73728 \
    --max-num-batched-token 512  \
    --trust-remote-code \
    --max-num-seqs 512 \
    --gpu-memory-utilization 0.92 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_consumer",
    "kv_connector_extra_config": {
        "connectors": [
        {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_consumer",
                "kv_port": "30100",
                "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
                "kv_connector_extra_config": {
                        "use_ascend_direct": true,
                        "prefill": {
                                "dp_size": 4,
                                "tp_size": 8
                        },
                        "decode": {
                                "dp_size": 8,
                                "tp_size": 4
                        }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_consumer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"1",
                    "backend": "mooncake"
                }
            }
        ]
    }
    }' \
    --compilation-config \
    '{
        "cudagraph_capture_sizes": [1,4,8,12,16,20,24,28,32,36,40,48,56,64,80,96],
        "cudagraph_mode": "FULL_DECODE_ONLY"
     }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/data/yxliu64/profiling",
        "torch_profiler_with_stack": false
     }' \
    2>&1 | tee glm.log

其他脚本同第5章
【预热】
每个卡之间都需要进行建链，以A2 8机64卡为例： （1）理论最大建链数 = 6463 -- 64张卡分别需要与另外63张卡进行建链 （2）每个node理论最大建链数 = 863 = 504 （3）warmup阶段建议至少完成90%左右的建链，以提升实际业务运行过程中的稳定性，每条建链对应一条日志“Connected to segment: 172.29.226.170:20585, cost:394435 us”，故可以统计每个节点该日志的数量：504*90%=450个条

grep -c "segment" /path/to/your.log

（4）建议预热场景：4k输入、1输出、4并发、1024条请求（工具：https://wiki.huawei.com/domains/143914/wiki/137614/WIKI202511048896614 为例）

python3 aisbench_test.py --input_len 4096 --output_len 4 --data_num 1024 --concurrency 4 --request_rate 4 --dataset_type prefix_cache --repeat_rate 0.8 --seed 1024

7、PD分离+kvstore+静态kernel：

【使能】 1、代码适配：临时修改 vllm_ascend/ops/fused_moe/fused_moe.py -- 457行 -- 将 (len(forward_context.all_moe_layers)) 替换为76（78-3+1）（层数、非moe层、mtp层） 2、增量配置：

export LOCAL_WORLD_SIZE=8

--additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "ascend_compilation_config": {"enable_npugraph_ex": true,"enable_static_kernel": true} 
    }'

--compilation-config \
    '{
        "cudagraph_capture_sizes": [4,16,64,96,128,160],
        "cudagraph_mode": "FULL_DECODE_ONLY",
        "max_cudagraph_capture_size": 160
     }' \

【注】 1、MTP+大EP可能存在图形捕获缓冲区溢出问题，建议补充"max_cudagraph_capture_size"配置 2、"enable_npugraph_ex"理论上应配置在"ascend_compilation_config"中，前期验证时直接放在--additional-config层级下，暂无影响，有待补充验证

P实例

# P实例
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export HCCL_OP_EXPANSION_MODE="AIV"

#debug
#export ASCEND_GLOBAL_LOG_LEVEL=3
#export ASCEND_HOST_LOG_FILE_NUM=100

nic_name="bond0"
local_ip=$(ifconfig $nic_name 2>/dev/null | grep 'inet ' | awk '{print $2}' | head -n 1)

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10

export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
#export VLLM_ASCEND_BALANCE_SCHEDULING=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1

export VLLM_ASCEND_ENABLE_FLASHCOMM1=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1

#kvstore
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
export PYTHONHASHSEED=0 
#export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export MOONCAKE_CONFIG_PATH="/data/yxliu64/kvstore_script/mooncake.json"
#export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3
export ACL_OP_INIT_MODE=1
#A3
#export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
export HCCL_INTRA_ROCE_ENABLE=1
# Unit: ms. The timeout for one-sided communication connection establishment is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039). Users can adjust this value based on their specific setup.
# The recommended formula is: ASCEND_CONNECT_TIMEOUT = connection_time_per_card (typically within 500ms) × total_number_of_Decode_cards.
# This ensures that even in the worst-case scenario?where all Decode cards simultaneously attempt to connect to the same Prefill card the connection will not time out.
export ASCEND_CONNECT_TIMEOUT=10000
# Unit: ms. The timeout for one-sided communication transfer is set to 10 seconds by default (see PR: https://github.com/kvcache-ai/Mooncake/pull/1039).
export ASCEND_TRANSFER_TIMEOUT=10000
export ASCEND_BUFFER_POOL=4:8
export HCCL_RDMA_TIMEOUT=16

#debug
#echo "/tmp/core.%u.%e.%p" > /proc/sys/kernel/core_pattern
#ulimit -c unlimited

vllm serve /data/shiqiu/Eco-Tech/GLM-5-w8a8/ \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --no-enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 73728 \
    --max-num-batched-tokens 4096 \
    --trust-remote-code \
    --max-num-seqs 160 \
    --gpu-memory-utilization 0.92 \
    --quantization ascend \
    --async-scheduling \
    --enforce-eager \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_producer",
    "kv_connector_extra_config": {
        "connectors": [
            {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_producer",
                "kv_port": "30000",
                "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
                "kv_connector_extra_config": {
                    "use_ascend_direct": true,
                    "prefill": {
                        "dp_size": 4,
                        "tp_size": 8
                    },
                    "decode": {
                        "dp_size": 8,
                        "tp_size": 4
                    }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_producer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"0",
                    "backend": "mooncake"
                }
            }
        ]
    }
    }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true
    }' \
    --speculative-config '{"num_speculative_tokens": 1, "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/data/yxliu64/profiling",
        "torch_profiler_with_stack": false
    }' \
    2>&1 | tee glm.log

D实例

# D实例
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/lib
export VLLM_ASCEND_ENABLE_MLAPO=1
export VLLM_ASCEND_ENABLE_NZ=1
export HCCL_OP_EXPANSION_MODE="AIV"

#debug
#export ASCEND_GLOBAL_LOG_LEVEL=3
#export ASCEND_HOST_LOG_FILE_NUM=100
#export TORCH_COMPILE_DEBUG=1
#export ASCEND_SLOG_PRINT_TO_STDOUT=1
#export ASCEND_MODULE_LOG_LEVEL=OP=1
#export ASCEND_GLOBAL_LOG_LEVEL=0

nic_name="bond0"
local_ip=$(ifconfig $nic_name 2>/dev/null | grep 'inet ' | awk '{print $2}' | head -n 1)

export HCCL_IF_IP=$local_ip
export GLOO_SOCKET_IFNAME=$nic_name
export TP_SOCKET_IFNAME=$nic_name
export HCCL_SOCKET_IFNAME=$nic_name

export OMP_PROC_BIND=false
export OMP_NUM_THREADS=10
export VLLM_USE_V1=1
export HCCL_BUFFSIZE=200
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
#export VLLM_ASCEND_BALANCE_SCHEDULING=1
export TASK_QUEUE_ENABLE=1
export CPU_AFFINITY_CONF=1
export ASCEND_AGGREGATE_ENABLE=1
export ASCEND_TRANSPORT_PRINT=1
export ACL_OP_INIT_MODE=1
#export VLLM_NIXL_ABORT_REQUEST_TIMEOUT=300000
export ASCEND_RT_VISIBLE_DEVICES=$1

#kvstore
#export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
#export PYTHONPATH=$PYTHONPATH:/xxxxx/vllm
export PYTHONHASHSEED=0 
export MOONCAKE_CONFIG_PATH="/data/yxliu64/kvstore_script/mooncake.json"
#export ASCEND_RT_VISIBLE_DEVICES=4,5,6,7
export ACL_OP_INIT_MODE=1
#A3
#export ASCEND_ENABLE_USE_FABRIC_MEM=1
#A2
export HCCL_INTRA_ROCE_ENABLE=1
export ASCEND_CONNECT_TIMEOUT=10000
export ASCEND_TRANSFER_TIMEOUT=10000
export ASCEND_BUFFER_POOL=4:8
export HCCL_RDMA_TIMEOUT=16

#static kernel
export LOCAL_WORLD_SIZE=8

#debug
#echo "/tmp/core.%u.%e.%p" > /proc/sys/kernel/core_pattern
#ulimit -c unlimited

vllm serve /data/shiqiu/Eco-Tech/GLM-5-w8a8/ \
    --host 0.0.0.0 \
    --port $2 \
    --data-parallel-size $3 \
    --data-parallel-rank $4 \
    --data-parallel-address $5 \
    --data-parallel-rpc-port $6 \
    --tensor-parallel-size $7 \
    --enable-expert-parallel \
    --enable-chunked-prefill \
    --no-enable-prefix-caching \
    --seed 1024 \
    --served-model-name glm5 \
    --max-model-len 73728 \
    --max-num-batched-token 160 \
    --trust-remote-code \
    --max-num-seqs 80 \
    --gpu-memory-utilization 0.85 \
    --async-scheduling \
    --quantization ascend \
    --enable-auto-tool-choice \
    --tool-call-parser glm47 \
    --reasoning-parser glm45 \
    --kv-transfer-config \
    '{
    "kv_connector": "MultiConnector",
    "kv_role": "kv_consumer",
    "kv_connector_extra_config": {
        "connectors": [
        {
                "kv_connector": "MooncakeConnectorV1",
                "kv_role": "kv_consumer",
                "kv_port": "30100",
                "kv_connector_module_path": "vllm_ascend.distributed.mooncake_connector",
                "kv_connector_extra_config": {
                        "use_ascend_direct": true,
                        "prefill": {
                                "dp_size": 4,
                                "tp_size": 8
                        },
                        "decode": {
                                "dp_size": 8,
                                "tp_size": 4
                        }
                }
            },
            {
                "kv_connector": "AscendStoreConnector",
                "kv_role": "kv_consumer",
                "kv_connector_extra_config": {
                    "lookup_rpc_port":"1",
                    "backend": "mooncake"
                }
            }
        ]
    }
    }' \
    --compilation-config \
    '{
        "cudagraph_capture_sizes": [4,16,64,96,128,160],
        "cudagraph_mode": "FULL_DECODE_ONLY"
     }' \
    --additional-config \
    '{
        "recompute_scheduler_enable":true,
        "multistream_overlap_shared_expert":true,
        "fuse_qknorm_rope": false,
        "fuse_muls_add": true,
        "enable_npugraph_ex": true,
	"ascend_compilation_config": {"enable_static_kernel": true}
    }' \
    --speculative-config '{"num_speculative_tokens": 1,  "method":"deepseek_mtp"}' \
    --profiler-config \
    '{
        "profiler": "torch",
        "torch_profiler_dir": "/data/yxliu64/profiling",
        "torch_profiler_with_stack": false
     }' \
    2>&1 | tee glm.log

7、推理验证：

curl http://localhost:8000/v1/chat/completions -H "Content-type: application/json" -d '{
        "model": "glm-5",
        "messages": [
                {
                        "role": "user",
                        "content": "请做一下自我介绍"
                }
        ],
        "stream": false,
        "ignore_eos": true,
        "max_tokens": 200
}'

8、性能测试：

参考指令如下，基于实际场景修改相关参数

python3 aisbench_test.py --input_len 32768 --output_len 2048 --data_num 16 --concurrency 4 --request_rate 0 --dataset_type prefix_cache --repeat_rate 0.5 --seed 1024

【附】部署脚本

（1）launch_online_dp.py

import argparse
import multiprocessing
import os
import subprocess
import sys

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--dp-size",
        type=int,
        required=True,
        help="Data parallel size."
    )
    parser.add_argument(
        "--tp-size",
        type=int,
        default=1,
        help="Tensor parallel size."
    )
    parser.add_argument(
        "--dp-size-local",
        type=int,
        default=-1,
        help="Local data parallel size."
    )
    parser.add_argument(
        "--dp-rank-start",
        type=int,
        default=0,
        help="Starting rank for data parallel."
    )
    parser.add_argument(
        "--dp-address",
        type=str,
        required=True,
        help="IP address for data parallel master node."
    )
    parser.add_argument(
        "--dp-rpc-port",
        type=str,
        default=12345,
        help="Port for data parallel master node."
    )
    parser.add_argument(
        "--vllm-start-port",
        type=int,
        default=9000,
        help="Starting port for the engine."
    )
    return parser.parse_args()

args = parse_args()
dp_size = args.dp_size
tp_size = args.tp_size
dp_size_local = args.dp_size_local
if dp_size_local == -1:
    dp_size_local = dp_size
dp_rank_start = args.dp_rank_start
dp_address = args.dp_address
dp_rpc_port = args.dp_rpc_port
vllm_start_port = args.vllm_start_port

def run_command(visible_devices, dp_rank, vllm_engine_port):
    command = [
        "bash",
        "./run_dp_template.sh",
        visible_devices,
        str(vllm_engine_port),
        str(dp_size),
        str(dp_rank),
        dp_address,
        dp_rpc_port,
        str(tp_size),
    ]
    subprocess.run(command, check=True)

if __name__ == "__main__":
    template_path = "./run_dp_template.sh"
    if not os.path.exists(template_path):
        print(f"Template file {template_path} does not exist.")
        sys.exit(1)

    processes = []
    num_cards = dp_size_local * tp_size
    for i in range(dp_size_local):
        dp_rank = dp_rank_start + i
        vllm_engine_port = vllm_start_port + i
        visible_devices = ",".join(str(x) for x in range(i * tp_size, (i + 1) * tp_size))
        process = multiprocessing.Process(target=run_command,
                                        args=(visible_devices, dp_rank,
                                                vllm_engine_port))
        processes.append(process)
        process.start()

    for process in processes:
        process.join()

（2）load_balance_proxy_server_example.py 通过github获取：https://github.com/vllm-project/vllm-ascend/blob/main/examples/disaggregated_prefill_v1/load_balance_proxy_server_example.py

声明

1、非商用发布产品，请勿直接用于生产环境。