This checkpoint extends the routed-expert MXFP4 quantization by also quantizing the shared expert to MXFP4 and fusing it into the routed MoE kernel (FSE: fused shared expert), and by quantizing the MTP experts to MXFP4 and the MTP self-attention to FP8.
The model was quantized from Qwen/Qwen3.5-397B-A17B using AMD-Quark. Weights and activations are quantized to OCP MXFP4.
Quantization scripts:
MODEL_PATH="Qwen/Qwen3.5-397B-A17B"
OUTPUT_DIR="amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3"
cd /path/to/Quark
python3 examples/torch/language_modeling/llm_ptq/quantize_quark.py \
--model_dir "${MODEL_PATH}" \
--quant_scheme mxfp4 \
--layer_quant_scheme '*linear_attn*' ptpc_fp8 \
--layer_quant_scheme '*self_attn*' ptpc_fp8 \
--quant_flow file2file \
--no_trust_remote_code \
--model_export hf_format \
--skip_evaluation \
--multi_gpu balanced \
--exclude_layers "lm_head" "*mlp.gate" "mtp.fc" "model.visual.*" "*shared_expert_gate*" "*linear_attn.conv1d*" \
--output_dir "${OUTPUT_DIR}"For further details or issues, please refer to the AMD-Quark documentation or contact the respective developers.
The model was evaluated on the GSM8K benchmark using the SGLang framework with lm-evaluation-harness, following the InferenceX SGLang evaluation recipe (local-chat-completions with the chat template applied, 5-shot, greedy). The baseline is the original Qwen/Qwen3.5-397B-A17B checkpoint, evaluated with the identical recipe on SGLang.
This model can be deployed efficiently using the SGLang backend based on the Docker image lmsysorg/sglang-rocm:v0.5.20-rocm720-mi35x-20260922.
| Benchmark | Qwen/Qwen3.5-397B-A17B | amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 (this model) | Recovery |
|---|---|---|---|
| gsm8k (flexible-extract, 5-shot) | 97.71 | 97.57 | 99.86% |
The GSM8K results were obtained on SGLang following the InferenceX SGLang lm-eval recipe. Both sides were served with EAGLE MTP.
export SGLANG_USE_AITER=1
export SGLANG_USE_AITER_UNIFIED_ATTN=1
export AITER_FLYDSL_FORCE=1
export SGLANG_MAMBA_SSM_DTYPE=bfloat16
export SGLANG_USE_AITER_FP8_PER_TOKEN=1
python3 -m sglang.launch_server \
--model-path amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3 \
--trust-remote-code \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 4 \
--attention-backend aiter \
--mem-fraction-static 0.90 \
--model-loader-extra-config '{"enable_multithread_load": true}' \
--watchdog-timeout 1200 \
--disable-radix-cache \
--reasoning-parser qwen3 \
--page-size 16 \
--enable-aiter-allreduce-fusion \
--enable-metrics \
--speculative-algorithm EAGLE \
--speculative-num-steps 3 \
--speculative-eagle-topk 1 \
--speculative-num-draft-tokens 4doc_to_text that asks the model to end its response with #### <number> so the reasoning model's final answer is parsed correctly:lm_eval --model local-chat-completions --apply_chat_template \
--tasks gsm8k.yaml \
--num_fewshot 5 \
--model_args "model=amd/Qwen3.5-397B-A17B-MXFP4-AttnFP8-V3,base_url=http://127.0.0.1:8000/v1/chat/completions,num_concurrent=16,tokenized_requests=False,max_length=16384" \
--gen_kwargs "max_tokens=12288,temperature=0,top_p=1"Modifications Copyright(c) 2026 Advanced Micro Devices, Inc. All rights reserved.