This is a quantized version of Mixtral-8x7B-Instruct-v0.1 created by AMD using LLM Compressor (compressed-tensors) for ZenDNN-optimized CPU inference.
The model was quantized from Mixtral-8x7B-Instruct-v0.1 using LLM Compressor.
compressed-tensors, num_bits=8, type=int, symmetric=trueQuantization script:
python llmcompressor_w8a8.py \
--model_name mistralai/Mixtral-8x7B-Instruct-v0.1 \
--output_dir ./Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor-v0.10.0.2from vllm import LLM, SamplingParams
model = LLM(
model="amd/Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor-v0.10.0.2",
dtype="bfloat16",
)
sampling_params = SamplingParams(temperature=0.7, max_tokens=256)
outputs = model.generate(["Hello, how are you?"], sampling_params)
print(outputs[0].outputs[0].text)torch==2.11.0
zentorch==2.11.0.3
vllm==0.25.1For optimal performance, set LD_PRELOAD with libomp.so (LLVM OpenMP) or libiomp5.so (Intel OpenMP):
# Using LLVM OpenMP (llvmopenmp)
export LD_PRELOAD=$(find /path/to/env -name "libomp.so" | head -1)
# Or using Intel OpenMP (libiomp)
export LD_PRELOAD=$(find /path/to/env -name "libiomp5.so" | head -1)Note: Set
LD_PRELOADbefore launching vLLM or any inference script.
The model was evaluated against the BF16 (unquantized) baseline on standard benchmarks using lm-evaluation-harness with the vLLM engine.
| Benchmark | BF16 Baseline | W8A8 (this model) | Recovery |
|---|---|---|---|
| MMLU (5-shot) | - | - | - |
| GSM8K_COT (8-shot) | - | - | - |
| Perplexity (wikitext2) | - | - | - |
Evaluation results will be updated after benchmarking.
lm_eval \
--model vllm \
--model_args pretrained="amd/Mixtral-8x7B-Instruct-v0.1-w8a8-llmcompressor-v0.10.0.2" \
--tasks mmlu \
--num_fewshot 5 \
--batch_size autoThis model is distributed under the same license as the source model. See the LICENSE file for details.
Modifications copyright (c) 2026 Advanced Micro Devices, Inc. All rights reserved.