Qwen3-0.6B-FP8 on vLLM-Ascend 0.18.0rc1

pip install modelscope

python3 -c "
from modelscope import AutoConfig, AutoTokenizer
config = AutoConfig.from_pretrained('Qwen/Qwen3-0.6B-FP8', trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained('Qwen/Qwen3-0.6B-FP8', trust_remote_code=True)
"

MODEL_SRC="/opt/atomgit/.cache/modelscope/hub/models/Qwen/Qwen3-0___6B-FP8"
MODEL_DST="/opt/atomgit/models/Qwen3-0.6B-FP8"
mkdir -p "$MODEL_DST"
cp "$MODEL_SRC/config.json" "$MODEL_DST/"
cp "$MODEL_SRC/tokenizer.json" "$MODEL_DST/"
cp "$MODEL_SRC/tokenizer_config.json" "$MODEL_DST/"
cp "$MODEL_SRC/vocab.json" "$MODEL_DST/"
cp "$MODEL_SRC/merges.txt" "$MODEL_DST/"
cp "$MODEL_SRC/generation_config.json" "$MODEL_DST/"
ln -sf "$MODEL_SRC/Qwen/Qwen3-0___6B-FP8/model.safetensors" "$MODEL_DST/model.safetensors"

export PYTHONPATH="/path/to/sitecustomize_dir:\$PYTHONPATH"

python3 -c "
import sitecustomize
from vllm import LLM, SamplingParams

llm = LLM(
    model='/opt/atomgit/models/Qwen3-0.6B-FP8',
    enforce_eager=True,
    max_model_len=2048,
    trust_remote_code=True,
    dtype='bfloat16',
)

outputs = llm.generate(['Hello!'], SamplingParams(temperature=0.0, max_tokens=32))
print(outputs[0].outputs[0].text)
"

python3 -m vllm.entrypoints.openai.api_server \
    --model /opt/atomgit/models/Qwen3-0.6B-FP8 \
    --enforce-eager \
    --max-model-len 2048 \
    --trust-remote-code \
    --dtype bfloat16 \
    --port 8000

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "/opt/atomgit/models/Qwen3-0.6B-FP8",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 64
  }'

from vllm import LLM, SamplingParams

llm = LLM(
    model='/opt/atomgit/models/Qwen3-0.6B-FP8',
    enforce_eager=True,
    max_model_len=2048,
    trust_remote_code=True,
    dtype='bfloat16',
)

params = SamplingParams(temperature=0.0, top_p=0.9, max_tokens=64)
outputs = llm.generate([
    'What is the capital of France?',
    'Write a short poem about AI.',
], params)
for o in outputs:
    print(f'Prompt: {o.prompt!r}')
    print(f'Output: {o.outputs[0].text!r}')

Q: What is the capital of France?
A: The capital of France is Paris. The answer is 2000. The question is why? Because...

Q: Who wrote the play Romeo and Juliet?
A: The play "Romeo and Juliet" was written by William Shakespeare.

Q: What is H2O?
A: H2O is a molecule composed of two hydrogen atoms and one oxygen atom. It is a common molecule found...