本文档记录 facebook/chameleon-7b 模型在 vLLM-Ascend 环境下的快速部署与验证结果。
Chameleon-7b 是 Meta 开发的 70 亿参数多模态基础模型,基于 Chameleon 架构,支持文本和图像的混合理解与生成。
相关获取地址:
| 组件 | 版本 |
|---|---|
vllm-ascend | 0.18.0rc1 |
vllm | 0.18.0 |
transformers | 4.57.6 |
torch-npu | 2.9.0 |
CANN | 8.5.1 |
SOC | ascend910_9391 |
1 逻辑卡(Ascend 910B2,64GB HBM)/home/openmind/.cache/modelscope/hub/models/facebook/chameleon-7b| 参数 | 值 |
|---|---|
| 架构 | ChameleonForConditionalGeneration |
| 参数量 | 7B |
| 层数 | 32 |
| 隐藏维度 | 4096 |
| 注意力头数 | 32 |
| KV 头数 | 32 |
| 词表大小 | 65536 |
| 精度 | BF16(原始)/ Float16(vLLM 推理) |
# 从 ModelScope 下载
modelscope download --model facebook/chameleon-7b使用 vLLM-Ascend 加载模型进行推理:
from vllm import LLM, SamplingParams
MODEL_PATH = "facebook/chameleon-7b"
llm = LLM(
model=MODEL_PATH,
trust_remote_code=True,
dtype="float16",
tensor_parallel_size=1,
max_model_len=2048,
gpu_memory_utilization=0.9,
)
sampling = SamplingParams(max_tokens=64, temperature=0)
outputs = llm.generate(["The capital of France is"], sampling)
print(outputs[0].outputs[0].text)
# Paris.运行 4 项冒烟测试验证模型功能:
python3 smoke_test.py=== chameleon-7b vLLM-Ascend Smoke Test ===
[1/4] Testing basic generation...
PASS: Paris.
[2/4] Testing factual knowledge...
PASS: The answer to 2 + 2 is 4.
[3/4] Testing batch generation...
[1] Jupiter. It is a gas giant, meaning it is primarily composed...
[2] The ocean is a vast and mysterious body of water that covers...
[3] The sky on a clear day is typically a deep blue color...
PASS: 3 prompts processed
[4/4] Testing reasoning...
PASS: John has 5 apples. If he gives 2 to Mary, he will have 5 - 2 = 3 apples left.
=== All 4/4 tests PASSED (96.3s) ===验证项说明:
| 测试项 | 输入 | 预期 | 结果 |
|---|---|---|---|
| 英文基础生成 | The capital of France is | 包含 Paris | PASS |
| 事实知识 | What is 2 + 2? | 包含 4 | PASS |
| 批量生成 | 3 个 prompt | 3 条非空输出 | PASS |
| 推理能力 | 数学推理题目 | 包含 3 | PASS |
modelscope/ai2_arcpython3 eval_arc_vllm.py --output logs/arc_challenge_vllm.json| 指标 | 基线 | NPU 结果 | 差异 |
|---|---|---|---|
| ARC-Challenge 25-shot | 58.02% | 57.85% (678/1172) | -0.17% |
ARC-Challenge NPU 准确率为 57.85%,与基线 58.02% 的差异为 -0.17%(< 1%),验证通过。
=== chameleon-7b ARC-Challenge 25-shot vLLM-Ascend ===
Total test samples: 1172
Loading model with vLLM-Ascend...
Model loaded.
Generating (1172 samples)...
============================================================
chameleon-7b ARC-Challenge 25-shot Results (vLLM-Ascend)
============================================================
Accuracy: 57.85% (678/1172)
Total time: 19s
Throughput: 61.97 samples/s
============================================================
Results saved to logs/arc_challenge_vllm.jsonpython3 benchmark.py| 场景 | 输出 tokens | 吞吐量 | TPOT |
|---|---|---|---|
| 单请求 | 128 | 17.19 tok/s | 58.2 ms/tok |
| 批量 (8 并发) | 512 | 135.03 tok/s | — |
=== chameleon-7b vLLM-Ascend Benchmark ===
--- Single prompt (128 tokens) ---
Run 1: 128 tokens, 7.51s, 17.05 tok/s, TPOT 58.6 ms/tok
Run 2: 128 tokens, 7.53s, 16.99 tok/s, TPOT 58.9 ms/tok
Run 3: 128 tokens, 7.50s, 17.06 tok/s, TPOT 58.6 ms/tok
Run 4: 128 tokens, 7.30s, 17.52 tok/s, TPOT 57.1 ms/tok
Run 5: 128 tokens, 7.38s, 17.34 tok/s, TPOT 57.7 ms/tok
Avg: 17.19 tok/s, TPOT 58.2 ms/tok
--- Batch (8 prompts, 64 tokens each) ---
Run 1: 512 tokens, 3.88s, 132.02 tok/s
Run 2: 512 tokens, 3.65s, 140.32 tok/s
Run 3: 512 tokens, 3.86s, 132.75 tok/s
Avg: 135.03 tok/s
==================================================
Single prompt: 17.19 tok/s, TPOT 58.2 ms/tok
Batch (8): 135.03 tok/s
==================================================多模态模型:Chameleon 支持文本和图像混合输入,本次验证仅测试文本生成能力。
显存占用:float16 推理时模型约占用 13.1 GB HBM,单卡即可运行。
BF16 转 Float16:模型原始精度为 BF16,vLLM-Ascend 自动转换为 Float16 推理(NPU 原生支持)。
推理速度:单请求 17.19 tok/s,8 并发批量 135.03 tok/s。