Meta-Llama-3-8B Ascend NPU 适配

# 从 ModelScope 下载（无需认证）
pip install modelscope
python3 -c "
from modelscope import snapshot_download
snapshot_download(
    'LLM-Research/Meta-Llama-3-8B',
    cache_dir='./Meta-Llama-3-8B'
)
"

huggingface-cli download meta-llama/Meta-Llama-3-8B --local-dir ./Meta-Llama-3-8B

python3 -m vllm.entrypoints.openai.api_server \
  --model ./Meta-Llama-3-8B/LLM-Research/Meta-Llama-3-8B \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len 4096 \
  --max-num-seqs 8 \
  --port 8000 \
  --trust-remote-code

# 查看已加载模型
curl http://127.0.0.1:8000/v1/models

# 文本续写推理（适合基座模型）
curl -s http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "./Meta-Llama-3-8B/LLM-Research/Meta-Llama-3-8B",
    "prompt": "The capital of France is",
    "temperature": 0,
    "max_tokens": 64
  }'

# 对话推理（需 chat template）
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "model": "./Meta-Llama-3-8B/LLM-Research/Meta-Llama-3-8B",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "Hello! Please introduce yourself briefly."}
    ],
    "temperature": 0.7,
    "max_tokens": 128
  }'

python3 inference_npu.py

+------------------------------------------------------------------------------------------------+
| NPU   Name                | Health        | Power(W)    Temp(C)           Hugepages-Usage(page)|
| Chip  Phy-ID              | Bus-Id        | AICore(%)   Memory-Usage(MB)  HBM-Usage(MB)        |
+===========================+===============+====================================================+
| 0     Ascend910           | OK            | 185.6       47                0    / 0             |
| 0     0                   | 0000:0A:00.0  | 0           0    / 0          59685/ 65536         |
+===========================+===============+====================================================+
| NPU     Chip              | Process id    | Process name             | Process memory(MB)      |
+===========================+===============+====================================================+
| 0       0                 | 17743         | VLLMEngineCor            | 56630                   |
+===========================+===============+====================================================+

{
  "model": "Meta-Llama-3-8B",
  "prompt": "The capital of France is",
  "temperature": 0,
  "max_tokens": 64
}

{
  "id": "cmpl-ac5e2c6ee15bcc2c",
  "object": "text_completion",
  "model": "Meta-Llama-3-8B",
  "choices": [{
    "index": 0,
    "text": " Paris. It is located in the north of the country. The city is situated on the banks of the Seine River. Paris is the largest city in France. It is also the largest city in the European Union. The city is home to many famous landmarks, including the Eiffel Tower, the Louvre Museum",
    "finish_reason": "length"
  }],
  "usage": {"prompt_tokens": 6, "total_tokens": 70, "completion_tokens": 64}
}

{
  "prompt": "Python is a programming language that",
  "temperature": 0,
  "max_tokens": 64
}

{
  "id": "cmpl-a7334e57c0f6d3bb",
  "choices": [{
    "text": " lets you work quickly and integrate systems more effectively. Python is a general-purpose language, but it’s particularly good at text processing, scripting, and automation. It’s also a great language for beginners to learn, because it’s easy to read and write.\nPython is a high-level, interpreted language that is used for a",
    "finish_reason": "length"
  }],
  "usage": {"prompt_tokens": 7, "total_tokens": 71, "completion_tokens": 64}
}

{
  "prompt": "Artificial intelligence is",
  "temperature": 0,
  "max_tokens": 64
}

{
  "id": "cmpl-8fa1ed2ec41708c9",
  "choices": [{
    "text": " a branch of computer science that deals with the creation of intelligent machines that work and react like humans. It is a branch of computer science that deals with the creation of intelligent machines that work and react like humans.",
    "finish_reason": "length"
  }],
  "usage": {"prompt_tokens": 5, "total_tokens": 69, "completion_tokens": 64}
}

{
  "model": "Meta-Llama-3-8B",
  "messages": [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Hello! Please introduce yourself briefly."}
  ],
  "temperature": 0.7,
  "max_tokens": 128
}

{
  "id": "chatcmpl-af9a0b0dbc48183f",
  "choices": [{
    "message": {
      "role": "assistant",
      "content": "I'm a helpful assistant. I can help you in many ways. What can I do for you?"
    },
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 28, "total_tokens": 50, "completion_tokens": 22}
}

.
├── README.md              # 本文档
├── inference_npu.py       # NPU 推理脚本
├── Meta-Llama-3-8B/       # 模型权重目录
│   └── LLM-Research/
│       └── Meta-Llama-3-8B/
│           ├── config.json
│           ├── tokenizer.json
│           ├── tokenizer_config.json
│           ├── model-00001-of-00004.safetensors
│           ├── model-00002-of-00004.safetensors
│           ├── model-00003-of-00004.safetensors
│           ├── model-00004-of-00004.safetensors
│           └── model.safetensors.index.json
└── LICENSE