由于Llama2本身的中文对齐较弱,我们采用中文指令集,对meta-llama/Llama-2-13b-chat-hf进行LoRA微调,使其具备较强的中文对话能力。
🎯 该版本为LoRA中文微调参数FlagAlpha/Llama2-Chinese-13b-Chat-LoRA和meta-llama/Llama-2-13b-chat-hf参数结合后的版本,可直接使用
# source environment variable
source /usr/local/Ascend/ascend-toolkit/set_env.sh
export OPENMIND_FRAMEWORK=ptOpenMind Library can be installed through pip, please select the corresponding command according to the actual environment for installation.
It should be noted that since torch npu depends on torch, which can be directly installed through pip in aarch64, but requires a specific URL to download the CPU version in x86, the installation commands in the two environments are different. The specific installation code has been distinguished and presented in the following text.
# aarch64
pip install openmind[all]
# x86
pip install openmind[all] --extra-index-url https://download.pytorch.org/whl/cpufrom openmind import AutoTokenizer, AutoModelForCausalLM
import torch
import torch_npu
model_dir = "HangZhou_Ascend/Llama2-Chinese-13b-Chat-ms"
tokenizer = AutoTokenizer.from_pretrained(model_dir, device_map="auto", trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and might cause OOM Error.
model = AutoModelForCausalLM.from_pretrained(model_dir, device_map="auto", trust_remote_code=True, torch_dtype=torch.float16)
model = model.eval()
response, history = model.chat(tokenizer, "1+1=", history=[], meta_instruction="")
print(response)