开放思维

import torch
from openmind import AutoTokenizer,is_torch_npu_available
import argparse
from openmind import AutoModel,pipeline
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] # model_output的第一个元素包含所有token嵌入
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to model",
        default="../",
    )
    args = parser.parse_args()
    return args
def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
    device='npu'
    
#     model = AutoModel.from_pretrained("./flan-t5-xxl").to(device)
#     tokenizer = AutoTokenizer.from_pretrained("./flan-t5-xxl")

    #inputs = tokenizer("Several states in the United States, including Texas and Florida, are considering prohibiting Chinese citizens from buying local real estate, mainly due to national security concerns. What is China's comment?", max_length=1024, return_tensors="pt")
    en_str = "Several states in the United States, including Texas and Florida, are considering prohibiting Chinese citizens from buying local real estate, mainly due to national security concerns. What is China's comment?"

    generator = pipeline(task="text2text-generation", 
                         model="./flan-t5-xxl", 
                         device="npu:0",
                         torch_dtype=torch.float16
                         )
    generated_text = generator(en_str)
    print(generated_text)
if __name__ == "__main__":
    main()

FLAN-T5 XXL 模型卡片

drawing

摘要

如果您已了解 T5，那么 FLAN-T5 在各方面都更胜一筹。在参数数量相同的情况下，这些模型在 1000 多个额外任务上进行了微调，涵盖的语言也更多。

正如摘要开头几行所述：

Flan-PaLM 540B 在多个基准测试中取得了最先进的性能，例如在五样本 MMLU 上达到 75.2%。我们还公开发布了 Flan-T5 检查点，1 即使与更大的模型（如 PaLM 62B）相比，也能实现强大的少样本性能。总体而言，指令微调是一种提高预训练语言模型性能和可用性的通用方法。

免责声明：本模型卡片的内容由 Hugging Face 团队撰写，部分内容复制粘贴自 T5 模型卡片。

模型详情

模型描述

模型类型：语言模型
支持语言（NLP）：英语、德语、法语
许可证：Apache 2.0
相关模型：所有 FLAN-T5 检查点
原始检查点：所有原始 FLAN-T5 检查点
更多信息资源：

使用方法

以下是一些如何在 transformers 中使用该模型的示例脚本：

使用 Pytorch 模型

在 CPU 上运行模型

点击展开


from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl")

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

在 GPU 上运行模型

点击展开

# pip install accelerate
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

在 GPU 上以不同精度运行模型

FP16

点击展开

# pip install accelerate
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", torch_dtype=torch.float16)

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

INT8

点击展开

# pip install bitsandbytes accelerate
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", load_in_8bit=True)

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

用途

直接用途和下游用途

作者在原始论文的模型卡片中写道：

主要用途是语言模型研究，包括：零样本自然语言处理任务和上下文少样本学习自然语言处理任务的研究，例如推理和问答；推进公平性和安全性研究，以及理解当前大型语言模型的局限性

有关更多详细信息，请参见研究论文。

超出范围的用途

需要更多信息。

偏差、风险和局限性

本节以下信息复制自模型的官方模型卡片：

Rae等人（2021）指出，包括Flan-T5在内的语言模型有可能被以有害方式用于语言生成。Flan-T5不应直接用于任何应用，除非事先针对该应用特有的安全性和公平性问题进行评估。

伦理考量和风险

Flan-T5是在大量文本数据语料库上进行微调的，这些数据未经过滤以排除显式内容，也未对现有偏差进行评估。因此，模型本身可能容易生成同样不适当的内容或复制底层数据中固有的偏差。

已知局限性

Flan-T5尚未在实际应用中进行测试。

敏感用途：

Flan-T5不应应用于任何不可接受的用例，例如生成辱骂性言论。

训练详情

训练数据

该模型在多种任务的混合数据上进行训练，包括下表中描述的任务（来自原始论文，图2）：

训练过程

根据原始论文的模型卡片：

这些模型基于预训练的T5（Raffel等人，2020），并通过指令进行微调，以获得更好的零样本和少样本性能。每个T5模型大小对应一个经过微调的Flan模型。

该模型已在TPU v3或TPU v4 pods上训练，使用t5x代码库以及jax。

评估

测试数据、因素与指标

作者在涵盖多种语言（共1836种）的各类任务上对模型进行了评估。部分定量评估结果如下表所示：完整详情请查阅研究论文。

结果

FLAN-T5-XXL的完整结果，请参见研究论文中的表3。

环境影响

可使用Lacoste等人（2019）提出的机器学习影响计算器来估算碳排放。

硬件类型： Google Cloud TPU Pods - TPU v3 或 TPU v4 | 芯片数量 ≥ 4。
使用时长： 需更多信息
云服务提供商： GCP
计算区域： 需更多信息
碳排放： 需更多信息

引用

BibTeX：

@misc{https://doi.org/10.48550/arxiv.2210.11416,
  doi = {10.48550/ARXIV.2210.11416},
  
  url = {https://arxiv.org/abs/2210.11416},
  
  author = {Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Eric and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and Webson, Albert and Gu, Shixiang Shane and Dai, Zhuyun and Suzgun, Mirac and Chen, Xinyun and Chowdhery, Aakanksha and Narang, Sharan and Mishra, Gaurav and Yu, Adams and Zhao, Vincent and Huang, Yanping and Dai, Andrew and Yu, Hongkun and Petrov, Slav and Chi, Ed H. and Dean, Jeff and Devlin, Jacob and Roberts, Adam and Zhou, Denny and Le, Quoc V. and Wei, Jason},
  
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {Scaling Instruction-Finetuned Language Models},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}

修改部分

CI测试

1、原模型

2、SFT之后的模型

损失对比

训练脚本

NPU

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=5,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

图形处理器

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=5,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

结果

NPU GPU

从训练后的总损失来看，NPU和GPU的损失基本保持一致

性能对比

对比脚本

NPU

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=10,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=1,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

图形处理器

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=10,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=1,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

结果

NPU GPU

从上面的对比来看 NPU/GPU=0.262/0.258=1.0155>0.7

开放思维

import torch
from openmind import AutoTokenizer,is_torch_npu_available
import argparse
from openmind import AutoModel,pipeline
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0] # model_output的第一个元素包含所有token嵌入
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to model",
        default="../",
    )
    args = parser.parse_args()
    return args
def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
    device='npu'
    
#     model = AutoModel.from_pretrained("./flan-t5-xxl").to(device)
#     tokenizer = AutoTokenizer.from_pretrained("./flan-t5-xxl")

    #inputs = tokenizer("Several states in the United States, including Texas and Florida, are considering prohibiting Chinese citizens from buying local real estate, mainly due to national security concerns. What is China's comment?", max_length=1024, return_tensors="pt")
    en_str = "Several states in the United States, including Texas and Florida, are considering prohibiting Chinese citizens from buying local real estate, mainly due to national security concerns. What is China's comment?"

    generator = pipeline(task="text2text-generation", 
                         model="./flan-t5-xxl", 
                         device="npu:0",
                         torch_dtype=torch.float16
                         )
    generated_text = generator(en_str)
    print(generated_text)
if __name__ == "__main__":
    main()

FLAN-T5 XXL 模型卡片

drawing

摘要

如果您已了解 T5，那么 FLAN-T5 在各方面都更胜一筹。在参数数量相同的情况下，这些模型在 1000 多个额外任务上进行了微调，涵盖的语言也更多。

正如摘要开头几行所述：

Flan-PaLM 540B 在多个基准测试中取得了最先进的性能，例如在五样本 MMLU 上达到 75.2%。我们还公开发布了 Flan-T5 检查点，1 即使与更大的模型（如 PaLM 62B）相比，也能实现强大的少样本性能。总体而言，指令微调是一种提高预训练语言模型性能和可用性的通用方法。

免责声明：本模型卡片的内容由 Hugging Face 团队撰写，部分内容复制粘贴自 T5 模型卡片。

模型详情

模型描述

模型类型：语言模型
支持语言（NLP）：英语、德语、法语
许可证：Apache 2.0
相关模型：所有 FLAN-T5 检查点
原始检查点：所有原始 FLAN-T5 检查点
更多信息资源：

使用方法

以下是一些如何在 transformers 中使用该模型的示例脚本：

使用 Pytorch 模型

在 CPU 上运行模型

点击展开


from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl")

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

在 GPU 上运行模型

点击展开

# pip install accelerate
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto")

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

在 GPU 上以不同精度运行模型

FP16

点击展开

# pip install accelerate
import torch
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", torch_dtype=torch.float16)

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

INT8

点击展开

# pip install bitsandbytes accelerate
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("google/flan-t5-xxl")
model = T5ForConditionalGeneration.from_pretrained("google/flan-t5-xxl", device_map="auto", load_in_8bit=True)

input_text = "translate English to German: How old are you?"
input_ids = tokenizer(input_text, return_tensors="pt").input_ids.to("cuda")

outputs = model.generate(input_ids)
print(tokenizer.decode(outputs[0]))

用途

直接用途和下游用途

作者在原始论文的模型卡片中写道：

主要用途是语言模型研究，包括：零样本自然语言处理任务和上下文少样本学习自然语言处理任务的研究，例如推理和问答；推进公平性和安全性研究，以及理解当前大型语言模型的局限性

有关更多详细信息，请参见研究论文。

超出范围的用途

需要更多信息。

偏差、风险和局限性

本节以下信息复制自模型的官方模型卡片：

Rae等人（2021）指出，包括Flan-T5在内的语言模型有可能被以有害方式用于语言生成。Flan-T5不应直接用于任何应用，除非事先针对该应用特有的安全性和公平性问题进行评估。

伦理考量和风险

Flan-T5是在大量文本数据语料库上进行微调的，这些数据未经过滤以排除显式内容，也未对现有偏差进行评估。因此，模型本身可能容易生成同样不适当的内容或复制底层数据中固有的偏差。

已知局限性

Flan-T5尚未在实际应用中进行测试。

敏感用途：

Flan-T5不应应用于任何不可接受的用例，例如生成辱骂性言论。

训练详情

训练数据

该模型在多种任务的混合数据上进行训练，包括下表中描述的任务（来自原始论文，图2）：

训练过程

根据原始论文的模型卡片：

这些模型基于预训练的T5（Raffel等人，2020），并通过指令进行微调，以获得更好的零样本和少样本性能。每个T5模型大小对应一个经过微调的Flan模型。

该模型已在TPU v3或TPU v4 pods上训练，使用t5x代码库以及jax。

评估

测试数据、因素与指标

作者在涵盖多种语言（共1836种）的各类任务上对模型进行了评估。部分定量评估结果如下表所示：完整详情请查阅研究论文。

结果

FLAN-T5-XXL的完整结果，请参见研究论文中的表3。

环境影响

可使用Lacoste等人（2019）提出的机器学习影响计算器来估算碳排放。

硬件类型： Google Cloud TPU Pods - TPU v3 或 TPU v4 | 芯片数量 ≥ 4。
使用时长： 需更多信息
云服务提供商： GCP
计算区域： 需更多信息
碳排放： 需更多信息

引用

BibTeX：

@misc{https://doi.org/10.48550/arxiv.2210.11416,
  doi = {10.48550/ARXIV.2210.11416},
  
  url = {https://arxiv.org/abs/2210.11416},
  
  author = {Chung, Hyung Won and Hou, Le and Longpre, Shayne and Zoph, Barret and Tay, Yi and Fedus, William and Li, Eric and Wang, Xuezhi and Dehghani, Mostafa and Brahma, Siddhartha and Webson, Albert and Gu, Shixiang Shane and Dai, Zhuyun and Suzgun, Mirac and Chen, Xinyun and Chowdhery, Aakanksha and Narang, Sharan and Mishra, Gaurav and Yu, Adams and Zhao, Vincent and Huang, Yanping and Dai, Andrew and Yu, Hongkun and Petrov, Slav and Chi, Ed H. and Dean, Jeff and Devlin, Jacob and Roberts, Adam and Zhou, Denny and Le, Quoc V. and Wei, Jason},
  
  keywords = {Machine Learning (cs.LG), Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
  
  title = {Scaling Instruction-Finetuned Language Models},
  
  publisher = {arXiv},
  
  year = {2022},
  
  copyright = {Creative Commons Attribution 4.0 International}
}

修改部分

CI测试

1、原模型

2、SFT之后的模型

损失对比

训练脚本

NPU

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=5,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

图形处理器

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=1,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=5,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

结果

NPU GPU

从训练后的总损失来看，NPU和GPU的损失基本保持一致

性能对比

对比脚本

NPU

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=10,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=1,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

图形处理器

#!/usr/bin/env python3

from datasets import load_dataset
from datasets import concatenate_datasets
from transformers import AutoTokenizer,TrainingArguments,Trainer
from peft import LoraConfig, get_peft_model, TaskType
import evaluate
import nltk
import numpy as np
from nltk.tokenize import sent_tokenize
from transformers import DataCollatorForSeq2Seq
# from huggingface_hub import HfFolder
from transformers import Seq2SeqTrainer, Seq2SeqTrainingArguments,AutoModelForSeq2SeqLM

# Load dataset from the hub
# dataset = load_dataset(dataset_id)

model_id="./google/flan-t5-xxl"
dataset = load_dataset('json', data_files={'train':'test.jsonl'})

print(f"Train dataset size: {len(dataset['train'])}")
# print(f"Test dataset size: {len(dataset['test'])}")

# Load tokenizer of FLAN-t5-base
tokenizer = AutoTokenizer.from_pretrained(model_id)

# The maximum total input sequence length after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded.
tokenized_inputs = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["prompt"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_source_length = max([len(x) for x in tokenized_inputs["input_ids"]])
print(f"Max source length: {max_source_length}")

# The maximum total sequence length for target text after tokenization.
# Sequences longer than this will be truncated, sequences shorter will be padded."
tokenized_targets = concatenate_datasets([dataset["train"]]).map(lambda x: tokenizer(x["reference_code"], truncation=True), batched=True, remove_columns=["prompt", "reference_code"])
max_target_length = max([len(x) for x in tokenized_targets["input_ids"]])
print(f"Max target length: {max_target_length}")

def preprocess_function(sample,padding="max_length"):
    # add prefix to the input for t5
    #inputs = ["question: " + item for item in sample["question"]]
    inputs = [item for item in sample["prompt"]]

    # tokenize inputs
    model_inputs = tokenizer(inputs, max_length=max_source_length, padding=padding, truncation=True)

    # Tokenize targets with the `text_target` keyword argument
    labels = tokenizer(text_target=sample["reference_code"], max_length=max_target_length, padding=padding, truncation=True)

    # If we are padding here, replace all tokenizer.pad_token_id in the labels by -100 when we want to ignore
    # padding in the loss.
    if padding == "max_length":
        labels["input_ids"] = [
            [(l if l != tokenizer.pad_token_id else -100) for l in label] for label in labels["input_ids"]
        ]

    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_dataset = dataset.map(preprocess_function, batched=True, remove_columns=["prompt", "reference_code"])
print(f"Keys of tokenized dataset: {list(tokenized_dataset['train'].features)}")

# load model from the hub
model = AutoModelForSeq2SeqLM.from_pretrained(model_id,device_map='auto')
model.gradient_checkpointing_enable()

def print_trainable_parameters(model):
    """
    Prints the number of trainable parameters in the model.
    """
    trainable_params = 0
    all_param = 0
    for _, param in model.named_parameters():
        all_param += param.numel()
        if param.requires_grad:
            trainable_params += param.numel()
    print(
        f"trainable params: {trainable_params} || all params: {all_param} || trainable%: {100 * trainable_params / all_param}"
    )


lora_config = LoraConfig(
    r=8, lora_alpha=16, target_modules=["q", "v"], lora_dropout=0.05, bias="none", task_type="SEQ_2_SEQ_LM"
)


model = get_peft_model(model, lora_config)
print_trainable_parameters(model)
# model.to("cuda")
model.enable_input_require_grads()
# nltk.download("punkt")

# Metric
metric = evaluate.load("rouge")
print("===============================")
# helper function to postprocess text
def postprocess_text(preds, labels):
    preds = [pred.strip() for pred in preds]
    labels = [label.strip() for label in labels]

    # rougeLSum expects newline after each sentence
    preds = ["\n".join(sent_tokenize(pred)) for pred in preds]
    labels = ["\n".join(sent_tokenize(label)) for label in labels]

    return preds, labels

def compute_metrics(eval_preds):
    preds, labels = eval_preds
    if isinstance(preds, tuple):
        preds = preds[0]
    decoded_preds = tokenizer.batch_decode(preds, skip_special_tokens=True)
    # Replace -100 in the labels as we can't decode them.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

    # Some simple post-processing
    decoded_preds, decoded_labels = postprocess_text(decoded_preds, decoded_labels)

    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {k: round(v * 100, 4) for k, v in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in preds]
    result["gen_len"] = np.mean(prediction_lens)
    return result


# we want to ignore tokenizer pad token in the loss
label_pad_token_id = -100
# Data collator
data_collator = DataCollatorForSeq2Seq(
    tokenizer,
    model=model,
    label_pad_token_id=label_pad_token_id,
    pad_to_multiple_of=8
)
print('=============')
training_args = Seq2SeqTrainingArguments(
    output_dir="training_output",
    per_device_train_batch_size=10,
    per_device_eval_batch_size=2,
    bf16=True,
    gradient_checkpointing=True,
    gradient_accumulation_steps=1,
    learning_rate=1e-4,
    num_train_epochs=1,
    warmup_steps=500,
    # logging & evaluation strategies
    logging_dir=f"logs",
    logging_strategy="steps",
    logging_steps=1,
    evaluation_strategy="no",
    save_strategy="epoch",
    save_total_limit=2,
    # report_to="tensorboard",
)

# Create Trainer instance
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset["train"]
#     eval_dataset=tokenized_dataset["test"],
#     compute_metrics=compute_metrics,
#     data_collator=DataCollatorForSeq2Seq(tokenizer=tokenizer, padding=True)
)

trainer.train()

# trainer.evaluate()

model.save_pretrained("final_save")
#trainer.create_model_card()
# Push the results to the hub
#trainer.push_to_hub()

结果

NPU GPU

从上面的对比来看 NPU/GPU=0.262/0.258=1.0155>0.7

开放思维

FLAN-T5 XXL 模型卡片

目录

摘要

模型详情

模型描述

使用方法

使用 Pytorch 模型

在 CPU 上运行模型

在 GPU 上运行模型

在 GPU 上以不同精度运行模型

FP16

INT8

用途

直接用途和下游用途

超出范围的用途

偏差、风险和局限性

伦理考量和风险

已知局限性

敏感用途：

训练详情

训练数据

训练过程

评估

测试数据、因素与指标

结果

环境影响

引用

修改部分

CI测试

1、原模型

2、SFT之后的模型

损失对比

训练脚本

结果

性能对比

对比脚本

结果

开放思维

FLAN-T5 XXL 模型卡片

目录

摘要

模型详情

模型描述

使用方法

使用 Pytorch 模型

在 CPU 上运行模型

在 GPU 上运行模型

在 GPU 上以不同精度运行模型

FP16

INT8

用途

直接用途和下游用途

超出范围的用途

偏差、风险和局限性

伦理考量和风险

已知局限性

敏感用途：

训练详情

训练数据

训练过程

评估

测试数据、因素与指标

结果

环境影响

引用

修改部分

CI测试

1、原模型

2、SFT之后的模型

损失对比

训练脚本

结果

性能对比

对比脚本

结果