模型 ID 的模型卡片

上下文长度为 1280 的 deberta-v3-base，在 tasksource 上微调了 250k 步。我对长文本 NLI 任务（ConTRoL、doc-nli）进行了过采样。训练数据包括 helpsteer v1/v2、逻辑推理任务（FOLIO、FOL-nli、LogicNLI 等）、OASST、hh/rlhf、面向语言学的 NLI 任务、tasksource-dpo、事实核查任务。

此检查点在许多任务上具有出色的零样本验证性能（例如在 WNLI 上达到 70%），可用于：

针对任意标签的零样本基于蕴涵的分类 [ZS]。
自然语言推理 [NLI]
在新任务或 tasksource 任务（分类、 token 分类、奖励建模或多项选择）上进一步微调 [FT]。

dataset	accuracy
anli/a1	63.3
anli/a2	47.2
anli/a3	49.4
nli_fever	79.4
FOLIO	61.8
ConTRoL-nli	63.3
cladder	71.1
zero-shot-label-nli	74.4
chatbot_arena_conversations	72.2
oasst2_pairwise_rlhf_reward	73.9
doc-nli	90.0

零样本 GPT-4 在 FOLIO（逻辑推理）上的得分是 61%，在 cladder（概率推理）上是 62%，在 ConTRoL（长上下文 NLI）上是 56.4%。

[ZS] 零样本分类流程

from openmind import AutoModelForCausalLM, AutoTokenizer
from openmind import is_torch_npu_available, pipeline
import torch
import argparse
import torch.nn.functional as F
import time

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        "-m",
        type=str,
        help="Path to model",
        default="zhouhui/deberta-base-long-nli",
    )
    args = parser.parse_args()
    return args

def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"  
    #device = "cpu"  
    
    from openmind import pipeline
    classifier = pipeline("zero-shot-classification",
                          model=model_path, device=device, use_fast=True, multi_label=True,trust_remote_code=True)
    # we will classify the following wikipedia entry about Sardinia"
    
    start_time = time.time()
    text = "one day I will see the world"
    candidate_labels = ['travel', 'cooking', 'dancing']
    res=classifier(text, candidate_labels)
    
    print(f"生成结果： {res}")
    end_time = time.time()
    print(f"硬件环境：{device},推理执行时间：{end_time - start_time}秒")


if __name__ == "__main__":
    main()

该模型的NLI训练数据包含label-nli，这是一个专门为改进此类零样本分类而构建的NLI数据集。

[NLI] 自然语言推理流水线

from transformers import pipeline
pipe = pipeline("text-classification",model="tasksource/deberta-base-long-nli")
pipe([dict(text='there is a cat',
  text_pair='there is a black cat')]) #list of (premise,hypothesis)
# [{'label': 'neutral', 'score': 0.9952911138534546}]

[TA] Tasksource-adapters：一行代码接入数百个任务

# !pip install tasknet
import tasknet as tn
pipe = tn.load_pipeline('tasksource/deberta-base-long-nli','glue/sst2') # works for 500+ tasksource tasks
pipe(['That movie was great !', 'Awful movie.'])
# [{'label': 'positive', 'score': 0.9956}, {'label': 'negative', 'score': 0.9967}]

任务列表可在模型 config.json 中查看。

这比 ZS 更高效，因为每个样本只需一次前向传播，但灵活性较低。

[FT] Tasknet: 3 lines fine-tuning

# !pip install tasknet
import tasknet as tn
hparams=dict(model_name='tasksource/deberta-base-long-nli', learning_rate=2e-5)
model, trainer = tn.Model_Trainer([tn.AutoTask("glue/rte")], hparams)
trainer.train()

引用

关于此内容的更多详情，请参见文章：

@inproceedings{sileo-2024-tasksource,
    title = "tasksource: A Large Collection of {NLP} tasks with a Structured Dataset Preprocessing Framework",
    author = "Sileo, Damien",
    editor = "Calzolari, Nicoletta  and
      Kan, Min-Yen  and
      Hoste, Veronique  and
      Lenci, Alessandro  and
      Sakti, Sakriani  and
      Xue, Nianwen",
    booktitle = "Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)",
    month = may,
    year = "2024",
    address = "Torino, Italia",
    publisher = "ELRA and ICCL",
    url = "https://aclanthology.org/2024.lrec-main.1361",
    pages = "15655--15684",
}