roberta_cnn_legal

概述

本仓库包含渥太华大学为LegalLens-2024共享任务中的子任务B（法律自然语言推理）开发的模型。该任务专注于法律文本间关系的分类，例如判断前提（如法律投诉摘要）与假设（如在线评论）之间是蕴含、矛盾还是中立关系。

模型详情

模型类型：基于Transformer的模型与卷积神经网络（CNN）相结合
框架：PyTorch、Transformers库
训练数据：LegalLens-2024组织者提供的LegalLensNLI数据集
架构：RoBERTa（ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli）与用于关键词模式检测的自定义CNN的集成
应用场景：法律文档间关系分类，适用于法律案例匹配和自动推理等应用

模型架构

模型架构包括：

RoBERTa模型：负责从输入文本中捕捉上下文信息。
CNN模型：用于关键词检测，包含一个嵌入层和三个卷积层，滤波器尺寸分别为（2, 3, 4）。
全连接层：结合RoBERTa和CNN的输出进行最终分类。

安装

要使用此模型，请克隆本仓库并确保已安装以下内容：

pip install torch
pip install transformers

在 Openmind 中的使用

from openmind import AutoTokenizer, AutoModel, is_torch_npu_available
from openmind_hub import snapshot_download
import torch.nn.functional as F
from torch import Tensor
import openmind
import torch
import argparse
import time

# Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to model",
        default="jeffding/roberta_cnn_legal-openmind",
    )
    args = parser.parse_args()
    return args

def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
        
    # Load model from HuggingFace Hub
    tokenizer = AutoTokenizer.from_pretrained(model_path,trust_remote_code=True)
    model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(device)
    start_time = time.time()
    sentences = ['如何更换花呗绑定银行卡', 'How to replace the Huabei bundled bank card']
    # Tokenize sentences
    encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt').to(device)

    # Compute token embeddings
    with torch.no_grad():
        model_output = model(**encoded_input)
    # Perform pooling. In this case, mean pooling.
    sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
    print("Sentence embeddings:")
    print(sentence_embeddings)
    
    end_time = time.time()
    print(f"硬件环境：{device},推理执行时间：{end_time - start_time}秒")
    
if __name__ == "__main__":
    main()

快速开始

使用 Hugging Face Transformers 库加载模型并运行推理：

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load the model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("nimamegh/roberta_cnn_legal")
tokenizer = AutoTokenizer.from_pretrained("nimamegh/roberta_cnn_legal")

# Example inputs
premise = "The cat is on the mat."
hypothesis = "The animal is on the mat."
inputs = tokenizer(premise, hypothesis, return_tensors='pt')

# Get predictions
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# Print the prediction result
print("Predicted class:", predictions.item())

# Interpretation (optional)
label_map = {0: "Entailment", 1: "Neutral", 2: "Contradiction"}
print("Result:", label_map[predictions.item()])

训练配置

学习率：2e-5
批处理大小：4（训练和评估）
训练轮数：20
权重衰减：0.01
优化器：AdamW
训练器类：用于微调，包含早停和预热步骤

评估指标

模型在验证集的多个领域上使用F1分数进行评估：

平均F1分数：88.6%

结果

隐藏测试集性能：F1分数为0.724，在LegalLens-2024竞赛中获得第5名。
对比：
- Falcon 7B：81.02%（各领域平均值）
- RoBERTa base：71.02%（平均值）
- uOttawa Model：88.6%（验证集平均值）

引用

@misc{meghdadi2024uottawalegallens2024transformerbasedclassification,
      title={uOttawa at LegalLens-2024: Transformer-based Classification Experiments}, 
      author={Nima Meghdadi and Diana Inkpen},
      year={2024},
      eprint={2410.21139},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.21139}, 
}

模型详情

模型类型：基于Transformer的模型与卷积神经网络（CNN）相结合

框架：PyTorch、Transformers库

训练数据：LegalLens-2024组织者提供的LegalLensNLI数据集

架构：RoBERTa（ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli）与用于关键词模式检测的自定义CNN的集成

应用场景：法律文档间关系分类，适用于法律案例匹配和自动推理等应用

在 Openmind 中的使用

from openmind import AutoTokenizer, AutoModel, is_torch_npu_available
from openmind_hub import snapshot_download
import torch.nn.functional as F
from torch import Tensor
import openmind
import torch
import argparse
import time

# Mean Pooling - Take attention mask into account for correct averaging
def mean_pooling(model_output, attention_mask):
    token_embeddings = model_output[0]  # First element of model_output contains all token embeddings
    input_mask_expanded = attention_mask.unsqueeze(-1).expand(token_embeddings.size()).float()
    return torch.sum(token_embeddings * input_mask_expanded, 1) / torch.clamp(input_mask_expanded.sum(1), min=1e-9)

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to model",
        default="jeffding/roberta_cnn_legal-openmind",
    )
    args = parser.parse_args()
    return args

def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
        
    # Load model from HuggingFace Hub
    tokenizer = AutoTokenizer.from_pretrained(model_path,trust_remote_code=True)
    model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(device)
    start_time = time.time()
    sentences = ['如何更换花呗绑定银行卡', 'How to replace the Huabei bundled bank card']
    # Tokenize sentences
    encoded_input = tokenizer(sentences, padding=True, truncation=True, return_tensors='pt').to(device)

    # Compute token embeddings
    with torch.no_grad():
        model_output = model(**encoded_input)
    # Perform pooling. In this case, mean pooling.
    sentence_embeddings = mean_pooling(model_output, encoded_input['attention_mask'])
    print("Sentence embeddings:")
    print(sentence_embeddings)
    
    end_time = time.time()
    print(f"硬件环境：{device},推理执行时间：{end_time - start_time}秒")
    
if __name__ == "__main__":
    main()

快速开始

使用 Hugging Face Transformers 库加载模型并运行推理：

from transformers import AutoTokenizer, AutoModelForSequenceClassification

# Load the model and tokenizer
model = AutoModelForSequenceClassification.from_pretrained("nimamegh/roberta_cnn_legal")
tokenizer = AutoTokenizer.from_pretrained("nimamegh/roberta_cnn_legal")

# Example inputs
premise = "The cat is on the mat."
hypothesis = "The animal is on the mat."
inputs = tokenizer(premise, hypothesis, return_tensors='pt')

# Get predictions
outputs = model(**inputs)
predictions = outputs.logits.argmax(dim=-1)

# Print the prediction result
print("Predicted class:", predictions.item())

# Interpretation (optional)
label_map = {0: "Entailment", 1: "Neutral", 2: "Contradiction"}
print("Result:", label_map[predictions.item()])

引用

@misc{meghdadi2024uottawalegallens2024transformerbasedclassification,
      title={uOttawa at LegalLens-2024: Transformer-based Classification Experiments}, 
      author={Nima Meghdadi and Diana Inkpen},
      year={2024},
      eprint={2410.21139},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
      url={https://arxiv.org/abs/2410.21139}, 
}