hf_mirrors/gpt2-xl-conversational
Model Introduction
Files & versions
Analysis

Model Card

Modification

  • Modify example and add npu support
  • Add denpendencies

Model Details

  • Model Name: gpt2-xl-conversational
  • Model Type: Language Modeling
  • Task: Generating Conversational Responses
  • Hardware: 1x Nvidia Titan V
  • Description: This model is trained on a dataset of conversations between a user and an AI assistant, with the goal of generating a coherent and relevant response to the user's input. It uses the GPT-2 architecture, a state-of-the-art transformer-based language model that is capable of generating high-quality text with a wide range of styles and tones. The model is fine-tuned on the conversational data using maximum likelihood estimation, and is evaluated based on its ability to generate responses that are both grammatically correct and semantically relevant to the user's input.

Intended Use

This model is intended to be used for generating conversational responses in a variety of contexts, such as chatbots, virtual assistants, and customer service applications. It is designed to provide natural and engaging responses to user input, with a focus on maintaining a consistent tone and style throughout the conversation. The model is suitable for use in both text-based and voice-based interfaces, and can be easily integrated into existing applications using the PyTorch and Transformers frameworks.

Training Data

The model is trained on a large dataset of conversational data, consisting of interactions between users and an AI assistant. The data is preprocessed to remove any sensitive information and is formatted in a way that is suitable for training a language model. The training data is split into a training set and a validation set, with the training set used to update the model parameters and the validation set used to evaluate the model performance. The model was trained on 300,000 examples and achieved excellent metrics.

Model Architecture

The model architecture used in this model is GPT-2, a transformer-based language model that is capable of generating high-quality text with a wide range of styles and tones. The GPT-2 architecture consists of a multi-layered decoder-only transformer, with self-attention mechanisms that allow the model to capture long-term dependencies and generate coherent text.

Evaluation Metrics

The model is evaluated based on several metrics, including loss, reward, penalty, BLEU score, and perplexity. The loss metric is calculated during training and reflects the difference between the predicted output and the actual output. The reward metric is based on the number of correct words generated by the model, while the penalty metric penalizes the model for repeating words consecutively. The BLEU score measures the similarity between the generated text and the ground truth text, while the perplexity metric measures how well the model is able to predict the next word in a sequence. During training, the model achieved the following metrics:

  • BLEU score: 52
  • Accuracy: 53
  • perplexity: 4.3

Evaluation metrics:

TaskVersionMetricValueStderr
pubmedqa0acc0.536±0.0223
arc_challenge0acc_norm0.2867±0.0132
arc_easy0acc0.5804±0.0101
arc_easy0acc_norm0.5707±0.0102
winogrande0acc0.5691±0.0139
truthfulqa_mc1mc20.3918±0.0144
anli_r10acc0.338±0.0150
anli_r20acc0.346±0.0151
anli_r30acc0.355±0.0138
drop1f10.0034±0.0004
hendrycksTest-abstract_algebra1acc0.32±0.0952
hendrycksTest-anatomy1acc0.44±0.1013
hendrycksTest-astronomy1acc0.24±0.0872
hendrycksTest-business_ethics1acc0.24±0.0872
hendrycksTest-clinical_knowledge1acc0.24±0.0872
hendrycksTest-college_biology1acc0.20±0.0816
hendrycksTest-college_chemistry1acc0.40±0.1000
hendrycksTest-college_computer_science1acc0.36±0.0980
hendrycksTest-college_mathematics1acc0.48±0.1020
hendrycksTest-college_medicine1acc0.20±0.0816
hendrycksTest-college_physics1acc0.44±0.1013
hendrycksTest-computer_security1acc0.16±0.0748
hendrycksTest-conceptual_physics1acc0.12±0.0663
hendrycksTest-econometrics1acc0.16±0.0748
hendrycksTest-electrical_engineering1acc0.28±0.0917
hendrycksTest-elementary_mathematics1acc0.36±0.0980
hendrycksTest-formal_logic1acc0.44±0.1013
hendrycksTest-global_facts1acc0.20±0.0816
hendrycksTest-high_school_biology1acc0.20±0.0816
hendrycksTest-high_school_chemistry1acc0.28±0.0917
hendrycksTest-high_school_computer_science1acc0.24±0.0872
hendrycksTest-high_school_european_history1acc0.32±0.0952
hendrycksTest-high_school_geography1acc0.32±0.0952
hendrycksTest-high_school_government_and_politics1acc0.28±0.0917
hendrycksTest-high_school_macroeconomics1acc0.28±0.0917
hendrycksTest-high_school_mathematics1acc0.20±0.0816
hendrycksTest-high_school_microeconomics1acc0.24±0.0872
hendrycksTest-high_school_physics1acc0.28±0.0917
hendrycksTest-high_school_psychology1acc0.32±0.0952
hendrycksTest-high_school_statistics1acc0.40±0.1000
hendrycksTest-high_school_us_history1acc0.32±0.0952
hendrycksTest-high_school_world_history1acc0.36±0.0980
hendrycksTest-human_aging1acc0.16±0.0748
hendrycksTest-human_sexuality1acc0.40±0.1000
hendrycksTest-international_law1acc0.24±0.0872
hendrycksTest-jurisprudence1acc0.08±0.0554
hendrycksTest-logical_fallacies1acc0.52±0.1020
hendrycksTest-machine_learning1acc0.12±0.0663
hendrycksTest-management1acc0.12±0.0663
hendrycksTest-marketing1acc0.16±0.0748
hendrycksTest-medical_genetics1acc0.12±0.0663
hendrycksTest-miscellaneous1acc0.36±0.0980
hendrycksTest-moral_disputes1acc0.08±0.0554
hendrycksTest-moral_scenarios1acc0.44±0.1013
hendrycksTest-nutrition1acc0.32±0.0952
hendrycksTest-philosophy1acc0.44±0.1013
hendrycksTest-prehistory1acc0.16±0.0748
hendrycksTest-professional_accounting1acc0.28±0.0917
hendrycksTest-professional_law1acc0.12±0.0663
hendrycksTest-professional_medicine1acc0.40±0.1000
hendrycksTest-professional_psychology1acc0.24±0.0872
hendrycksTest-public_relations1acc0.08±0.0554
hendrycksTest-security_studies1acc0.24±0.0872
hendrycksTest-sociology1acc0.28±0.0917
hendrycksTest-us_foreign_policy1acc0.24±0.0872
hendrycksTest-virology1acc0.20±0.0816
hendrycksTest-world_religions1acc0.16±0.0748

Denpendencies

  • transformers==4.44.2
  • psutil==6.0.0
  • better_profanity==0.7.0
  • einops==0.6.1
  • protobuf==5.28.2

Limitations and Bias

This model is not suitable for all use cases due to its limited training time on a weak computer. As a result, it may produce irrelevant or nonsensical responses. For optimal performance, I recommend using a NPU with at least 16 GB of VRAM and downloading the model manually instead of using the Transformers library. Here's how you should deploy the model:

import argparse
import torch
from openmind import pipeline, is_torch_npu_available
from openmind_hub import snapshot_download

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument(
        "--model_name_or_path",
        type=str,
        help="Path to model",
        default=None,
    )
    args = parser.parse_args()
    return args

def main():
    args = parse_args()
    model_path = args.model_name_or_path

    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
        
    generator = pipeline('text-generation', model=model_path, device=device)

    output = generator("Hello, I'm a language model,", max_length=30, num_return_sequences=5)
    print(f">>>output={output}", flush=True)

if __name__ == "__main__":
    main()

Deploying and training the model

The model has been fine-tuned on a specific input format that goes like this "<|USER|> {user prompt} <|ASSISTANT|> {model prediction} ".