hf_mirrors/bilingual-gpt-neox-4b-instruction-sft
Model Introduction
Files & versions
Analysis

bilingual-gpt-neox-4b-instruction-sft

rinna-icon


Update

  • 2023/08/02 We uploaded the newly trained rinna/bilingual-gpt-neox-4b-instruction-sft with the MIT license.
    • Please refrain from using the previous model released on 2023/07/31 for commercial purposes if you have already downloaded it.
    • The new model released on 2023/08/02 is built from datasets with less strict licenses and has better evaluation performance, so we suggest using the new model.
    • For reference, we provide the MD5 checksum values for the pytorch_model.bin files of the previous and current models.
      • 2023/07/31 model: edf190a323c0ae63f71476700fb0b462
      • 2023/08/02 model: de72aa5b66beee7b65783c96f687d186
  • 2023/07/31 In the previously released rinna/bilingual-gpt-neox-4b-instruction-sft, we found that part of the training data have a non-commercial license, and thus it does not comply with the MIT license. We decided to remove the previous version and build a new SFT model from datasets with less strict licenses. The new model will be uploaded in a few days. We sincerely apologize for our careless mistake.

Overview

This repository provides an English-Japanese bilingual GPT-NeoX model of 3.8 billion parameters.

The model is based on rinna/bilingual-gpt-neox-4b and has been finetuned to serve as an instruction-following conversational agent.

  • Model architecture

    A 36-layer, 2816-hidden-size transformer-based language model.

  • Fine-tuning

    The fine-tuning data is the subset of the following datasets.

    • [Anthropic HH RLHF data] and its Japanese translation
    • [FLAN Instruction Tuning data] and its Japanese translation
  • Model Series

    VariantLink
    Bilingual 4B MiniGPT4
    Bilingual 4B PPOhttps://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b-instruction-ppo
    Bilingual 4B SFThttps://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b-instruction-sft
    Bilingual 4B 8K
    Bilingual 4Bhttps://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b
    Japanese 3.6B PPO
    Japanese 3.6B SFT-v2
    Japanese 3.6B SFT
    Japanese 3.6B
  • Contributors

    [Tianyu Zhao] and [Kei Sawada]


Benchmarking

Our evaluation experiments suggest that the bilingual-gpt-neox-4b-instruction-sft model performs slightly better than the previous [Japanese GPT-NeoX 3.6B PPO] in Japanese tasks.

  • The 4-task average accuracy is based on results of JCommonsenseQA, JNLI, MARC-ja, and JSQuAD.
  • The 6-task average accuracy is based on results of JCommonsenseQA, JNLI, MARC-ja, JSQuAD, XWinograd, and JAQKET-v2.
Model4-task average accuracy6-task average accuracy
bilingual-gpt-neox-4b-instruction-ppo61.0161.16
bilingual-gpt-neox-4b-instruction-sft61.0261.69
bilingual-gpt-neox-4b56.1251.83
japanese-gpt-neox-3.6b-instruction-ppo59.8660.07
japanese-gpt-neox-3.6b55.0750.32

How to use the model

Notice: Since the model is sensitive to decoding hyper-parameters (e.g. temperature, top_p, top_k, repetition_penalty), it is suggested to explore the best setting for your task.

import argparse
from openmind import AutoModel, AutoTokenizer
from openmind import is_torch_npu_available

def parse_args():
    parser = argparse.ArgumentParser()
    parser.add_argument("--model_name_or_path",type=str,help="Path to model",default=None,)
    args = parser.parse_args()
    return args


if __name__ == '__main__':
    if is_torch_npu_available():
        device = "npu:0"
    else:
        device = "cpu"
    args = parse_args()
    model_path = args.model_name_or_path

    # Note: CodeSage requires adding eos token at the end of
    # each tokenized sequence to ensure good performance
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, add_eos_token=True)

    model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(device)

    inputs = tokenizer.encode("def print_hello_world():\tprint('Hello World!')", return_tensors="pt").to(device)
    embedding = model(inputs)[0]
    print(f'Dimension of the embedding: {embedding[0].size()}')
    print(embedding)

Tokenization

The model uses a [sentencepiece]-based tokenizer.

  • The tokenizer has a vocabulary size of 65,536.
  • It uses byte fallback to decompose unknown text pieces into UTF-8 byte pieces to avoid producing <UNK> tokens.
  • It can recognize consecutive whitespaces, newlines, and tabs to handle structured texts better.
  • We turned off the default behaviour of prepending leading whitespace because it is not beneficial for processing Japanese.
  • Specifically, single whitespace is always processed as one token so that any English word won't have a preceding whitespace like in many other tokenizers (e.g. _Hello).
    • This decision trades the English processing efficiency for a unified way to treat whitespaces.
    • It leads to a significantly lower loss of next token prediction on English data because whitespaces are easy to predict.
  • Don't forget to set use_fast=False to make the above features function correctly.