
rinna/bilingual-gpt-neox-4b-instruction-sft with the MIT license.
pytorch_model.bin files of the previous and current models.
edf190a323c0ae63f71476700fb0b462de72aa5b66beee7b65783c96f687d186rinna/bilingual-gpt-neox-4b-instruction-sft, we found that part of the training data have a non-commercial license, and thus it does not comply with the MIT license. We decided to remove the previous version and build a new SFT model from datasets with less strict licenses. The new model will be uploaded in a few days. We sincerely apologize for our careless mistake.This repository provides an English-Japanese bilingual GPT-NeoX model of 3.8 billion parameters.
The model is based on rinna/bilingual-gpt-neox-4b and has been finetuned to serve as an instruction-following conversational agent.
Model architecture
A 36-layer, 2816-hidden-size transformer-based language model.
Fine-tuning
The fine-tuning data is the subset of the following datasets.
Model Series
| Variant | Link |
|---|---|
| Bilingual 4B MiniGPT4 | |
| Bilingual 4B PPO | https://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b-instruction-ppo |
| Bilingual 4B SFT | https://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b-instruction-sft |
| Bilingual 4B 8K | |
| Bilingual 4B | https://modelers.cn/models/FuJianAscend/bilingual-gpt-neox-4b |
| Japanese 3.6B PPO | |
| Japanese 3.6B SFT-v2 | |
| Japanese 3.6B SFT | |
| Japanese 3.6B |
Contributors
[Tianyu Zhao] and [Kei Sawada]
Our evaluation experiments suggest that the bilingual-gpt-neox-4b-instruction-sft model performs slightly better than the previous [Japanese GPT-NeoX 3.6B PPO] in Japanese tasks.
| Model | 4-task average accuracy | 6-task average accuracy |
|---|---|---|
| bilingual-gpt-neox-4b-instruction-ppo | 61.01 | 61.16 |
| bilingual-gpt-neox-4b-instruction-sft | 61.02 | 61.69 |
| bilingual-gpt-neox-4b | 56.12 | 51.83 |
| japanese-gpt-neox-3.6b-instruction-ppo | 59.86 | 60.07 |
| japanese-gpt-neox-3.6b | 55.07 | 50.32 |
Notice: Since the model is sensitive to decoding hyper-parameters (e.g. temperature, top_p, top_k, repetition_penalty), it is suggested to explore the best setting for your task.
import argparse
from openmind import AutoModel, AutoTokenizer
from openmind import is_torch_npu_available
def parse_args():
parser = argparse.ArgumentParser()
parser.add_argument("--model_name_or_path",type=str,help="Path to model",default=None,)
args = parser.parse_args()
return args
if __name__ == '__main__':
if is_torch_npu_available():
device = "npu:0"
else:
device = "cpu"
args = parse_args()
model_path = args.model_name_or_path
# Note: CodeSage requires adding eos token at the end of
# each tokenized sequence to ensure good performance
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True, add_eos_token=True)
model = AutoModel.from_pretrained(model_path, trust_remote_code=True).to(device)
inputs = tokenizer.encode("def print_hello_world():\tprint('Hello World!')", return_tensors="pt").to(device)
embedding = model(inputs)[0]
print(f'Dimension of the embedding: {embedding[0].size()}')
print(embedding)
The model uses a [sentencepiece]-based tokenizer.
<UNK> tokens._Hello).
use_fast=False to make the above features function correctly.