奖励模型（RM）经过训练，可在给定问题的情况下预测哪个人工生成的答案更优。
奖励模型在以下领域十分有用：
问答模型评估
在RLHF中作为奖励分数
通过排序检测潜在的有害响应
所有模型均在以下数据集上训练，且跨数据集使用相同的拆分种子（若未提供验证拆分）：
from transformers import AutoModelForSequenceClassification, AutoTokenizer
reward_name = "OpenAssistant/reward-model-deberta-v3-large-v2"
rank_model, tokenizer = AutoModelForSequenceClassification.from_pretrained(reward_name), AutoTokenizer.from_pretrained(reward_name)
question, answer = "Explain nuclear fusion like I am five", "Nuclear fusion is the process by which two or more protons and neutrons combine to form a single nucleus. It is a very important process in the universe, as it is the source of energy for stars and galaxies. Nuclear fusion is also a key process in the production of energy for nuclear power plants."
inputs = tokenizer(question, answer, return_tensors='pt')
score = rank_model(**inputs).logits[0].cpu().detach()
print(score)有害响应检测
from transformers import AutoModelForSequenceClassification, AutoTokenizer
reward_name = "OpenAssistant/reward-model-deberta-v3-large-v2"
rank_model, tokenizer = AutoModelForSequenceClassification.from_pretrained(reward_name), AutoTokenizer.from_pretrained(reward_name)
question = "I just came out of from jail, any suggestion of my future?"
helpful = "It's great to hear that you have been released from jail."
bad = "Go back to jail you scum"
inputs = tokenizer(question, helpful, return_tensors='pt')
good_score = rank_model(**inputs).logits[0].cpu().detach()
inputs = tokenizer(question, bad, return_tensors='pt')
bad_score = rank_model(**inputs).logits[0].cpu().detach()
print(good_score > bad_score) # tensor([True])验证集准确率
| 模型 | WebGPT | Summary | SytheticGPT | Anthropic RLHF |
|---|---|---|---|---|
| electra-large-discriminator | 59.30 | 68.66 | 99.85 | 54.33 |
| deberta-v3-large-v2 | 61.57 | 71.47 | 99.88 | 69.25 |
| deberta-v3-large | 61.13 | 72.23 | 99.94 | 55.62 |
| deberta-v3-base | 59.07 | 66.84 | 99.85 | 54.51 |
| deberta-v2-xxlarge | 58.67 | 73.27 | 99.77 | 66.74 |
SytheticGPT 可能在所选-被拒的文本对上存在某种表面模式，这使得区分哪个答案更好变得非常容易。
衷心感谢 stability.ai 提供的 A100 计算资源支持。他们的贡献对本研究项目的顺利完成起到了至关重要的作用。