hf_mirrors/deberta_base
Model Introduction
Files & versions
Analysis

DeBERTa: Decoding-enhanced BERT with Disentangled Attention

DeBERTa improves the BERT and RoBERTa models using disentangled attention and enhanced mask decoder. It outperforms BERT and RoBERTa on majority of NLU tasks with 80GB training data.

Please check the official repository for more details and updates.

Modification

Add example with how to use DeBERTa in openmind.

Fine-tuning on NLU tasks

We present the dev results on SQuAD 1.1/2.0 and MNLI tasks.

ModelSQuAD 1.1SQuAD 2.0MNLI-m
RoBERTa-base91.5/84.683.7/80.587.6
XLNet-Large-/--/80.286.8
DeBERTa-base93.1/87.286.2/83.188.8

How to use DeBERTa in openmind

import torch
from openmind import AutoTokenizer, is_torch_npu_available
from transformers import DebertaForMaskedLM

if is_torch_npu_available():
    device = "npu:0"
elif torch.cuda.is_available():
    device = "cuda:0"
else:
    device = "cpu"

model_path= "PyTorch-NPU/deberta_base"

#推理
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = DebertaForMaskedLM.from_pretrained(model_path).to(device)
inputs = tokenizer("The capital of France is [MASK].", return_tensors="pt").to(device)

with torch.no_grad():
    logits = model(**inputs).logits

mask_token_index = (inputs.input_ids == tokenizer.mask_token_id)[0].nonzero(as_tuple=True)[0]

predicted_token_id = logits[0, mask_token_index].argmax(axis=-1)
print(">>>", tokenizer.decode(predicted_token_id))

Citation

If you find DeBERTa useful for your work, please cite the following paper:

@inproceedings{
he2021deberta,
title={DEBERTA: DECODING-ENHANCED BERT WITH DISENTANGLED ATTENTION},
author={Pengcheng He and Xiaodong Liu and Jianfeng Gao and Weizhu Chen},
booktitle={International Conference on Learning Representations},
year={2021},
url={https://openreview.net/forum?id=XPZIaotutsD}
}