hf_mirrors/Multilingual-MiniLM-L12-H384
Model Introduction
Files & versions
Analysis
You are visiting the mirror address of https://modelers.cn/wuhaicc/Multilingual-MiniLM-L12-H384. The repo is synchronized in real-time and is only for mirror acceleration access.

MiniLM: Small and Fast Pre-trained Models for Language Understanding and Generation

Please note: This checkpoint uses BertModel with XLMRobertaTokenizer so AutoTokenizer won't work with this checkpoint!

Multilingual Pretrained Model

  • Multilingual-MiniLMv1-L12-H384: 12-layer, 384-hidden, 12-heads, 21M Transformer parameters, 96M embedding parameters

We evaluate the multilingual MiniLM on cross-lingual natural language inference benchmark (XNLI) and cross-lingual question answering benchmark (MLQA).

Model#Layers#Hidden#Transformer ParametersAverageenfresdeelbgrutrarvithzhhiswur
[mBERT]1276885M66.382.173.874.371.166.468.969.061.664.969.555.869.360.050.458.0
[XLM-100]161280315M70.783.276.777.774.072.774.172.768.768.672.968.972.565.658.262.4
[XLM-R Base]1276885M74.584.678.478.976.875.977.375.473.271.575.472.574.971.165.266.5
mMiniLM-L12xH3841238421M71.181.574.875.772.973.074.571.369.768.872.167.870.066.263.364.2

This example code fine-tunes 12-layer multilingual MiniLM on XNLI.

python3 examples/inference.py --model_name_or_path=./

Cross-Lingual Question Answering - [MLQA]

Following [Lewis et al. (2019b)], we adopt SQuAD 1.1 as training data and use MLQA English development data for early stopping.

Model F1 Score#Layers#Hidden#Transformer ParametersAverageenesdearhivizh
[mBERT]1276885M57.777.764.357.945.743.857.157.5
[XLM-15]121024151M61.674.968.062.254.848.861.461.1
[XLM-R Base] (Reported)1276885M62.977.867.260.853.057.963.160.2
[XLM-R Base] (Our fine-tuned)1276885M64.980.367.062.755.060.466.562.3
mMiniLM-L12xH3841238421M63.279.466.161.254.958.563.159.0

Citation

If you find MiniLM useful in your research, please cite the following paper:

@misc{wang2020minilm,
    title={MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers},
    author={Wenhui Wang and Furu Wei and Li Dong and Hangbo Bao and Nan Yang and Ming Zhou},
    year={2020},
    eprint={2002.10957},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}