ZEN2 模型参数量达6.68亿，采用N-gram编码增强文本语义理解，擅长处理各类自然语言理解（NLU）任务。
ZEN2 model, which uses N-gram to enhance text semantic and has 668M parameters, is adept at NLU tasks.
| 需求 Demand | 任务 Task | 系列 Series | 模型 Model | 参数 Parameter | 额外 Extra |
|---|---|---|---|---|---|
| 通用 General | 自然语言理解 NLU | 二郎神 Erlangshen | ZEN2 | 668M | 中文-Chinese |
我们与ZEN团队合作，基于封神框架开源发布了ZEN2模型。具体来说，ZEN通过融合无监督学习提取的知识，利用N-gram方法学习不同粒度的文本信息。ZEN2则采用大规模数据集和特殊的预训练策略，对N-gram增强编码器进行预训练。接下来，我们将继续与ZEN团队携手，探索预训练语言模型（PLM）的优化方向，提升下游任务性能。
We open source and publicly release ZEN2 using our Fengshen Framework in collaboration with the ZEN team. More precisely, by bringing together knowledge extracted by unsupervised learning, ZEN learns different textual granularity information through N-gram methods. ZEN2 pre-trains the N-gram-enhanced encoders with large-scale datasets and special pre-training strategies. In the next step, we continue with the ZEN team to explore the optimization of PLM and improve the performance on downstream tasks.
分类任务 Classification
| 模型(准确率) | afqmc | tnews | iflytek | ocnli | cmnli |
|---|---|---|---|---|---|
| Erlangshen-ZEN2-345M-Chinese | 0.741 | 0.584 | 0.599 | 0.788 | 0.80 |
| Erlangshen-ZEN2-668M-Chinese | 0.75 | 0.60 | 0.589 | 0.81 | 0.82 |
抽取任务 Extraction
| 模型(F1值) | WEIBO(test) | Resume(test) | MSRA(test) | OntoNote4.0(test) | CMeEE(dev) | CLUENER(dev) |
|---|---|---|---|---|---|---|
| Erlangshen-ZEN2-345M-Chinese | 65.26 | 96.03 | 95.15 | 78.93 | 62.81 | 79.27 |
| Erlangshen-ZEN2-668M-Chinese | 70.02 | 96.08 | 95.13 | 80.89 | 63.37 | 79.22 |
由于transformers库中未包含ZEN2相关的模型结构，因此您可以在我们的Fengshenbang-LM中找到该模型结构并运行代码。
Since there is no structure of ZEN2 in transformers library, you can find the structure of ZEN2 and run the codes in Fengshenbang-LM.
git clone https://github.com/IDEA-CCNL/Fengshenbang-LM.gitfrom fengshen.models.zen2.ngram_utils import ZenNgramDict
from fengshen.models.zen2.tokenization import BertTokenizer
from fengshen.models.zen2.modeling import ZenForSequenceClassification, ZenForTokenClassification
pretrain_path = 'IDEA-CCNL/Erlangshen-ZEN2-668M-Chinese'
tokenizer = BertTokenizer.from_pretrained(pretrain_path)
model_classification = ZenForSequenceClassification.from_pretrained(pretrain_path)
model_extraction = ZenForTokenClassification.from_pretrained(pretrain_path)
ngram_dict = ZenNgramDict.from_pretrained(pretrain_path, tokenizer=tokenizer)
你可以从下方的链接获得我们做分类和抽取的详细示例。
You can get classification and extraction examples below.
分类 classification example on fengshen
抽取 extraction example on fengshen
如果您在您的工作中使用了我们的模型，可以引用我们的对该模型的论文：
If you are using the resource for your work, please cite the our paper for this model:
@article{Sinovation2021ZEN2,
title="{ZEN 2.0: Continue Training and Adaption for N-gram Enhanced Text Encoders}",
author={Yan Song, Tong Zhang, Yonggang Wang, Kai-Fu Lee},
journal={arXiv preprint arXiv:2105.01279},
year={2021},
}如果您在您的工作中使用了我们的模型，也可以引用我们的总论文：
如果您将本资源用于您的工作，请引用我们的总论文：
@article{fengshenbang,
author = {Jiaxing Zhang and Ruyi Gan and Junjie Wang and Yuxiang Zhang and Lin Zhang and Ping Yang and Xinyu Gao and Ziwei Wu and Xiaoqun Dong and Junqing He and Jianheng Zhuo and Qi Yang and Yongfeng Huang and Xiayu Li and Yanghan Wu and Junyu Lu and Xinyu Zhu and Weifeng Chen and Ting Han and Kunhao Pan and Rui Wang and Hao Wang and Xiaojun Wu and Zhongshen Zeng and Chongpei Chen},
title = {Fengshenbang 1.0: Being the Foundation of Chinese Cognitive Intelligence},
journal = {CoRR},
volume = {abs/2209.02970},
year = {2022}
}也可以引用我们的网站:
您也可以引用我们的网站:
@misc{Fengshenbang-LM,
title={Fengshenbang-LM},
author={IDEA-CCNL},
year={2021},
howpublished={\url{https://github.com/IDEA-CCNL/Fengshenbang-LM}},
}