MiniCPM 技术报告 Technical Report | OmniLMM 多模态模型 Multi-modal Model | CPM-C 千亿模型试用 ~100B Model Trial
MiniCPM 是面壁与清华大学自然语言处理实验室共同开源的系列端侧语言大模型，主体语言模型 MiniCPM-1B 仅有 12亿（1.2B）的非词嵌入参数量。
我们将完全开源MiniCPM-2B的模型参数供学术研究和有限商用，以及训练过程中的所有Checkpoint和大部分非专有数据供模型机理研究。
MiniCPM is an End-Size LLM developed by ModelBest Inc. and TsinghuaNLP, with only 1.2B parameters excluding embeddings.
We release all model parameters for research and limited commercial use. We also release all the checkpoint during training and most public training data for research on model mechanism.
详细的评测结果位于github仓库
Detailed evaluation results are in github repo
注意：我们发现使用Huggingface生成质量略差于vLLM，因此推荐使用vLLM进行测试。我们正在排查原因。
Notice: We discovered that the quality of Huggingface generation is slightly lower than vLLM, thus benchmarking using vLLM is recommended. We are investigating the cause now.
受限于模型规模，模型可能出现幻觉性问题。其中由于DPO模型生成的回复内容更长，更容易出现幻觉。我们也将持续进行MiniCPM模型的迭代改进；
为了保证在学术研究用途上模型的通用性，我们未对模型进行任何身份认同训练。同时由于我们用ShareGPT开源语料作为部分训练数据，模型可能会输出类似GPT系列模型的身份认同信息；
受限于模型规模，模型的输出受到提示词（prompt）的影响较大，可能多次尝试产生不一致的结果；
受限于模型容量，模型的知识记忆较不准确，后续我们将结合RAG方法来增强模型的知识记忆能力。
Due to limitations in model size, the model may experience hallucinatory issues. As DPO model tend to generate longer response, hallucinations are more likely to occur. We will also continue to iterate and improve the MiniCPM model.
To ensure the universality of the model for academic research purposes, we did not conduct any identity training on the model. Meanwhile, as we use ShareGPT open-source corpus as part of the training data, the model may output identity information similar to the GPT series models.
Due to the limitation of model size, the output of the model is greatly influenced by prompt words, which may result in inconsistent results from multiple attempts.
Due to limited model capacity, the model's knowledge memory is not accurate. In the future, we will combine the RAG method to enhance the model's knowledge memory ability.
| HuggingFace | ModelScope | WiseModel |
|---|---|---|
| sft-bf16 | sft-bf16 | sft-bf16 |
| sft-fp32 | sft-fp32 | sft-fp32 |
| dpo-bf16 | dpo-bf16 | dpo-bf16 |
| dpo-fp16 | dpo-fp16 | dpo-fp16 |
| dpo-fp32 | dpo-fp32 | dpo-fp32 |
transformers>=4.36.0以及accelerate后，运行以下代码from_pretrained中明确指明模型的数据类型，否则会引起较大计算误差transformers>=4.36.0 and acceleratefrom transformers import AutoModelForCausalLM, AutoTokenizer
import torch
torch.manual_seed(0)
path = 'openbmb/MiniCPM-2B-sft-bf16'
tokenizer = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, torch_dtype=torch.bfloat16, device_map='cuda', trust_remote_code=True)
responds, history = model.chat(tokenizer, "山东省最高的山是哪座山, 它比黄山高还是矮？差距多少？", temperature=0.8, top_p=0.8)
print(responds)山东省最高的山是泰山，海拔1545米。
相对于黄山（海拔1864米），泰山海拔较低，相差约319米。本仓库中代码依照 Apache-2.0 协议开源
MiniCPM 模型权重的使用则需要遵循 “通用模型许可协议-来源说明-宣传限制-商业授权”。
MiniCPM 模型权重对学术研究完全开放。
如需将模型用于商业用途，请联系cpm@modelbest.cn来获取书面授权，在登记后亦允许免费商业使用。
This repository is released under the Apache-2.0 License.
The usage of MiniCPM model weights must strictly follow the General Model License (GML).
The models and weights of MiniCPM are completely free for academic research.
If you intend to utilize the model for commercial purposes, please reach out to cpm@modelbest.cn to obtain the certificate of authorization.
作为一个语言模型，MiniCPM 通过学习大量的文本来生成内容，但它无法理解、表达个人观点或价值判断，它所输出的任何内容都不代表模型开发者的观点和立场。
因此用户在使用 MiniCPM 生成的内容时，应自行负责对其进行评估和验证。
如果由于使用 MinCPM 开源模型而导致的任何问题，包括但不限于数据安全问题、公共舆论风险，或模型被误导、滥用、传播或不当利用所带来的任何风险和问题，我们将不承担任何责任。
As a language model, MiniCPM generates content by learning from a vast amount of text.
However, it does not possess the ability to comprehend or express personal opinions or value judgments.
Any content generated by MiniCPM does not represent the viewpoints or positions of the model developers.
Therefore, when using content generated by MiniCPM, users should take full responsibility for evaluating and verifying it on their own.
@inproceedings{minicpm2024,
title={MiniCPM：Unveiling the Potential of End-side Large Language Models},
booktitle={OpenBMB Blog},
year={2024}
}