InternLM
简介
InternLM2.5 开源了 70 亿参数的基础模型和一款专为实际场景打造的对话模型。该模型具有以下特点：
-
卓越的推理能力：在数学推理方面表现领先，性能超越 Llama3 和 Gemma2-9B 等模型。
-
百万级上下文窗口：在百万长度上下文下的“大海捞针”任务中表现近乎完美，在 LongBench 等长上下文任务上性能领先。可搭配 LMDeploy 体验百万级上下文推理。
-
更强的工具调用能力：InternLM2.5 支持从超 100 个网页中获取信息，相关实现即将在 Lagent 中发布。InternLM2.5 在指令遵循、工具选择和反思等工具使用相关能力上均有提升。详见 示例。
InternLM2.5-7B-Chat-1M
InternLM2.5-7B-Chat-1M 是 InternLM2.5-7B-Chat 的百万级长上下文版本。由于 huggingface Transformers 不直接支持百万级上下文推理，我们推荐使用 LMDeploy。下文也展示了基于 huggingface Transformers 的常规使用方法。
LMDeploy
LMDeploy 是由 MMRazor 和 MMDeploy 团队开发的大语言模型压缩、部署与服务工具包。
以下是 1M 长上下文推理的示例。注意：1M 上下文长度需 4 张 A100-80G 显卡！
pip install lmdeploy你可以通过以下 Python 代码在本地运行批量推理：
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
backend_config = TurbomindEngineConfig(
rope_scaling_factor=2.5,
session_len=1048576, # 1M context length
max_batch_size=1,
cache_max_entry_count=0.7,
tp=4) # 4xA100-80G.
pipe = pipeline('internlm/internlm2_5-7b-chat-1m', backend_config=backend_config)
prompt = 'Use a long prompt to replace this sentence'
response = pipe(prompt)
print(response)更多细节请参见 LMDeploy 文档
从 openMind 导入
由于 openMind 不支持 1M 长上下文，因此这里仅展示非长上下文的使用方法。 若要通过 openMind 加载 InternLM2 7B Chat 模型，请使用以下代码：
import torch
from openmind import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("AI-Research/internlm2_5-7b-chat-1m", trust_remote_code=True)
# Set `torch_dtype=torch.float16` to load model in float16, otherwise it will be loaded as float32 and cause OOM Error.
model = AutoModelForCausalLM.from_pretrained("AI-Research/internlm2_5-7b-chat-1m", torch_dtype=torch.float16, trust_remote_code=True).npu()
model = model.eval()
response, history = model.chat(tokenizer, "hello", history=[])
print(response)
# Hello! How can I help you today?
response, history = model.chat(tokenizer, "please provide three suggestions about time management", history=history)
print(response)可以使用 stream_chat 实现响应的流式输出：
import torch
from openmind import AutoModelForCausalLM, AutoTokenizer
model_path = "AI-Research/internlm2_5-7b-chat-1m"
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dtype=torch.float16, trust_remote_code=True).npu()
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = model.eval()
length = 0
for response, history in model.stream_chat(tokenizer, "Hello", history=[]):
print(response[length:], flush=True, end="")
length = len(response)vLLM
使用 vLLM>=0.3.2 启动与 OpenAI 兼容的服务器：
pip install vllmpython -m vllm.entrypoints.openai.api_server --model internlm/internlm2_5-7b-chat-1m --served-model-name internlm2_5-7b-chat-1m --trust-remote-code如果遇到 OOM 问题，请尝试减小 --max-model-len 或增大 --tensor-parallel-size。
之后，您可以向服务器发送聊天请求：
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "internlm2_5-7b-chat-1m",
"messages": [
{"role": "system", "content": "You are a helpful assistant."},
{"role": "user", "content": "Introduce deep learning to me."}
]
}'更多详情请参见 vLLM 文档
开源许可
代码采用 Apache-2.0 许可协议，模型权重完全开放用于学术研究，同时允许免费商业使用。如需申请商业许可，请填写 申请表（英文）/申请表（中文）。其他问题或合作事宜，请联系 internlm@pjlab.org.cn。
引用
@misc{cai2024internlm2,
title={InternLM2 Technical Report},
author={Zheng Cai and Maosong Cao and Haojiong Chen and Kai Chen and Keyu Chen and Xin Chen and Xun Chen and Zehui Chen and Zhi Chen and Pei Chu and Xiaoyi Dong and Haodong Duan and Qi Fan and Zhaoye Fei and Yang Gao and Jiaye Ge and Chenya Gu and Yuzhe Gu and Tao Gui and Aijia Guo and Qipeng Guo and Conghui He and Yingfan Hu and Ting Huang and Tao Jiang and Penglong Jiao and Zhenjiang Jin and Zhikai Lei and Jiaxing Li and Jingwen Li and Linyang Li and Shuaibin Li and Wei Li and Yining Li and Hongwei Liu and Jiangning Liu and Jiawei Hong and Kaiwen Liu and Kuikun Liu and Xiaoran Liu and Chengqi Lv and Haijun Lv and Kai Lv and Li Ma and Runyuan Ma and Zerun Ma and Wenchang Ning and Linke Ouyang and Jiantao Qiu and Yuan Qu and Fukai Shang and Yunfan Shao and Demin Song and Zifan Song and Zhihao Sui and Peng Sun and Yu Sun and Huanze Tang and Bin Wang and Guoteng Wang and Jiaqi Wang and Jiayu Wang and Rui Wang and Yudong Wang and Ziyi Wang and Xingjian Wei and Qizhen Weng and Fan Wu and Yingtong Xiong and Chao Xu and Ruiliang Xu and Hang Yan and Yirong Yan and Xiaogui Yang and Haochen Ye and Huaiyuan Ying and Jia Yu and Jing Yu and Yuhang Zang and Chuyu Zhang and Li Zhang and Pan Zhang and Peng Zhang and Ruijie Zhang and Shuo Zhang and Songyang Zhang and Wenjian Zhang and Wenwei Zhang and Xingcheng Zhang and Xinyue Zhang and Hui Zhao and Qian Zhao and Xiaomeng Zhao and Fengzhe Zhou and Zaida Zhou and Jingming Zhuo and Yicheng Zou and Xipeng Qiu and Yu Qiao and Dahua Lin},
year={2024},
eprint={2403.17297},
archivePrefix={arXiv},
primaryClass={cs.CL}
}简介
InternLM2.5，即书生·浦语大模型第2.5代，开源了面向实用场景的70亿参数基础模型与对话模型（InternLM2.5-7B-Chat）。该模型具有以下特点：
- 卓越的推理性能：在数学推理方面取得了同量级模型最优精度，超越了Llama3和Gemma2-9B。
- 有效支持百万字超长上下文：模型在1百万字长输入中几乎完美地实现长文“大海捞针”，而且在LongBench等长文任务中的表现也达到开源模型中的领先水平。可以通过LMDeploy尝试百万字超长上下文推理。
- 工具调用能力整体升级：InternLM2.5支持从上百个网页搜集有效信息进行分析推理，相关实现将于近期开源到Lagent。InternLM2.5具有更强和更具泛化性的指令理解、工具筛选与结果反思等能力，新版模型可以更可靠地支持复杂智能体的搭建，支持对工具进行有效的多轮调用，完成较复杂的任务。可以查看更多样例。
InternLM2.5-7B-Chat-1M
InternLM2.5-7B-Chat-1M支持1百万字超长上下文推理，且性能和InternLM2.5-7B-Chat相当。考虑到huggingface Transformers不直接支持1M上下文推理，我们优先推荐使用lmdeploy进行百万字超长上下文推理演示。在非超长上下文推理的情况下，你仍然可以使用huggingface transformers，参考下面的样例代码。
LMDeploy
LMDeploy由MMDeploy和MMRazor团队联合开发，是涵盖了LLM任务的全套轻量化、部署和服务解决方案。
以下是一个1M上下文推理的例子。注意：1M上下文需要4xA100-80G！
pip install lmdeploy你可以使用以下 Python 代码进行本地批量推理:
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
backend_config = TurbomindEngineConfig(
rope_scaling_factor=2.5,
session_len=1048576, # 1M context length
max_batch_size=1,
cache_max_entry_count=0.7,
tp=4) # 4xA100-80G.
pipe = pipeline('internlm/internlm2_5-7b-chat-1m', backend_config=backend_config)
prompt = 'Use a long prompt to replace this sentence'
response = pipe(prompt)
print(response)通过 openMind 加载
由于 openMind 无法支持 1M 长上下文推理，这里仅演示非长文本的用法。
通过以下的代码加载 InternLM2.5 7B Chat 1M 模型
import torch
from openmind import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("AI-Research/internlm2_5-7b-chat-1m", trust_remote_code=True)
# `torch_dtype=torch.float16` 可以令模型以 float16 精度加载，否则 transformers 会将模型加载为 float32，导致显存不足
model = AutoModelForCausalLM.from_pretrained("AI-Research/internlm2_5-7b-chat-1m", torch_dtype=torch.float16, trust_remote_code=True).npu()
model = model.eval()
response, history = model.chat(tokenizer, "你好", history=[])
print(response)
# 你好！有什么我可以帮助你的吗？
response, history = model.chat(tokenizer, "请提供三个管理时间的建议。", history=history)
print(response)如果想进行流式生成，则可以使用 stream_chat 接口：
import torch
from openmind import AutoModelForCausalLM, AutoTokenizer
model_path = "AI-Research/internlm2_5-7b-chat-1m"
model = AutoModelForCausalLM.from_pretrained(model_path, torch_dype=torch.float16, trust_remote_code=True).npu()
tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
model = model.eval()
length = 0
for response, history in model.stream_chat(tokenizer, "你好", history=[]):
print(response[length:], flush=True, end="")
length = len(response)vLLM
使用vLLM>=0.3.2启动兼容 OpenAI API 的服务:
pip install vllmpython -m vllm.entrypoints.openai.api_server --model internlm/internlm2_5-7b-chat-1m --trust-remote-code如果你遇到 OOM，请减小 --max-model-len 或增加 --tensor-parallel-size 参数。
然后你可以向服务端发起一个聊天请求：
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "internlm2_5-7b-chat-1m",
"messages": [
{"role": "system", "content": "你是个友善的AI助手。"},
{"role": "user", "content": "介绍一下深度学习。"}
]
}'更多信息请查看 vLLM 文档
开源许可证
本仓库的代码依照 Apache-2.0 协议开源。模型权重对学术研究完全开放，也可申请免费的商业使用授权（申请表）。其他问题与合作请联系 internlm@pjlab.org.cn。
引用
@misc{cai2024internlm2,
title={InternLM2 Technical Report},
author={Zheng Cai and Maosong Cao and Haojiong Chen and Kai Chen and Keyu Chen and Xin Chen and Xun Chen and Zehui Chen and Zhi Chen and Pei Chu and Xiaoyi Dong and Haodong Duan and Qi Fan and Zhaoye Fei and Yang Gao and Jiaye Ge and Chenya Gu and Yuzhe Gu and Tao Gui and Aijia Guo and Qipeng Guo and Conghui He and Yingfan Hu and Ting Huang and Tao Jiang and Penglong Jiao and Zhenjiang Jin and Zhikai Lei and Jiaxing Li and Jingwen Li and Linyang Li and Shuaibin Li and Wei Li and Yining Li and Hongwei Liu and Jiangning Liu and Jiawei Hong and Kaiwen Liu and Kuikun Liu and Xiaoran Liu and Chengqi Lv and Haijun Lv and Kai Lv and Li Ma and Runyuan Ma and Zerun Ma and Wenchang Ning and Linke Ouyang and Jiantao Qiu and Yuan Qu and Fukai Shang and Yunfan Shao and Demin Song and Zifan Song and Zhihao Sui and Peng Sun and Yu Sun and Huanze Tang and Bin Wang and Guoteng Wang and Jiaqi Wang and Jiayu Wang and Rui Wang and Yudong Wang and Ziyi Wang and Xingjian Wei and Qizhen Weng and Fan Wu and Yingtong Xiong and Chao Xu and Ruiliang Xu and Hang Yan and Yirong Yan and Xiaogui Yang and Haochen Ye and Huaiyuan Ying and Jia Yu and Jing Yu and Yuhang Zang and Chuyu Zhang and Li Zhang and Pan Zhang and Peng Zhang and Ruijie Zhang and Shuo Zhang and Songyang Zhang and Wenjian Zhang and Wenwei Zhang and Xingcheng Zhang and Xinyue Zhang and Hui Zhao and Qian Zhao and Xiaomeng Zhao and Fengzhe Zhou and Zaida Zhou and Jingming Zhuo and Yicheng Zou and Xipeng Qiu and Yu Qiao and Dahua Lin},
year={2024},
eprint={2403.17297},
archivePrefix={arXiv},
primaryClass={cs.CL}
}