MooER-MTL-5K 摩耳多任务语音识别/翻译模型 (昇腾 NPU 适配版)

输入音频 (16kHz WAV)
    │
    ▼
Fbank 特征提取 (80维 filter bank)
    │
    ▼
Paraformer 编码器 ← CMVN 归一化
    │
    ▼
Adapter 投影层 (将编码器输出映射到 LLM 空间)
    │
    ▼
Qwen2-7B-Instruct LLM (带 LoRA 权重)
    │
    ▼
文本输出 (ASR: 中文 / AST: 英文)

# 安装 MooER 推理代码
git clone https://github.com/MooreThreads/MooER.git /tmp/MooER

# 安装 CANN 昇腾驱动 (如未安装)
# 请参考华为官方 CANN 安装文档

pip install -r requirements.txt

# 下载 MooER-MTL-5K 模型
git clone https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K.git

# 下载 Qwen2-7B-Instruct 基座模型 (约 15GB，必需)
git clone https://huggingface.co/Qwen/Qwen2-7B-Instruct
# 或
git clone https://modelscope.cn/models/qwen/Qwen2-7B-Instruct

# 一键下载脚本 (推荐)
# 从 ModelScope 下载 MooER-MTL-5K 完整模型
git clone https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K.git /tmp/MooER-MTL-5K-weights
cp -r /tmp/MooER-MTL-5K-weights/model/* ./model/

# 或手动下载单个文件:
# 1. 编码器权重
wget https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K/resolve/master/model/paraformer_encoder/paraformer-encoder.pth -O model/paraformer_encoder/paraformer-encoder.pth

# 2. ASR LoRA 权重
wget https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K/resolve/master/model/asr/lora_weights/adapter_model.bin -O model/asr/lora_weights/adapter_model.bin

# 3. Qwen2-7B-Instruct 基座模型 (~15GB)
git clone https://modelscope.cn/models/qwen/Qwen2-7B-Instruct model/Qwen2-7B-Instruct

MooER-MTL-5K/
├── inference.py                 # NPU 推理脚本
├── compare_cpu_npu.py           # CPU vs NPU 精度对比
├── requirements.txt             # Python 依赖
├── test_audio.wav               # 测试音频
├── README.md                    # 本文件
├── model/
│   ├── configuration.json              # 模型配置
│   ├── paraformer_encoder/
│   │   ├── paraformer-encoder.pth      # 编码器权重 (~80MB, 需下载)
│   │   └── am.mvn                      # CMVN 归一化参数 ✓
│   ├── Qwen2-7B-Instruct/             # Qwen2-7B-Instruct (~15GB, 需下载)
│   ├── asr/
│   │   ├── adapter_project.pt          # ASR 投影层权重 (37MB) ✓
│   │   └── lora_weights/
│   │       ├── adapter_model.bin       # ASR LoRA 权重 (~617MB, 需下载)
│   │       └── adapter_config.json     # LoRA 配置 ✓
│   ├── ast/
│   │   ├── adapter_project.pt          # AST 投影层权重 (37MB) ✓
│   │   └── lora_weights/
│   │       ├── adapter_model.bin       # AST LoRA 权重 (需下载)
│   │       └── adapter_config.json     # LoRA 配置 ✓
│   ├── asr_ast_mtl/
│   │   ├── adapter_project.pt          # 多任务投影层权重 (37MB) ✓
│   │   └── lora_weights/
│   │       ├── adapter_model.bin       # 多任务 LoRA 权重 (需下载)
│   │       └── adapter_config.json     # LoRA 配置 ✓
│   ├── assets/
│   │   ├── MTLogo.png                  # 模型 Logo ✓
│   │   └── framework.png               # 架构图 ✓
│   ├── LICENSE                         # 许可证 ✓
│   └── README.md                       # 原始模型 README ✓
└── ...

# Qwen2-7B-Instruct (必需，基座模型)
git clone https://modelscope.cn/models/qwen/Qwen2-7B-Instruct model/Qwen2-7B-Instruct
# 或: git clone https://huggingface.co/Qwen/Qwen2-7B-Instruct model/Qwen2-7B-Instruct

# MooER-MTL-5K 完整权重 (包含编码器、LoRA 等)
git clone https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K.git
# 然后将 model/ 目录下的文件复制过来

git clone https://github.com/MooreThreads/MooER.git /tmp/MooER

# Step 1: 克隆 MooER 推理代码
git clone https://github.com/MooreThreads/MooER.git /tmp/MooER

# Step 2: 安装 Python 依赖
pip install -r requirements.txt

# Step 3: 下载 Qwen2-7B-Instruct 基座模型
git clone https://modelscope.cn/models/qwen/Qwen2-7B-Instruct model/Qwen2-7B-Instruct

# Step 4: 下载 MooER-MTL-5K 模型完整权重
git clone https://modelscope.cn/models/MooreThreadsSpeech/MooER-MTL-5K.git /tmp/MooER-MTL-5K-weights
cp -r /tmp/MooER-MTL-5K-weights/model/* ./model/

# Step 5: 运行推理
python inference.py --wav_path test_audio.wav --task asr

# NPU 推理 (自动检测)
python inference.py --wav_path test_audio.wav --task asr --device auto

# 指定 NPU
python inference.py --wav_path test_audio.wav --task asr --device npu

# CPU 推理
python inference.py --wav_path test_audio.wav --task asr --device cpu

# NPU 推理
python inference.py --wav_path test_audio.wav --task ast --device npu

# CPU 推理
python inference.py --wav_path test_audio.wav --task ast --device cpu

# AST 任务
python inference.py \
    --task ast \
    --adapter_path model/ast/adapter_project.pt \
    --lora_dir model/ast/lora_weights \
    --wav_path test_audio.wav

# ASR+AST 多任务
python inference.py \
    --task asr \
    --adapter_path model/asr_ast_mtl/adapter_project.pt \
    --lora_dir model/asr_ast_mtl/lora_weights \
    --wav_path test_audio.wav

import torch
import torchaudio
import soundfile as sf

# 1. 设备选择
device = 'npu:0' if torch.npu.is_available() else 'cpu'

# 2. 加载模型
model_config = asr_config.ModelConfig()
model_config.llm_path = args.llm_path
model_config.encoder_path = args.encoder_path
model_config.adapter_path = args.adapter_path
model_config.cmvn_path = args.cmvn_path
model_config.lora_dir = args.lora_dir
model_config.is_inference = True

model, tokenizer = mooer_model.init_model(model_config=model_config)
model.to(device)
model.eval()

# 3. 加载音频并进行 Fbank 特征提取
waveform, sample_rate = sf.read(wav_path)
feats = torchaudio.compliance.kaldi.fbank(
    audio_tensor * (1 << 15),
    num_mel_bins=80, frame_length=25, frame_shift=10
)

# 4. 编码器推理
encoder_out, _ = model.encoder(feats, feats_len)

# 5. 投影到 LLM 空间
projected = model.adapter(encoder_out)

# 6. LLM 生成
generated_ids = model.llm.generate(
    inputs=input_ids,
    encoder_outputs=projected,
    max_new_tokens=200
)

# 7. 解码输出
response = tokenizer.decode(generated_ids[0], skip_special_tokens=True)

# CPU 和 NPU 精度对比
python compare_cpu_npu.py

# NPU 推理
python3 inference.py --device npu

# CPU 推理
python3 inference.py --device cpu