CosyVoice-300M: 基于昇腾 NPU 的高质量 Zero-Shot 语音合成

# 安装依赖
pip install pyworld diffusers lightning soundfile onnx onnxruntime
# NPU 环境依赖（已安装）
pip install torch-npu

pretrained_models/CosyVoice-300M/
├── cosyvoice.yaml               # 模型配置
├── configuration.json            # 框架配置
├── README.md                     # 官方 README
├── llm.pt                        # LLM 权重 (~1.2GB)
├── llm.llm.fp16.zip              # LLM JIT 加速 (~810MB)
├── llm.llm.fp32.zip              # LLM JIT 加速 (~810MB)
├── llm.text_encoder.fp16.zip     # Text Encoder JIT 加速
├── llm.text_encoder.fp32.zip     # Text Encoder JIT 加速
├── flow.pt                       # 流模型权重 (~420MB)
├── flow.decoder.estimator.fp32.onnx  # 流解码器 ONNX (~1.1GB)
├── flow.encoder.fp16.zip         # Flow Encoder JIT 加速
├── flow.encoder.fp32.zip         # Flow Encoder JIT 加速
├── hift.pt                       # HiFT 声码器 (~82MB)
├── campplus.onnx                 # 说话人嵌入提取 (~27MB)
├── speech_tokenizer_v1.onnx      # 语音 tokenizer (~925MB)
└── asset/
    ├── zero_shot_prompt.wav      # Zero-shot 参考音频
    ├── cross_lingual_prompt.wav  # 跨语言参考音频
    └── dingding.png

# Zero-Shot 语音克隆 (NPU)
python3 inference.py \
  --model_dir pretrained_models/CosyVoice-300M \
  --mode zero_shot \
  --device npu \
  --tts_text "收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。" \
  --prompt_text "希望你以后能够做的比我还好呦。" \
  --prompt_wav pretrained_models/CosyVoice-300M/asset/zero_shot_prompt.wav \
  --output_dir ./outputs

# 跨语言合成 (NPU)
python3 inference.py \
  --model_dir pretrained_models/CosyVoice-300M \
  --mode cross_lingual \
  --device npu \
  --tts_text "你好，今天天气很不错，我们一起去公园散步吧。" \
  --prompt_wav pretrained_models/CosyVoice-300M/asset/cross_lingual_prompt.wav \
  --output_dir ./outputs

# CPU 基线测试
python3 inference.py \
  --model_dir pretrained_models/CosyVoice-300M \
  --mode zero_shot \
  --device cpu \
  --tts_text "收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。" \
  --prompt_text "希望你以后能够做的比我还好呦。" \
  --prompt_wav pretrained_models/CosyVoice-300M/asset/zero_shot_prompt.wav \
  --output_dir ./outputs

# 性能基准测试 (NPU)
python3 inference.py \
  --model_dir pretrained_models/CosyVoice-300M \
  --mode zero_shot \
  --device npu \
  --tts_text "收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。" \
  --prompt_text "希望你以后能够做的比我还好呦。" \
  --prompt_wav pretrained_models/CosyVoice-300M/asset/zero_shot_prompt.wav \
  --output_dir ./outputs/bench \
  --benchmark --warmup 1 --iterations 3

{
  "sample_count": 222076,
  "sample_rate": 24000,
  "metrics": {
    "mse": 0.000000,
    "mae": 0.000000,
    "max_diff": 0.000000,
    "relative_mae": 0.000000,
    "cosine_similarity": 1.000000,
    "snr_db": "Infinity",
    "spectral_correlation": 1.000000,
    "mfcc_similarity": 1.000000
  }
}

[Info] Using device: npu:0
[Info] Accelerator available: True
[Info] Model loaded in 17.70s
[Info] Model sample rate: 22050
2026-05-14 08:41:10,289 INFO yield speech len 9.253, rtf 1.836
[Info] Inference completed:
  Device: npu
  Mode: zero_shot
  Output files: 1
  Audio duration: 9.253s
  Inference time: 18.061s
  RTF: 1.9518

[Info] Using device: cpu
[Info] Model loaded in 13.47s
[Info] Model sample rate: 22050
2026-05-14 08:43:03,886 INFO yield speech len 9.253, rtf 1.866
[Info] Inference completed:
  Device: cpu
  Mode: zero_shot
  Audio duration: 9.253s
  Inference time: 18.440s
  RTF: 1.9928

[Info] Using device: npu:0
[Info] Model loaded in 14.21s
[Info] Model sample rate: 22050
2026-05-14 08:46:05,762 INFO yield speech len 4.470, rtf 1.966
[Info] Inference completed:
  Device: npu
  Mode: cross_lingual
  Audio duration: 4.470s
  Inference time: 13.078s
  RTF: 2.9259

=== Evaluation Results ===
Samples: 222076
  mse: 0.000000
  mae: 0.000000
  max_diff: 0.000000
  relative_mae: 0.000000
  cosine_similarity: 1.000000
  snr_db: inf
  spectral_correlation: 1.000000
  mfcc_similarity: 1.000000

=== Pass Criteria ===
  relative_mae < 1%: PASS
  cosine_similarity > 0.99: PASS
  snr > 40dB: PASS
  spectral_correlation > 0.99: PASS

✅ NPU accuracy validation PASSED (error < 1%)

[Benchmark] Running 3 iterations...
  Iter 1/3: RTF=2.2652, Time=24.326s
  Iter 2/3: RTF=2.3107, Time=22.266s
  Iter 3/3: RTF=2.2823, Time=25.040s

Benchmark Summary:
{
  "rtf_mean": 2.286,
  "rtf_std": 0.019,
  "rtf_min": 2.265,
  "rtf_max": 2.311,
  "time_mean_sec": 23.877
}