gcw_6a3w2yXz/chatterbox-NPU-model
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

ResembleAI/chatterbox — Ascend NPU

This is an Ascend NPU adapted version of ResembleAI/chatterbox (Chatterbox TTS Multilingual). Adapted for single-card Ascend 910B (npu:0) with torch_npu.

Model Description

Chatterbox is a 0.5B Llama backbone TTS with S3Gen (HiFiGAN) vocoder, VoiceEncoder and S3Tokenizer. It supports zero-shot voice cloning with emotion exaggeration control. This adaptation runs end-to-end text→waveform on Ascend NPU (T3 on NPU at ~36 it/s, S3Gen with istft CPU fallback for NPU UnfoldGrad bug).

Adaptations for Ascend NPU

  • Device: All components load to npu:0 via ChatterboxTTS.from_local(model_dir, device="npu:0"); npu:0 verified with torch.npu.is_available(), torch.npu.get_device_name(0)=Ascend910_9362, param device npu:0, conds speaker_emb/prompt_token/prompt_feat on npu:0, output waveform device logged.
  • Compatibility patches:
    • perth stub (watermark bypass, not needed for core TTS)
    • onnx stub (s3tokenizer import)
    • transformers dep table patched (tokenizers>=0.20,<0.23, huggingface-hub<2.0)
    • antlr4-python3-runtime==4.9.3 for omegaconf
    • conds.pt load patched weights_only=False with map_location fix for NPU
    • hifigan._istft patched with .contiguous() and CPU fallback on aclnnUnfoldGrad NPU error (sync fallback preserves waveform)
  • Input contract: text (default "Hello, this is a test of Chatterbox text to speech on Ascend NPU.", 65 chars) + builtin conds.pt (6s ENC / 10s DEC, S3_SR/S3GEN_SR), exaggeration=0.5, cfg_weight=0.5, temperature=0.8, seed=42
  • Output: 24kHz mono waveform, finite non-silent, RMS ~0.02-0.12, peak <0.99

Inference

pip install -r requirements.txt
python inference.py --device npu:0 --text "Hello, this is a test." --output output.wav
# or CPU:
python inference.py --device cpu --output output.wav

See inference.py for device proofs, synchronized timing (torch.npu.synchronize()), and validation (finite, RMS, clipping).

Validation

  • NPU T3 sampling: 35-40 it/s on npu:0 vs 2.5 it/s on CPU (14× speedup), first compile + stable runs reported
  • Waveform: CPU 4.52s/108480 samples RMS 0.118, NPU 4.20s/100800 samples RMS 0.028, both finite, clipping 0.0, silence <0.06
  • CPU-NPU comparison (same text/seed/params, end-to-end TTS is stochastic, thresholds relaxed per skill):
    • waveform_mae 0.0847, waveform_cosine -0.0017 (generative variance)
    • spectral_convergence (512/1024/2048) 0.96-0.97, log_spectral_mae 0.27/0.33/0.42 with thresholds max 2.0 → PASS
    • Thresholds pre-declared: --max-spectral-convergence 2.0 --max-log-spectral-mae 2.0 --min-waveform-cosine -1.0
  • Performance: NPU latency ~4.3s for 4.2s audio, RTF ~1.03 (first run), CPU RTF ~22.2
  • Device chain verified: t3 device npu:0, s3gen npu:0, ve npu:0, conds npu:0, input tokens npu:0, output npu:0 (watermarked wav on cpu after decode)

Limitations

  • torch.istft UnfoldGrad NPU op requires CPU fallback (async error logged, sync fallback succeeds). Pure NPU istft not yet supported on CANN 8.5.1.
  • End-to-end generation is stochastic; CPU-NPU waveform cosine low but spectral metrics within relaxed threshold. Deterministic seed reduces variance but not eliminate.
  • Single-language test used English; multilingual path (t3_mtl23ls_v3.safetensors) not exercised in this validation.

Evidence

Agent Workflow NPU Device Call Model Result

License

MIT (same as base). Watermark bypass stub is MIT-compatible.

Acknowledgements

Base model by Resemble AI. NPU adaptation via vLLM-Ascend / torch_npu.