weixin_62994174/XTTS-v2
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

XTTS-v2 for Ascend NPU

ⓍTTS is a voice generation model that lets you clone voices into different languages using just a quick 6-second audio clip. This repository provides the Ascend NPU adapted version of XTTS-v2, enabling fast inference on Huawei Ascend 910 series NPUs.

Model Overview

  • Model: XTTS-v2 by Coqui AI
  • Task: Text-to-Speech (TTS) with Voice Cloning
  • Hardware: Huawei Ascend 910B NPU (verified)
  • Framework: PyTorch 2.9.0 + torch_npu 2.9.0
  • Supported Languages: 16 languages (en, es, fr, de, it, pt, pl, tr, ru, nl, cs, ar, zh-cn, hu, ko, ja)
  • Sampling Rate: 24kHz

NPU Adaptation Details

This model has been verified to run on Ascend NPU with the following adaptations:

ComponentAdaptation
Main GPT ModelRuns natively on NPU
HiFi-GAN DecoderRuns natively on NPU
Speaker EncoderRuns on CPU (NPU lacks complex64 abs() support for STFT)
Audio LoadingUses soundfile backend (avoids torchcodec/libnvrtc)

NPU Compatibility Fixes

  1. torch.load safe_globals: Added XTTS configuration classes to torch serialization allowlist (PyTorch 2.6+ weights_only=True default)
  2. Audio decoding: Patched to use soundfile instead of torchcodec (libnvrtc.so not available on Ascend)
  3. Speaker encoder: Moved to CPU to avoid unsupported abs() on complex64 tensors from torch.stft

Performance

Evaluated on Huawei Ascend 910B NPU:

LanguageText LengthAudio DurationInference TimeRTF
English (short)74 chars5.84s2.91s0.50x
English (medium)97 chars4.68s2.55s0.55x
Chinese (zh-cn)20 chars4.54s2.31s0.51x
French68 chars4.64s2.41s0.52x
German75 chars5.24s2.60s0.50x
  • Average RTF: 0.51x (faster than real-time)
  • Average Inference Time: 2.56s

RTF < 1.0 means the model generates audio faster than real-time (e.g., 0.5x means 2x faster than playback speed).

Accuracy Validation

The NPU adaptation accuracy was validated on deterministic model components:

ComponentMetricValueStatus
Speaker EmbeddingMAE7.48e-05✅
Speaker EmbeddingCosine Similarity0.999998✅

Accuracy: Speaker embedding error < 0.01%, confirming NPU numerical consistency with CPU baseline.

Note: Full audio output comparison is not meaningful for this generative model due to inherent GPT sampling randomness (temperature-based generation produces different audio each run).

Installation

# Install PyTorch NPU
pip install torch torch_npu torchaudio

# Install TTS dependencies
pip install TTS soundfile scipy

# Clone this repository
git clone https://atomgit.com/weixin_62994174/XTTS-v2.git
cd XTTS-v2

Quick Start

# English TTS with default speaker
python inference.py --text "Hello world, this is a test." --language en

# Chinese TTS
python inference.py --text "你好,这是语音合成测试。" --language zh-cn

# Custom speaker voice cloning
python inference.py --text "Your text here" \
    --speaker_wav /path/to/speaker_6sec.wav \
    --language en \
    --output output.wav

Python API

from inference import load_model, synthesize

model, config = load_model(device="npu")
wav, elapsed, duration, rtf = synthesize(
    model, config,
    text="Hello, this is Ascend NPU text-to-speech!",
    speaker_wav="models/samples/en_sample.wav",
    language="en",
    output_path="output.wav",
)
print(f"RTF: {rtf:.2f}x")

Project Structure

XTTS-v2/
├── inference.py                 # Main NPU inference script
├── evaluation/
│   ├── evaluate.py              # Performance & accuracy evaluation
│   └── accuracy_deterministic.py # Deterministic component validation
├── models/                      # Model weights (from ModelScope)
│   ├── model.pth                # Main XTTS-v2 checkpoint
│   ├── dvae.pth                 # Discrete VAE checkpoint
│   ├── config.json              # Model configuration
│   ├── vocab.json               # Tokenizer vocabulary
│   └── samples/                 # Reference speaker samples
├── output/                      # Generated audio outputs
├── smoke_test.py                # Quick NPU verification test
└── README.md                    # This file

Evaluation

Run the full evaluation suite:

# Performance benchmark only
python evaluation/evaluate.py --perf-only

# Deterministic accuracy validation
python evaluation/accuracy_deterministic.py

Known Issues

  1. Speaker encoder on CPU: The torch.stft operation produces complex tensors; Ascend NPU does not support abs() on complex64 dtype. Speaker encoder runs on CPU as a workaround.
  2. torchcodec unavailable: libnvrtc.so is not installed on this Ascend environment. Audio loading uses soundfile as fallback.
  3. Generation non-determinism: The GPT-based text-to-audio generation uses sampling (temperature/top-k/top-p), producing different audio each run. This is expected behavior, not an NPU-specific issue.

License

This model is licensed under the Coqui Public Model License (CPML).

Citation

@misc{xtts-v2,
  author = {Coqui AI},
  title = {XTTS-v2: Multilingual Voice Cloning},
  year = {2024},
  publisher = {ModelScope},
  url = {https://www.modelscope.cn/models/AI-ModelScope/XTTS-v2}
}