weixin_62994174/Belle-whisper-large-v3-zh
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

Belle-whisper-large-v3-zh (Ascend NPU Adaptation)

Belle-whisper-large-v3-zh is a Chinese-enhanced Whisper large-v3 model fine-tuned by the BELLE team. This repository provides the Ascend NPU adaptation for inference on Huawei Ascend hardware (Ascend 910B2).

Validation Status: PASS | Encoder Precision PASS | Decoder Precision PASS | E2E Transcription PASS

Model Information

ItemDetails
ModelBelle-whisper-large-v3-zh
ArchitectureWhisperForConditionalGeneration (Encoder-Decoder)
Original SourceXorbits/Belle-whisper-large-v3-zh (ModelScope)
Parameters~1.5B
d_model1280
Encoder Layers32
Decoder Layers32
Attention Heads20
FFN Dimension5120
Vocabulary51866
Mel Bins128
PrecisionFP32
LanguageChinese (zh)
TaskAutomatic Speech Recognition (ASR)

Environment

ComponentVersion
NPUAscend 910B2 (2x, 64GB HBM each)
CANN8.5.1
PyTorch2.9.0
torch_npu2.9.0.post1+gitee7ba04
transformers>= 4.34.0
Python3.11

Quick Start

1. Download Model

pip install modelscope
modelscope download --model Xorbits/Belle-whisper-large-v3-zh \
    --local_dir ./models/Belle-whisper-large-v3-zh \
    --exclude "pytorch_model.bin"

2. Install Dependencies

pip install torch torch_npu transformers librosa soundfile

3. Run Inference

# Single audio file transcription
python inference.py --audio /path/to/audio.wav

# Benchmark mode
python inference.py --benchmark --num-runs 50

4. Run Full Evaluation

python evaluate.py --num-runs 30 --warmup 5

Performance Benchmarks

Tested on Ascend 910B2, FP32 precision, single NPU (npu:0).

Fixed-Length Audio (5 seconds synthetic)

MetricValue
Avg Latency0.2089 s
P50 Latency0.2085 s
P90 Latency0.2115 s
P99 Latency0.2156 s
Std Dev0.0023 s
Throughput9.58 tokens/s
Peak Memory6526 MB

Variable-Length Audio Scaling

Audio DurationInput FramesAvg LatencyOutput Tokens
3s30000.216 s2.0
5s30000.223 s2.0
10s30000.212 s2.0
15s30000.340 s5.0
30s30000.817 s15.0

Note: Synthetic tone audio produces minimal output tokens. Real speech would produce more tokens proportionally to audio content. Encoder runs in ~50ms (fixed cost), decoder generates ~10 tok/s.

Accuracy Validation

NPU (FP32) outputs compared against CPU (FP32) reference on identical inputs.

Encoder Hidden State Precision

MetricValueThresholdStatus
Cosine Similarity0.9998005> 0.999PASS
SNR33.95 dB> 30 dBPASS
Max Absolute Error1.978--
Mean Absolute Error0.007--
Relative Error1.63%--
Frobenius Relative0.0201--

Decoder First-Step Logit Precision (Isolated)

MetricValueThresholdStatus
Cosine Similarity0.9999263> 0.999PASS
SNR29.95 dB> 20 dBPASS
Top-1 Overlap100%--
Top-5 Overlap100%>= 80%PASS
Top-10 Overlap100%--
Top-100 Overlap99%--
Relative Error3.33%< 5%PASS

End-to-End Transcription

MetricValueThresholdStatus
Token Match Accuracy100%> 90%PASS
Exact Transcript MatchTrue-PASS
CPU Output嗯嗯--
NPU Output嗯嗯--

Overall Result: PASS

All three validation stages pass. The encoder hidden states achieve cosine similarity > 0.9998. The decoder logits achieve cosine similarity > 0.9999. End-to-end transcription produces bit-identical results.

Project Structure

Belle-whisper-large-v3-zh/
├── README.md                  # This document
├── inference.py               # NPU inference script (single audio / benchmark)
├── evaluate.py                # Precision & performance evaluation suite
├── eval_report.json           # Full evaluation results (JSON)
└── test_audio/                # Test audio generation script
    └── generate_test.py

Architecture Details

The Whisper encoder-decoder pipeline on NPU:

  1. Audio Preprocessing: Mel spectrogram extraction (CPU)
  2. Encoder: 32-layer Transformer encoder (NPU) — ~50ms fixed cost
  3. Decoder: 32-layer Transformer decoder with cross-attention (NPU) — ~10 tok/s
  4. Token Decoding: Greedy/beam search with forced Chinese decoder IDs

Known Limitations

  1. FP32 only; FP16 may cause accuracy degradation due to NPU GELU differences.
  2. Single-audio inference (batch=1); multi-audio batching not yet optimized.
  3. Memory footprint ~6.5GB in FP32 on a single NPU.
  4. The forced decoder prompt IDs are required for Chinese transcription.

License

This model adaptation is based on Xorbits/Belle-whisper-large-v3-zh on ModelScope. Please refer to the original model for license terms.

Tags

#NPU #Ascend #Whisper #ASR #Speech-Recognition #Hardware-NPU