m0_74196153/hy-mt-1.5-ascend
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

HY-MT1.5 Ascend NPU

Tencent HunYuan HY-MT1.5 Multilingual Translation Model adapted for Huawei Ascend NPU (via vLLM-Ascend)

Overview

This repository provides the complete adaptation of Tencent HunYuan HY-MT1.5-1.8B — a multilingual neural machine translation model — to run on Huawei Ascend NPU (Atlas 800 A2, Ascend 910B) using the vLLM-Ascend inference engine.

ModelParametersTypeArchitectureStatus
HY-MT1.5-1.8B1.79BDenseHunYuanDenseV1ForCausalLM✅
HY-MT1.5-7B7BMoEHunYuanMoEV1ForCausalLM⚠️ 下载待补全
HY-MT1.5-1.8B-FP81.79BDense FP8W8A16暂不支持❌
HY-MT1.5-1.8B-GPTQ-Int41.79BQuantized暂不支持❌
HY-MT1.5-1.8B-GGUF1.79BGGUF暂不支持❌

Performance

Benchmark results on Ascend 910B NPU (2 devices):

MetricValue
Peak Throughput (batch=4)146.4 tok/s
Single-batch Throughput71.1 tok/s
Time to First Token (TTFT)83.1 ms
Time per Output Token (TPOT)11.4 ms
Tokens per Second87.7 tok/s
Batch 1 Latency (p50)0.25 s
Batch 4 Latency (p50)1.97 s

Accuracy

CPU vs NPU output comparison shows < 1% error across all test cases:

CPU vs NPU Accuracy Comparison

Ascend 910B NPU produces nearly identical outputs to CPU reference (cosine similarity > 0.9998, 23.4× speedup).

  • Logit-level cosine similarity: 0.9998+
  • Output text match rate: 99.7%
  • Translation quality preserved identically

See evaluation/screenshots/ for visual verification.

Demo

HY-MT1.5-1.8B Inference on Ascend 910B

Interactive translation on Ascend 910B NPU — 5 language pairs in under 1 second total.

Quick Start

Requirements

  • Python 3.10+
  • Huawei Ascend NPU (Atlas 800 A2 / Ascend 910B)
  • vLLM-Ascend (pip install vllm-ascend)
  • PyTorch with NPU support

Installation

# Install dependencies
pip install vllm vllm-ascend transformers torch-npu

# Clone model from HuggingFace / ModelScope
pip install modelscope
python3 -c "
from modelscope.hub.snapshot_download import snapshot_download
snapshot_download('Tencent-Hunyuan/HY-MT1.5-1.8B', 
                  cache_dir='/path/to/models/HY-MT1.5-1.8B')
"

Inference

# Interactive mode
python3 inference.py --model /path/to/HY-MT1.5-1.8B --mode interactive

# Batch mode
python3 inference.py --model /path/to/HY-MT1.5-1.8B --mode batch \
    --input test_inputs.jsonl --output results.json

# Benchmark mode
python3 inference.py --model /path/to/HY-MT1.5-1.8B --mode benchmark \
    --output benchmark.json

# Evaluation with accuracy comparison
python3 inference.py --model /path/to/HY-MT1.5-1.8B --mode eval \
    --output eval_results.json

Interactive Usage

>>> Hello world
  [en->zh] (0.25s): 你好,世界

>>> zh:en:你好世界
  [zh->en] (0.28s): Hello world

>>> en:ja:Good morning
  [en->ja] (0.31s): おはようございます

Project Structure

hy-mt-ascend/
├── inference.py                 # Main inference script
├── readme.md                    # This file
├── evaluation/
│   ├── eval_accuracy.py         # CPU vs NPU accuracy comparison
│   ├── eval_performance.py      # Throughput & latency benchmark
│   ├── run_eval.sh             # Full evaluation runner
│   ├── logs/
│   │   ├── accuracy.json        # Accuracy evaluation results
│   │   ├── performance.json     # Performance benchmark results
│   │   └── *.log               # Run logs
│   └── screenshots/
│       ├── accuracy_comparison.png     # Accuracy comparison screenshot
│       ├── inference_demo.png          # Interactive inference demo
│       └── performance_benchmark.png   # Performance results screenshot

## Model Details

HY-MT1.5 is a multilingual translation model supporting:

- **Languages**: Chinese, English, Japanese, Korean, French, German, Spanish, Portuguese, Russian, Arabic
- **Context Window**: 8192 tokens
- **Architecture**: Transformer Decoder (HunYuanDenseV1 variant)
- **Training**: Sequence-level knowledge distillation + contrastive learning

## Known Limitations

1. **CPU Inference**: Full model requires ~7GB RAM in BF16; CPU-only inference may fail on low-memory systems
2. **FP8 Models**: vLLM-Ascend does not natively support `compressed-tensors` format FP8 quantization; use the conversion script in `npu_ref/`
3. **GPTQ/GGUF**: Not supported by vLLM-Ascend
4. **Output Quality**: With temperature=0.0 (greedy), the model may occasionally repeat tokens; use temperature=0.7 with top_k=20 for best results

## License

This project is licensed under the Apache License 2.0.

The underlying model (Tencent HunYuan HY-MT1.5) is subject to Tencent's original license terms.

---

*Adapted for Huawei Ascend NPU by @m0_74196153*