liaogx_/patchtst-etth1-pretrain
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

PatchTST pre-trained on ETTh1 (ibm-research/patchtst-etth1-pretrain) on Ascend NPU

IBM Research PatchTST masked-pretraining checkpoint for the 7-channel ETTh1 dataset (ICLR 2023, arXiv:2211.14730), adapted for Ascend NPU via torch_npu.

Model Description

PatchTST (A Time Series is Worth 64 Words: Long-term Forecasting with Transformers) segments a time series into subseries-level patches served as input tokens to a vanilla Transformer encoder. Each channel is processed independently (channel independence) while sharing the same embedding and Transformer weights. This checkpoint is the masked-pretraining model pre-trained on the ETTh1 training split (7 channels: HUFL/HULL/MUFL/MULL/LUFL/LULL/OT). When fine-tuned and evaluated on the ETTh1 test split it reports MSE 0.3881 at 96-step-ahead forecasting from a 512-hour window.

Architecture: PatchTSTForPretraining (config architectures: PatchTSTForMaskPretraining)

  • Patch embedding: patch_length=12, patch_stride=12 → 42 patches (no CLS)
  • Transformer encoder: 6 layers, d_model=128, 16 heads, FFN 512
  • sincos positional encoding, shared_embedding, BatchNorm sublayers
  • mean scaling (internal), masked-patch pretraining head (reconstructs patches)
  • Context 512, prediction 24, 7 input channels
  • 1,198,220 parameters (~1.2M), fp32

Zero-shot inference via forecast masking

As a masked-pretraining checkpoint the model is served through its native forecast masking objective (mask_type="forecast"): the trailing prediction_length // patch_length patches of the 512-length window are masked and reconstructed by the pre-trained head. This yields a point forecast for the next prediction_length steps (default 24, also 12/48/96 supported) with zero fine-tuning. It captures the recent trend/seasonality of ETTh1 (24-step forecast RMSE ≈ 4.2 on the test split). For production accuracy on a specific horizon, fine-tune the head following the IBM demo notebook (MSE 0.3881 @96 after fine-tuning).

NPU Adaptation

vLLM-Ascend 0.18 does not support this non-autoregressive time-series architecture (not in the LLM registry), so we use torch_npu + transformers 4.57.6 native PatchTSTForPretraining + FastAPI.

The checkpoint was saved with transformers 4.33.0.dev0; loading with 4.57.6 requires 4 compatibility fixes (all handled in inference.py at load time, no source modification of transformers):

  1. config.stride → config.patch_stride (json field stride: 12)
  2. config.encoder_layers → config.num_hidden_layers (json encoder_layers: 6)
  3. config.encoder_attention_heads → config.num_attention_heads (json 16)
  4. state-dict key remap (old 4.33 names → 4.57.6): model.encoder.w_p.* → model.encoder.embedder.input_embedding.*, model.encoder.w_pos → model.encoder.positional_encoder.position_enc, model.encoder.encoder.layers.* → model.encoder.layers.*, norm_sublayer{1,3}.1.* → norm_sublayer{1,3}.batchnorm.*

Attention is forced to attn_implementation="eager" (NPU-stable, no flash-attn). All operators (eager matmul, BatchNorm, GELU, sincos, mean scaler) are natively supported by the Ascend NPU.

Environment

  • Ascend 910 (64GB HBM), CANN 8.5.1, torch_npu 2.9.0.post1
  • Python 3.11, torch 2.9.0, transformers 4.57.6, fastapi/uvicorn

Steps

# 1. download weights (ModelScope, sha256-verified)
python download_model.py

# 2. generate demo data (real ETTh1 test windows)
python make_test_data.py

# 3. start the service (npu:1, port 8044)
./start_serve.sh            # or: python server.py --port 8044

# 4. test the service
python test_server.py --port 8044

# 5. verify NPU vs CPU numerics
python verify_npu_vs_cpu.py

Test Case

curl -X POST http://127.0.0.1:8044/v1/forecast \
  -H 'Content-Type: application/json' \
  -d '{"past_values": <history>, "prediction_length": 24}'
# -> {"forecast": [[[...], ...]], "loc": [...], "scale": [...],
#     "prediction_length": 24, "context_length": 512, "latency_ms": 5.0, ...}

Request body (past_values) accepts [T], [T, C] or [B, T, C] (float32, ETTh1 7-channel). Each series must have length ≥ 512 - prediction_length; the last 512 - prediction_length values are used as visible context and the next prediction_length steps are predicted. Response forecast has shape [B, H, C]. A /v1/reconstruct endpoint returns the full 512-length window reconstruction (masked imputation, the model's native task).

Performance (Ascend 910, npu:1)

MetricValue
Model load~2.9 s
Steady-state forward (NPU, H=24)~5 ms
Steady-state HTTP p505.3 ms
CPU single-core forward~29.5 ms
NPU vs CPU max_abs_diff2.6e-4 (fp32)
NPU vs CPU cosine0.99998
HBM usage (process)156 MB
Port / device8044 / npu:1

Repository

  • inference.py — PatchTSTEngine (torch_npu, forecast-masking zero-shot)
  • server.py — FastAPI service (/v1/forecast, /v1/reconstruct, /health)
  • download_model.py — ModelScope downloader with sha256 idempotent check
  • make_test_data.py — real ETTh1 demo windows
  • verify_npu_vs_cpu.py — NPU vs CPU numerical equivalence check
  • test_server.py — HTTP end-to-end smoke test
  • start_serve.sh — one-click start script
  • assets/ — screenshot materials (workflow / result / npu-smi)

Model weights are not committed to this repository; run download_model.py to fetch them.