sheepss/Datadog-Toto-2.0-22m-NPU
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

Datadog/Toto-2.0-22m on Ascend NPU (#NPU)

Zero-shot multivariate probabilistic time-series forecasting foundation model adapted to run on a single Ascend 910 NPU (npu:0).

  • Model: Datadog/Toto-2.0-22m (21.9M params, decoder-only patched transformer with alternating time/variate attention, u-μP scaling, quantile output head trained with pinball loss)
  • Task: time-series forecasting (observability / generic), probabilistic quantile forecast at levels [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9]
  • License: apache-2.0
  • Official code: https://github.com/DataDog/toto (Apache-2.0, toto2 module)

Model

Toto 2.0 is the current-generation Datadog time-series foundation model family. The 22m checkpoint is the efficient default size (~7x fewer parameters than Toto 1.0 at comparable quality). It predicts quantiles for multivariate series in a single forward pass using contiguous patch masking (CPM) and a block-decode KV cache. This repo verifies real inference of the released checkpoint on an Ascend NPU.

Data Contract (input)

FieldValue
Inputtarget tensor (batch, n_variates, time_steps) fp32
target_maskbool tensor same shape (observed mask)
series_idslong tensor (batch, n_variates)
Batch1
Variates4 (default)
Context length512 (default)
Horizon96 (default)
Outputquantiles (9, batch, n_variates, horizon) at levels 0.1..0.9
Axis order[quantile, batch, variate, time]
Scalinginternal causal std scaler + arcsinh, inverse-transformed to real space

Environment

  • Hardware: Ascend 910 (single card, npu:0)
  • OS / arch: Linux aarch64
  • Software: torch 2.9.0, torch_npu 2.9.0, CANN 8.5.1, unit-scaling 0.3.5

Installation

pip install -r requirements.txt
# torch_npu + torch must be installed for your Ascend CANN environment

NPU Inference

Weights must be downloaded first to a local model/ directory:

huggingface-cli download Datadog/Toto-2.0-22m --local-dir model
python inference.py --model_dir model --context_length 512 --horizon 96 --n_variates 4

inference.py is fully self-contained (embeds the Apache-2.0 toto2 model and the dd_unit_scaling unit-scaling layer). It loads the checkpoint from local disk, moves the model to npu:0, and produces a quantile forecast. It exits non-zero on any failure. The script requires the unit-scaling pip package at runtime.

Real Results (npu:0)

  • Total params: 21.92M (fp32)
  • Input: 1 batch x 4 variates x 512 context
  • Output quantiles shape: (9, 1, 4, 96), all finite, quantiles monotonic
  • Median (first variate, first 8 steps): [0.0436, 0.0452, 0.0370, 0.0382, 0.0389, 0.0387, 0.0395, 0.0415]
  • 10th percentile: [-1.2693, -1.2481, -1.2563, -1.2573, -1.2551, -1.2530, -1.2477, -1.2541]
  • 90th percentile: [1.3846, 1.3729, 1.3559, 1.3625, 1.3477, 1.3619, 1.3523, 1.3507]
  • Inference wall time (horizon 96, decode_block_size 768): ~0.35 s (first run), ~0.034 s (steady state)
  • Status: OK

Consistency (CPU vs NPU)

Same weights, identical input values, dtype fp32, seed fixed, eval mode:

  • shapes equal (9, 1, 4, 96), all finite
  • max abs error: 3.58e-06, mean abs error: 4.35e-07
  • np.allclose(atol=1e-4, rtol=1e-3) → PASS
  • quantiles monotonic across levels on both devices → PASS

Performance (npu:0, context 512, horizon 96, 1 batch, 4 variates, fp32)

MetricValue
first run (kernel compile)0.354 s
avg latency (10 runs)0.034 s
p50 / p950.034 s / 0.034 s
throughput~29.5 forecast/s
peak NPU memory~0.15 GB (of 64 GB HBM)

Evidence

  • assets/agent_workflow.png - end-to-end adaptation workflow log (xterm.js from real log)
  • assets/npu_device_call.png - NPU device / compute evidence (xterm.js from real log)
  • assets/model_result.png - default python inference.py run on npu:0 (xterm.js from real log)

Limitations

  • This is a zero-shot forecasting model; a fixed deterministic synthetic series is used as the demonstrable input. For your data, provide your own target / target_mask / series_ids following the data contract above.
  • The internal scaler operates in float64 where supported; on NPU it falls back to float32 (the model remains numerically consistent, see Consistency).
  • Single-variate or very long contexts (beyond 512) increase latency and memory; the decode_block_size controls block-decode caching for longer horizons.