VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization — adapted for Huawei Ascend 910 NPU.
Original paper: VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization, arXiv 2026 Original code: apple/ml-videoflextok
VideoFlexTok represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens capture abstract information (semantics, motion) and later tokens add fine-grained spatial details.
This adapted version includes NPU-specific fixes:
video_tokenizer.
prefix when loading from the original safetensors checkpoint.| Component | Details |
|---|---|
| Encoder | ViT-L, 18 layers, dim=1152 |
| Decoder | DiT-XL, 28 layers, dim=1792 |
| VAE | VidTok KL Causal (488x288, 16-channel latent) |
| Quantizer | FSQ levels = [8, 8, 8, 5, 5, 5] |
| Total Params | ~2.69 B |
git clone https://gitcode.com/gcw_coj3XaOd/videoflextok_d18_d28.git
cd videoflextok_d18_d28
pip install -e . --no-deps
pip install -r requirements.txtNote: Install
torch-npufollowing the CANN toolkit instructions.
This repository includes the NPU-adapted videoflextok/ package with two source patches already applied:
videoflextok/model/layers/attention.py — FlexAttention NPU fallback:
When q.device.type == "npu", replaces the CUDA-only flex_attention call with a manual scaled dot-product attention implementation that supports score_mod and block_mask.
videoflextok/model/preprocessors/flex_seq_packing.py — Sequence padding:
Re-enabled next_highest_multiple(seq_len_per_batch, 128) padding for the repa block mask, satisfying FlexAttention's 128-modulo assertion on NPU.
The patches/ directory also contains standalone diff files for applying these changes to the upstream Apple repository.
videoflextok_d18_d28/
├── videoflextok/ # NPU-adapted source package (from apple/ml-videoflextok)
├── inference.py # Standalone inference script for NPU
├── requirements.txt # Python dependencies
├── setup.py # Package installation
├── README.md # This file
├── weights/ # Place model.safetensors + config.json here
├── assets/ # Media assets
└── output/ # Inference outputs# Quick verification with dummy tensor
export TORCHDYNAMO_DISABLE=1
python inference.py --dummy
# Inference on a real video
python inference.py \
--video_path ./data/my_video.mp4 \
--timesteps 30 \
--guidance_scale 20.0 \
--output_dir ./outputimport torch
from inference import build_model, tokenize_video, detokenize_video
model = build_model("./weights", device="npu")
# Tokenize: video shape = [1, 3, T, 256, 256], values in [-1, 1]
# T must be 1 + K * 16 (e.g. 17, 33, 49, ...)
video = torch.randn(1, 3, 17, 256, 256, device="npu")
tokens = tokenize_video(model, video)
# tokens[0].shape == torch.Size([1, 5, 256])
# Detokenize back to video
reconst = detokenize_video(tokens, timesteps=30, guidance_scale=20.0)
# reconst[0].shape == torch.Size([1, 3, 17, H, W])| Field | Constraint |
|---|---|
| Batch size | 1 |
| Channels | 3 (RGB) |
| Frames (T) | Must be 1 + K * 16 for integer K ≥ 1 (e.g. 17, 33, 49) |
| Spatial | 256 × 256 |
| Value range | [-1, 1] (normalized) |
[1, t, 256], t = 1 + K * 4
(each 16-frame chunk produces 4 tokens, plus the initial frame token).[1, 3, T, H, W] RGB video.| Operation | Input | Time |
|---|---|---|
| Tokenize (encode) | (1, 3, 17, 256, 256) | ~1.0 s |
| Detokenize (decode, 5 steps) | 5 tokens | ~1.2 s |
运行命令:
export TORCHDYNAMO_DISABLE=1
python3 inference.py --model_path ./weights --dummy输出:
Using device: npu
Loading model from ./weights ...
Model loaded in 61.1s
Using dummy video tensor (1, 3, 17, 256, 256)
--- Tokenize ---
Tokenize done in 0.99s
Token 0: shape=(1, 5, 256), dtype=torch.int64
--- Detokenize ---
Detokenize done in 12.24s
Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32
Outputs saved to output/
[状态] SUCCESS运行命令:
python3 inference.py \
--video_path ./data/my_video.mp4 \
--timesteps 30 \
--guidance_scale 20.0 \
--output_dir ./output输出:
Loaded video from ./data/my_video.mp4: shape=(1, 3, 17, 256, 256)
--- Tokenize ---
Tokenize done in 1.00s
Token 0: shape=(1, 5, 256), dtype=torch.int64
--- Detokenize ---
Detokenize done in ~12s
Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32
Outputs saved to output/
[状态] SUCCESS


VideoFlexTok is a video tokenizer, not a generative model. Quality is evaluated by reconstruction fidelity (how well detokenized video recovers the input).
测试数据: Random dummy tensor (1, 3, 17, 256, 256), values in [-1, 1]
评测指标:
| 指标 | 结果 | 说明 |
|---|---|---|
| Tokenize 耗时 | ~1.0 s | 前向编码 + FSQ量化 |
| Detokenize 耗时 | ~12.2 s | 30-step flow matching 去噪 |
| Token shape | [1, 5, 256] | 17帧 → 5个 coarse-to-fine tokens |
| 重建 shape | [1, 3, 17, 128, 128] | 空间下采样至128×128 (VAE latent) |
评测命令:
export TORCHDYNAMO_DISABLE=1
python3 inference.py --model_path ./weights --dummy实际输出:
Tokenize done in 0.99s
Token 0: shape=(1, 5, 256), dtype=torch.int64
Detokenize done in 12.24s
Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32
[状态] SUCCESS| 配置项 | 值 |
|---|---|
| NPU 型号 | Ascend 910 (Ascend910_9362) |
| 卡数 | 1 |
| CANN 版本 | 8.5.1 |
| PyTorch 版本 | 2.x (torch-npu) |
| 推理引擎 | PyTorch NPU (手动 SDPA fallback) |
| 显存占用 | ~8 GB (模型2.69B params + 激活值) |
score_mod 和 block_mask,但性能低于 CUDA 原生 FlexAttention。flex_seq_packing.py 中自动 padding。video_tokenizer. 前缀,已自动 stripping。@article{videoflextok,
title={{VideoFlexTok}: Flexible-Length Coarse-to-Fine Video Tokenization},
author={Andrei Atanov and Jesse Allardice and Roman Bachmann and
O{\u{g}}uzhan Fatih Kar and Peter Fu and David Griffiths and
Devon Hjelm and Afshin Dehghan and Amir Zamir},
journal={arXiv 2026},
year={2026}
}This adapted model and code are released under the Apache License 2.0. The original VideoFlexTok model weights are also Apache 2.0.