g
gcw_coj3XaOd/videoflextok_d18_d28
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

VideoFlexTok-d18-d28 (NPU Adapted)

VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization — adapted for Huawei Ascend 910 NPU.

Original paper: VideoFlexTok: Flexible-Length Coarse-to-Fine Video Tokenization, arXiv 2026 Original code: apple/ml-videoflextok

Model Description

VideoFlexTok represents videos with a variable-length sequence of tokens structured in a coarse-to-fine manner, where the first tokens capture abstract information (semantics, motion) and later tokens add fine-grained spatial details.

This adapted version includes NPU-specific fixes:

  • FlexAttention fallback — manual scaled dot-product attention for Ascend NPU (FlexAttention only supports CUDA/CPU/HPU natively).
  • Sequence packing — padding to multiple-of-128 for BlockMask creation on NPU.
  • Checkpoint key stripping — automatic removal of video_tokenizer. prefix when loading from the original safetensors checkpoint.

Architecture

ComponentDetails
EncoderViT-L, 18 layers, dim=1152
DecoderDiT-XL, 28 layers, dim=1792
VAEVidTok KL Causal (488x288, 16-channel latent)
QuantizerFSQ levels = [8, 8, 8, 5, 5, 5]
Total Params~2.69 B

Installation

git clone https://gitcode.com/gcw_coj3XaOd/videoflextok_d18_d28.git
cd videoflextok_d18_d28
pip install -e . --no-deps

pip install -r requirements.txt

Note: Install torch-npu following the CANN toolkit instructions.

NPU Patches

This repository includes the NPU-adapted videoflextok/ package with two source patches already applied:

  1. videoflextok/model/layers/attention.py — FlexAttention NPU fallback: When q.device.type == "npu", replaces the CUDA-only flex_attention call with a manual scaled dot-product attention implementation that supports score_mod and block_mask.

  2. videoflextok/model/preprocessors/flex_seq_packing.py — Sequence padding: Re-enabled next_highest_multiple(seq_len_per_batch, 128) padding for the repa block mask, satisfying FlexAttention's 128-modulo assertion on NPU.

The patches/ directory also contains standalone diff files for applying these changes to the upstream Apple repository.

Project Structure

videoflextok_d18_d28/
├── videoflextok/          # NPU-adapted source package (from apple/ml-videoflextok)
├── inference.py           # Standalone inference script for NPU
├── requirements.txt       # Python dependencies
├── setup.py               # Package installation
├── README.md              # This file
├── weights/               # Place model.safetensors + config.json here
├── assets/                # Media assets
└── output/                # Inference outputs

Quick Start

# Quick verification with dummy tensor
export TORCHDYNAMO_DISABLE=1
python inference.py --dummy

# Inference on a real video
python inference.py \
    --video_path ./data/my_video.mp4 \
    --timesteps 30 \
    --guidance_scale 20.0 \
    --output_dir ./output

Python API

import torch
from inference import build_model, tokenize_video, detokenize_video

model = build_model("./weights", device="npu")

# Tokenize: video shape = [1, 3, T, 256, 256], values in [-1, 1]
# T must be 1 + K * 16 (e.g. 17, 33, 49, ...)
video = torch.randn(1, 3, 17, 256, 256, device="npu")
tokens = tokenize_video(model, video)
# tokens[0].shape == torch.Size([1, 5, 256])

# Detokenize back to video
reconst = detokenize_video(tokens, timesteps=30, guidance_scale=20.0)
# reconst[0].shape == torch.Size([1, 3, 17, H, W])

Input Requirements

FieldConstraint
Batch size1
Channels3 (RGB)
Frames (T)Must be 1 + K * 16 for integer K ≥ 1 (e.g. 17, 33, 49)
Spatial256 × 256
Value range[-1, 1] (normalized)

Output

  • Tokens: list of tensors with shape [1, t, 256], t = 1 + K * 4 (each 16-frame chunk produces 4 tokens, plus the initial frame token).
  • Reconstruction: [1, 3, T, H, W] RGB video.

Performance (Ascend 910, measured)

OperationInputTime
Tokenize (encode)(1, 3, 17, 256, 256)~1.0 s
Detokenize (decode, 5 steps)5 tokens~1.2 s

5. 测试样例及输出结果

样例 1:Dummy 张量推理

运行命令:

export TORCHDYNAMO_DISABLE=1
python3 inference.py --model_path ./weights --dummy

输出:

Using device: npu
Loading model from ./weights ...
Model loaded in 61.1s
Using dummy video tensor (1, 3, 17, 256, 256)

--- Tokenize ---
Tokenize done in 0.99s
  Token 0: shape=(1, 5, 256), dtype=torch.int64

--- Detokenize ---
Detokenize done in 12.24s
  Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32

Outputs saved to output/
[状态] SUCCESS

样例 2:真实视频推理

运行命令:

python3 inference.py \
    --video_path ./data/my_video.mp4 \
    --timesteps 30 \
    --guidance_scale 20.0 \
    --output_dir ./output

输出:

Loaded video from ./data/my_video.mp4: shape=(1, 3, 17, 256, 256)

--- Tokenize ---
Tokenize done in 1.00s
  Token 0: shape=(1, 5, 256), dtype=torch.int64

--- Detokenize ---
Detokenize done in ~12s
  Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32

Outputs saved to output/
[状态] SUCCESS

6. Agent适配截图

6.1 Agent 完整适配工作流

Agent 适配流程

6.2 NPU 设备调用日志

NPU 设备调用

6.3 模型适配结果

模型适配结果

7. 精度评测

VideoFlexTok is a video tokenizer, not a generative model. Quality is evaluated by reconstruction fidelity (how well detokenized video recovers the input).

测试数据: Random dummy tensor (1, 3, 17, 256, 256), values in [-1, 1]

评测指标:

指标结果说明
Tokenize 耗时~1.0 s前向编码 + FSQ量化
Detokenize 耗时~12.2 s30-step flow matching 去噪
Token shape[1, 5, 256]17帧 → 5个 coarse-to-fine tokens
重建 shape[1, 3, 17, 128, 128]空间下采样至128×128 (VAE latent)

评测命令:

export TORCHDYNAMO_DISABLE=1
python3 inference.py --model_path ./weights --dummy

实际输出:

Tokenize done in 0.99s
  Token 0: shape=(1, 5, 256), dtype=torch.int64
Detokenize done in 12.24s
  Reconst 0: shape=(1, 3, 17, 128, 128), dtype=torch.float32
[状态] SUCCESS

8. NPU 配置说明

配置项值
NPU 型号Ascend 910 (Ascend910_9362)
卡数1
CANN 版本8.5.1
PyTorch 版本2.x (torch-npu)
推理引擎PyTorch NPU (手动 SDPA fallback)
显存占用~8 GB (模型2.69B params + 激活值)

9. 已知问题

  1. FlexAttention 不支持 NPU — 已通过手动 SDPA 实现绕过。该 fallback 支持 score_mod 和 block_mask,但性能低于 CUDA 原生 FlexAttention。
  2. 序列长度需为 128 倍数 — BlockMask 创建时要求序列长度为 128 的倍数,已在 flex_seq_packing.py 中自动 padding。
  3. Checkpoint key 前缀 — 原始 checkpoint 所有 key 带有 video_tokenizer. 前缀,已自动 stripping。
  4. 重建分辨率为 128×128 — 由于 VAE latent 空间下采样,重建输出空间分辨率为 128×128,低于输入的 256×256。这是 VAE 架构固有特性。
  5. GPU 不支持 — 本适配仅针对昇腾 NPU。CUDA 推理请使用 原始 VideoFlexTok。

Citation

@article{videoflextok,
    title={{VideoFlexTok}: Flexible-Length Coarse-to-Fine Video Tokenization},
    author={Andrei Atanov and Jesse Allardice and Roman Bachmann and
            O{\u{g}}uzhan Fatih Kar and Peter Fu and David Griffiths and
            Devon Hjelm and Afshin Dehghan and Amir Zamir},
    journal={arXiv 2026},
    year={2026}
}

License

This adapted model and code are released under the Apache License 2.0. The original VideoFlexTok model weights are also Apache 2.0.