atlasleong/videoflextok-d18-d18-k600-npu
Model Introduction
Files & versions
Pull Requests
Discussion
Analysis

VideoFlexTok d18/d18 K600 — Ascend NPU port

This delivery adapts the EPFL-VILAB/videoflextok_d18_d18_k600 video tokenizer / reconstructor so that it runs end-to-end on a physical Ascend NPU (torch_npu, device npu:7 — Ascend910B4-1, CANN 8.5.1) while preserving the official VideoFlexTokFromHub architecture and strict-loading all real pretrained tensors.

AssetPreview
Agent / pipeline workflowagent workflow
NPU-7 device-call capturenpu device call
Input vs reconstructionmodel result
Live session (added by orchestrator)live session

License note. The README frontmatter uses license: other because the delivery mixes two licenses: the model weights are Apache-2.0 (per model card) and the official source code is Apple/EPFL Sample Code License 1.1, non-commercial. See LICENSE and ACKNOWLEDGEMENTS.md.


模型简介 (Model overview)

VideoFlexTok is a flexible-length coarse-to-fine video tokenizer. The d18/d18 K600 checkpoint uses 18 transformer encoder layers and 18 rectified-flow decoder layers, an FSQ bottleneck with levels [8, 8, 8, 5, 5, 5], and a causal VidTok VAE (16 channels, 4×8×8 compression). The full neural path exercised here is:

VidTok VAE encode  -> 18 encoder layers (dim 1152, 18 heads)
  -> FSQ tokens (levels [8,8,8,5,5,5])
  -> 18 rectified-flow decoder layers (adaLN, >= 1 genuine denoising step)
  -> VidTok VAE decode

Ascend NPU adaptations (in the vendored videoflextok/ package)

  1. torch.compile(flex_attention) (CUDA/Inductor-specific) is replaced with native torch.nn.functional.scaled_dot_product_attention. The existing BlockMask.mask_mod is materialized into a dense boolean token mask on the NPU. self.scale is preserved; a non-null score_mod is explicitly rejected (no silent semantic drop).
  2. The incidental torchvision dependency in utils/misc.py is removed; the channel-wise denormalize is implemented with tensor arithmetic.
  3. Hard-coded disabled-CUDA autocast contexts/decorators in the flow / FSQ path are replaced with device-derived contexts (npu/cuda/cpu from the tensor).
  4. diffusers is not required (only a type annotation upstream); annotations are relaxed to Any.
  5. Weights are strict-loaded via safetensors.torch.load_file(..., device="cpu") and verified (741 F32 tensors, 1,057,722,429 elements, prefix video_tokenizer., SHA-256 matches the immutable audit hash).

No transformer layer is approximated, pruned, fabricated or skipped.


环境依赖清单 (Environment & dependencies)

ComponentVersionNote
Container OS—Ascend container
torch2.9.0+cpuprovided by runtime — do NOT install/upgrade
torch_npu2.9.0provided by runtime — do NOT install/upgrade
CANN8.5.1ASCEND_HOME_PATH=/usr/local/Ascend/cann-8.5.1
Python3.11.14venv
numpy1.26.4pinned in requirements.txt
safetensors0.8.0pinned
hydra-core / omegaconf1.3.5 / 2.3.1pinned
mup1.0.0installed with --no-deps
einops / PyYAML / tqdm / huggingface_hubpinnedsee requirements.txt
Physical devicenpu:7Ascend910B4-1, 8 NPUs, ~62.4 GiB usable HBM per device

Install command (all pip via the Aliyun mirror, never touching torch/torch_npu):

pip install -i https://mirrors.aliyun.com/pypi/simple/ \
    --trusted-host mirrors.aliyun.com \
    hydra-core==1.3.5 omegaconf==2.3.1
pip install -i https://mirrors.aliyun.com/pypi/simple/ \
    --trusted-host mirrors.aliyun.com --no-deps mup==1.0.0

分步推理操作流程 (Step-by-step inference)

The model weights stay external at the immutable cache:

  • config: /work/pipeline/models/videoflextok/videoflextok-d18-d18-k600/config.json
  • weights: /work/pipeline/models/videoflextok/videoflextok-d18-d18-k600/model.safetensors

1. (Optional) Narrow construction / strict-load check — no NPU required

python scripts/narrow_check.py

2. Final candidate command (NPU smoke workload)

python inference.py --device npu:7 --out-dir .

This runs the deterministic CC0 moving-gradient fixture [1, 3, 17, 128, 128] normalized to [-1, 1] through the complete neural path with timesteps=1 and guidance_scale=1.0 (honestly labeled a compatibility smoke test, not a quality claim). Artifacts written under logs/:

  • logs/inference.log — full text log
  • logs/inference.pid — OS PID written by os.getpid()
  • logs/npu_result.json — machine-readable result payload
  • logs/npu_device_call.log — concurrent npu-smi device-call monitor capture
  • logs/input_fixture.npy, logs/reconst.npy — small render intermediates

3. Render assets from the real logs

python scripts/render_assets.py

4. Validate the delivery

python scripts/validate_delivery.py

完整测试用例 (Complete test case)

Deterministic, self-authored CC0 moving-gradient fixture:

  • shape [1, 3, 17, 128, 128], normalized to [-1, 1]
  • fixture SHA-256 7d9d740e6fd478556f5516189cd88c35b7849a4f16c04fccd4e2492359e00756
  • timesteps=1, guidance_scale=1.0, no normalized guidance

Neural path executed: VidTok VAE encode → 18 encoder layers → FSQ → 18 rectified-flow decoder layers (one genuine denoising step) → VidTok VAE decode.

Final command and observed output:

$ python inference.py --device npu:7 --out-dir .
...
2026-08-21 09:40:40 INFO  Weight integrity verified (size + sha256)
2026-08-21 09:40:42 INFO  Fixture ready: shape=(1, 3, 17, 128, 128) range=[-0.762, 0.762] dtype=bfloat16 checksum=7d9d740e6fd478556f5516189cd88c35b7849a4f16c04fccd4e2492359e00756
2026-08-21 09:40:50 INFO  Tokenize done in 7.48s: 1 chunk(s), tokens shape=(1, 5, 256) (device=npu:7)
2026-08-21 09:40:50 INFO  Detokenize done in 0.19s: 1 chunk(s), reconst shape=(1, 3, 17, 128, 128) (device=npu:7)
2026-08-21 09:40:50 INFO  Inference finished successfully (total 38.89s)
$ echo $?
0

Exit code 0. logs/npu_device_call.log contains the exact Python PID 214264 in the physical NPU-7 process table.


输出结果 (Output results)

Recorded in logs/npu_result.json (final run, PID 214264):

ItemValue
exit_code0
devicenpu:7 (Ascend910B4-1, 8 NPUs, 62,420 MB total HBM)
dtypebfloat16 (recommended on Ascend; fp32 available via --dtype fp32)
strict load741 F32 tensors, 0 missing / 0 unexpected keys
weight4,230,998,164 bytes, SHA-256 62164b...c0347b5f, verified
tokenizationtokens shape [1, 5, 256] (1280 tokens, 1116 unique, range 6..63974)
reconstructionshape [1, 3, 17, 128, 128], finite, range [-1.000, 1.047], mean 0.043, std 0.602
MSE0.02442
PSNR22.14 dB
timingsconstruction 19.86 s, strict load 0.03 s, sha256 4.23 s, tokenize 7.48 s, detokenize 0.19 s, total 38.89 s

MSE/PSNR are labeled compatibility metrics computed on [-1,1] normalized pixels (dynamic range 2.0) against the deterministic input. With a single denoising step they measure that the pipeline runs and produces finite, structured output — they are not a claim about reconstruction quality.


故障排查 (Troubleshooting)

SymptomLikely cause / fix
torch.npu.is_available() is Falsetorch_npu not loaded; confirm container torch==2.9.0+cpu + torch_npu==2.9.0, CANN 8.5.1
NPU device npu:7 unavailablefail-closed; check npu-smi info that device 7 exists
strict load has N missing/unexpected keysfail-closed; model/checkpoint mismatch — verify the immutable model cache revision
tensors found off the target devicefail-closed; model not fully placed on npu:7
ArgSort ... running on AiCpu warningbenign CANN kernel warning (int32/64 sort falls back to AiCPU)
TypedStorage is deprecated warningbenign torch_npu internal warning
posembs[slices] UserWarningupstream indexing deprecation notice, harmless

限制与许可证 (Limitations & license)

Limitations

  • This is a compatibility smoke test (timesteps=1) on a synthetic moving gradient. It validates placement, dtype, strict load and the full neural path; it does not demonstrate reconstruction quality or generation fidelity.
  • Only the assigned physical device npu:7 is used (per container contract).
  • Weights remain external at the documented immutable path; the delivery is self-contained code only.
  • torch / torch_npu must not be modified by pip.

License

  • Model weights: Apache-2.0 (per the EPFL-VILAB/videoflextok_d18_d18_k600 model card).
  • Official source code: Apple/EPFL Sample Code License 1.1 (non-commercial).
  • Test fixture: CC0 — self-authored moving-gradient tensor, no external data.
  • The README frontmatter therefore declares license: other.

See LICENSE and ACKNOWLEDGEMENTS.md.


版本与校验 (Revisions & hashes)

EntityValue
Model revisionb1e468168ba7a1869206d3fc74e369c095895722
Official source revision4fd72ec0c0c8c05f879012cce285d5a14c81920f
model.safetensors size4,230,998,164 bytes
model.safetensors SHA-25662164b971db5e226ef36a4212aef7ea256ad8accd09638c8d7bb63edc0347b5f
Test-input fixture SHA-2567d9d740e6fd478556f5516189cd88c35b7849a4f16c04fccd4e2492359e00756