TIPSv2 — SO400m/14

TIPSv2（具备空间感知能力的图文预训练）是一个对比式视觉-语言模型家族，能够生成与文本嵌入对齐的、富含空间信息的图像特征。这是 SO400m 变体，包含 412M 视觉参数和 448M 文本参数。您可以尝试下方的代码片段，或查看 GitHub 仓库以获取更多用例和可视化结果，包括零样本分割。

变体	视觉参数	文本参数	嵌入维度	DPT 头
B/14	86M	110M	768	B/14-dpt
L/14	303M	184M	1024	L/14-dpt
SO400m/14	412M	448M	1152	SO400m/14-dpt
g/14	1.1B	389M	1536	g/14-dpt

用法

pip install transformers torch torchvision sentencepiece scikit-learn

加载模型

from transformers import AutoModel

model = AutoModel.from_pretrained("google/tipsv2-so400m14", trust_remote_code=True)
model.eval()

编码图像

图像应为 [0, 1] 范围内的张量（只需使用 ToTensor()，无需进行 ImageNet 归一化）。

from torchvision import transforms
from PIL import Image
import requests

transform = transforms.Compose([
    transforms.Resize((448, 448)),
    transforms.ToTensor(),
])

url = "https://huggingface.co/spaces/google/TIPSv2/resolve/main/examples/zeroseg/pascal_context_00049_image.png"
image = Image.open(requests.get(url, stream=True).raw)
pixel_values = transform(image).unsqueeze(0)
out = model.encode_image(pixel_values)

print(out.cls_token.shape)     # (1, 1, 1152) — global image embedding
print(out.patch_tokens.shape)  # (1, 1024, 1152) — per-patch spatial features

编码文本

text_emb = model.encode_text(["a photo of a bus", "a photo of a dog"])
print(text_emb.shape)  # (2, 1152) — one embedding per query

零样本分类

import torch.nn.functional as F

classes = ["bus", "car", "dog", "cat"]
cls = F.normalize(out.cls_token[:, 0, :], dim=-1)
text_emb = F.normalize(model.encode_text(classes), dim=-1)
similarity = cls @ text_emb.T
print(classes[similarity.argmax()])  # bus — predicted class

空间特征可视化

import numpy as np
from sklearn.decomposition import PCA

spatial = out.patch_tokens.reshape(1, 32, 32, 1152)
feat = spatial[0].detach().cpu().numpy().reshape(-1, 1152)
rgb = PCA(n_components=3, whiten=True).fit_transform(feat).reshape(32, 32, 3)
rgb = 1 / (1 + np.exp(-2.0 * rgb))  # sigmoid for [0, 1] range with good contrast
print(rgb.shape)  # (32, 32, 3) — PCA of patch features as RGB

GPU 推理

model = model.cuda()
out = model.encode_image(pixel_values.cuda())
text_emb = model.encode_text(["a city"])

模型详情

架构：ViT 视觉编码器（27 层） + Transformer 文本编码器（27 层）
图像预处理：缩放至任意分辨率，转换为 [0, 1]（无需 ImageNet 归一化）
文本预处理：SentencePiece 分词器，转为小写，最多 64 个 token
图像块大小：14x14 像素

许可证

Apache 2.0

引用

@inproceedings{cao2026tipsv2,
  title     = {{TIPSv2: Advancing Vision-Language Pretraining with Enhanced Patch-Text Alignment}},
  author    = {Cao, Bingyi and Chen, Koert and Maninis, Kevis-Kokitsi and Chen, Kaifeng and Karpur, Arjun and Xia, Ye and Dua, Sahil and Dabral, Tanmaya and Han, Guangxing and Han, Bohyung and Ainslie, Joshua and Bewley, Alex and Jacob, Mithun and Wagner, Rene and Ramos, Washington and Choromanski, Krzysztof and Seyedhosseini, Mojtaba and Zhou, Howard and Araujo, Andre},
  booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year      = {2026},
  url       = {https://arxiv.org/abs/2604.12012}
}