hf_mirrors/llava-calm2-siglip
Model Introduction
Files & versions
Analysis

Model Description

llava-calm2-siglip is an experimental Vision Language Model that can answer questions in Japanese about images.

Usage

from PIL import Image
import requests
from transformers import AutoProcessor, LlavaForConditionalGeneration
import torch

model = LlavaForConditionalGeneration.from_pretrained(
    "cyberagent/llava-calm2-siglip",
    torch_dtype=torch.bfloat16,
).to(0)

processor = AutoProcessor.from_pretrained("cyberagent/llava-calm2-siglip")

prompt = """USER: <image>
この画像を説明してください。
ASSISTANT: """

url = "https://unsplash.com/photos/LipkIP4fXbM/download?force=true&w=640"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")

inputs = processor(text=prompt, images=image, return_tensors="pt").to(0, torch.bfloat16)
generate_ids = model.generate(**inputs,
                              max_length=500,
                              do_sample=True,
                              temperature=0.2,
                    )
output = processor.tokenizer.decode(generate_ids[0][:-1], clean_up_tokenization_spaces=False)

print(output)

# USER: <image>
# この画像を説明してください。
# ASSISTANT: 画像には、木製のテーブルの上に置かれた、たこ焼き器で焼かれた3つのたこ焼きが映っています。たこ焼きは、小麦粉をベースにした生地を丸く焼き、中にタコや天かす、紅ショウガなどの具材を入れたものです。たこ焼きは、ソース、マヨネーズ、青海苔、かつおぶしをかけて食べることが多いです。

Chat Template

USER: <image>
{user_message1}
ASSISTANT: {assistant_message1}<|endoftext|>
USER: {user_message2}
ASSISTANT: {assistant_message2}<|endoftext|>
USER: {user_message3}
ASSISTANT: {assistant_message3}<|endoftext|>

Model Details

  • Model size: 7B
  • Model type: Transformer-based Vision Language Model
  • Language(s): Japanese, English
  • Developed by: CyberAgent, Inc.
  • License: Apache-2.0

Use and Limitations

Intended Use

This model is designed for use by the open-source community in vision-language applications and academic research.

Limitations and biases

This model, a general-purpose Japanese VLM, reaches optimal performance when specifically tuned with relevant data for each task. Though technically possible, commercial use is advised with caution, and the implementation of mechanisms to filter out inappropriate content is strongly recommended when deployed in production systems. This model is not advisable for use in applications that could potentially harm individuals or groups, or cause distress. CyberAgent expressly disclaims any liability for direct, indirect, special, incidental, or consequential damages, as well as for any losses that may result from using this model, regardless of the outcomes. Users must fully understand these limitations before employing the model.