Understanding VLM in One Article

VLM (Vision-Language Model): large vision-language model

Vision-Language Models (VLMs) are one of the most exciting developments in artificial intelligence. They allow machines to “see” images and “understand” language at the same time, opening a new paradigm for human-computer interaction. This article takes you from zero to a comprehensive understanding of how VLMs work, how they have evolved, and where they are headed.

1. What Is a Vision-Language Model?

1.1 An intuitive explanation: giving AI “eyes”

Imagine if you could have a conversation like this with a machine:

You: “Please describe what is in this picture.”
AI: “There is a golden retriever running on the grass, with a blue sky and white clouds in the background, and a few trees in the distance.”

Or something a little more complex:

You: “What trend does this chart show?”
AI: “This bar chart shows the company’s revenue growing year over year from 2020 to 2024, with the highest growth rate in 2023, reaching 35%.”

This is the capability of a vision-language model (VLM)—it can process visual information (images, video) and linguistic information (text) at the same time, and establish connections between the two.

According to IBM’s definition, a vision-language model is a multimodal model that fuses computer vision (CV) and natural language processing (NLP) capabilities into a unified system. Put simply, VLM = visual understanding + language understanding.

1.2 Technical definition: the culmination of multimodal AI

From a technical perspective, a VLM (also often called Large Vision-Language Models, or LVLMs) usually consists of three core components:

The roles of the three core components:

1. Visual Encoder: responsible for extracting visual features from images. It usually uses the Vision Transformer (ViT) architecture, such as CLIP-ViT, EVA-CLIP, etc. It converts the input image into a sequence of “visual tokens” that the model can understand.

2. Vision-Language Adapter: acts as a “bridge” between vision and language, mapping visual features into the embedding space of the large language model. Common designs include:

   • a single linear projection (e.g., LLaVA)
   • an MLP with nonlinearity (e.g., MiniGPT-4)
   • the Q-Former architecture (e.g., BLIP-2)

3. Large Language Model (LLM): responsible for understanding visual and textual information and generating natural language responses. It usually uses a pre-trained large language model, such as LLaMA, Vicuna, Qwen, etc.

1.3 How VLMs differ from vision-only and language-only models

The core advantage of VLMs lies in cross-modal understanding: they can not only recognize objects in an image, but also understand the relationships between objects and the semantic information of the scene, and express it in natural language.

2. The Development of VLMs: From Early Beginnings to Breakout

2.1 Early exploration (2019–2020): the rise of dual-tower architectures

The development of VLMs can be traced back to 2019–2020. Research during this period mainly adopted dual-stream architectures, in which separate encoders processed visual and linguistic information, which was then aligned in a shared space.

Representative work:

• ViLBERT (2019): extended the BERT architecture to the vision-language domain, using a dual-stream architecture to process images and text separately
• VisualBERT (2020): the first image-text pre-training model, using Faster R-CNN to extract visual features
• UNITER (2020): large-scale vision-language pre-training, achieving SOTA on multiple downstream tasks

These early models share the following characteristics:

• They use object detectors (such as Faster R-CNN) to extract region features
• They use single-stream or dual-stream Transformers for cross-modal fusion
• Their pre-training tasks mainly include masked language modeling (MLM) and image-text matching (ITM)

2.2 The CLIP era (2021): the contrastive learning revolution

In 2021, OpenAI released CLIP (Contrastive Language-Image Pre-training), a milestone in the history of VLMs.

CLIP’s core innovations:

1. Contrastive learning paradigm: learning cross-modal representations by maximizing the similarity of matching image-text pairs and minimizing the similarity of non-matching pairs
2. Dual-tower architecture: the image encoder and text encoder work independently, and their outputs are mapped into the same semantic space
3. Large-scale data: trained on 400 million image-text pairs collected from the internet

CLIP’s revolutionary significance lies in:

• Zero-shot capability: can be applied to downstream tasks without fine-tuning
• Open vocabulary: not limited to predefined categories, and able to recognize arbitrary concepts
• Strong transferability: widely used as a visual encoder in subsequent VLMs

Around the same time, Google released ALIGN, which adopted a similar contrastive learning approach but trained on 1.8 billion noisier image-text pairs, further confirming the importance of scale.

2.3 Unified frameworks (2022): integrating understanding and generation

In 2022, Salesforce released BLIP (Bootstrapping Language-Image Pre-training), proposing a unified framework for vision-language understanding and generation.

BLIP’s core contributions:

1. MED (Multimodal Mixture of Encoder-Decoder) architecture:

   • Unimodal encoder: encodes images and text separately
   • Image-conditioned text encoder: fuses visual information for understanding
   • Image-conditioned text decoder: generates image captions

2. CapFilt (Captioning and Filtering):

   • Uses a “generator + filter” strategy to clean web data
   • The generator (Captioner) generates synthetic captions for web images
   • The filter (Filter) removes noisy captions

BLIP achieved SOTA performance on image-text retrieval, image captioning, visual question answering, and other tasks, and demonstrated strong zero-shot transfer capability.

2.4 The large-model era (2023): LLM-powered visual assistants

2023 was the breakout year for VLMs, with the core trend being combining pre-trained large language models (LLMs) with visual capabilities.

Representative models:

Common characteristics of this period:

1. Frozen pre-trained models: the visual encoder and LLM remain frozen, and only the adapter is trained, greatly reducing training cost
2. Instruction tuning: fine-tuning the model on instruction data to improve instruction-following ability
3. End-to-end training: generating text output directly from visual input, simplifying the pipeline

Taking LLaVA as an example, it was the first to introduce visual instruction tuning to VLMs:

• Uses GPT-4 to generate multimodal instruction-following data
• Connects a CLIP visual encoder with a Vicuna LLM
• End-to-end fine-tuning enables strong multimodal dialogue capabilities

BLIP-2, by contrast, proposed a two-stage training strategy:

• Stage one: train the Q-Former with a frozen visual encoder for vision-language representation learning
• Stage two: connect the Q-Former to a frozen LLM for vision-to-language generative learning

2.5 Recent progress (2024–2025): natively multimodal large models

Entering 2024–2025, VLMs are moving toward larger scale, stronger capabilities, and more modalities.

Main trends:

1. Growing model scale: from 7B parameters to 70B+ parameters
2. Higher resolution: support for higher-resolution inputs (e.g., 448×448, 896×896)
3. Multimodal expansion: from image-text to video, audio, and other modalities
4. Natively multimodal architectures: no longer simply stitching together unimodal models, but designing multimodal architectures from scratch

Taking Qwen2-VL as an example, it introduced:

• native dynamic resolution support, capable of handling images of any resolution
• multimodal rotary position embedding (M-RoPE), handling images and video uniformly
• strong document understanding and OCR capabilities

3. Core Technical Principles of VLMs

3.1 Model architecture: single-stream vs. dual-stream

According to the VLP survey, VLM architectures can be classified along two dimensions:

3.1.1 From the perspective of multimodal fusion: single-stream vs. dual-stream

Single-stream architecture:

• concatenates visual and text features and feeds them into a single Transformer
• uses a unified self-attention mechanism for fusion
• more parameter-efficient, but with greater interference between modalities
• examples: VisualBERT, UNITER

Dual-stream architecture:

• feeds vision and text into separate Transformers
• achieves cross-modal interaction through cross-attention
• better preserves intra-modal information, but has more parameters
• examples: CLIP, BLIP-2

3.1.2 From the perspective of overall design: encoder-only vs. encoder-decoder

Encoder-only:

• uses only an encoder to extract cross-modal representations
• suitable for understanding tasks (e.g., image-text retrieval, visual question answering)
• examples: CLIP, ALBEF

Encoder-Decoder:

• the encoder extracts features and the decoder generates text
• suitable for generation tasks (e.g., image captioning)
• examples: BLIP, OFA

3.2 Pre-training objectives

The pre-training objective of a VLM is to learn the semantic correspondence between vision and language. According to the VLP survey, the main pre-training objectives can be divided into four categories:

3.2.1 Completion

Masked language modeling (MLM):

• randomly masks some tokens in the text and lets the model predict them based on visual information and context
• formula: $\mathcal{L}_{MLM} = -\mathbb{E}_{(v,w)\sim D} \log P(w_m|w_{\setminus m}, v)$
• used in: VisualBERT, UNITER

Masked vision modeling (MVM):

• randomly masks some image regions and lets the model predict them based on text and other visual information
• variants: masked feature regression, masked feature classification
• used in: UNITER, OSCAR

3.2.2 Matching

Image-text contrastive learning (ITC):

• pulls matching image-text pairs closer in feature space and pushes non-matching pairs apart
• uses the InfoNCE loss function
• formula: $\mathcal{L}_{ITC} = -\frac{1}{2}\mathbb{E}_{(I,T)\sim D}[CE(y^{v2t}, p^{v2t}(I)) + CE(y^{t2v}, p^{t2v}(T))]$
• used in: CLIP, ALBEF, BLIP

Image-text matching (ITM):

• a binary classification task that determines whether an image-text pair matches
• uses hard negative mining
• used in: ALBEF, BLIP

3.2.3 Temporal

Frame order modeling (FOM):

• shuffles the order of video frames and asks the model to predict the correct order
• used for video-language pre-training
• used in: VideoBERT

3.2.4 Task-specific objectives

Visual question answering (VQA):

• treats VQA as a pre-training task
• classification-based or generative answer prediction

Visual captioning (VC):

• autoregressively generates image captions
• uses a language modeling loss

3.3 The evolution of visual feature extraction

Visual feature extraction in VLMs has gone through three generations:

3.3.1 First generation: object-detection-based region features (OD-RFs)

• uses a pre-trained object detector (e.g., Faster R-CNN) to extract region features
• each region feature is a 2048-dimensional vector with bounding-box coordinates
• advantages: semantically explicit, containing object-level information
• disadvantages: computationally expensive; only a limited number of regions can be extracted
• examples: VisualBERT, UNITER, OSCAR

3.3.2 Second generation: CNN-based grid features (CNN-GFs)

• uses a CNN (e.g., ResNet) to extract grid features
• can be trained end to end
• advantages: more computationally efficient
• disadvantages: less semantically explicit than region features
• examples: PixelBERT

3.3.3 Third generation: ViT-based patch features (ViT-PFs)

• uses a Vision Transformer to split the image into a sequence of patches
• each patch serves as a token
• advantages: unified with the token format of NLP, facilitating cross-modal fusion
• examples: CLIP, BLIP, LLaVA

3.4 Training strategies: two-stage vs. three-stage

Modern VLMs usually adopt multi-stage training strategies:

3.4.1 Two-stage training (BLIP-2 as an example)

Stage one: vision-language representation learning

• freeze the visual encoder
• train the Q-Former to extract visual features most relevant to the text
• use three objectives: ITC, ITM, and ITG

Stage two: vision-to-language generative learning

• connect the Q-Former to a frozen LLM
• train the Q-Former so that its output can be understood by the LLM
• use a language modeling loss

3.4.2 Three-stage training (Qwen-VL as an example)

Stage one: pre-training

• freeze the LLM; train the visual encoder and adapter
• use large-scale image-text pair data (1.4 billion pairs)
• low-resolution input (224×224)

Stage two: multi-task pre-training

• unfreeze the LLM; train end to end
• introduce high-quality fine-grained data (VQA, OCR, grounding, etc.)
• high-resolution input (448×448)

Stage three: supervised fine-tuning

• freeze the visual encoder; fine-tune the LLM and adapter
• use instruction-following data
• enhance dialogue and instruction-understanding capabilities

4. A Detailed Look at Mainstream VLM Models

4.1 CLIP: the pioneer of cross-modal contrastive learning

Basic information:

• Release date: February 2021
• Institution: OpenAI
• Parameters: about 400 million (largest version)
• Training data: 400 million image-text pairs

Core architecture:

# CLIP 伪代码
image_features = image_encoder(images)  # ViT 或 ResNet
text_features = text_encoder(texts)      # Transformer

# 归一化
image_features = normalize(image_features)
text_features = normalize(text_features)

# 计算相似度
similarity = image_features @ text_features.T

# 对比损失
loss = cross_entropy_loss(similarity, labels)

Main capabilities:

• zero-shot image classification
• image-text retrieval
• used as a visual encoder for downstream tasks

Limitations:

• supports only understanding tasks, not generation tasks
• limited fine-grained visual understanding

4.2 BLIP/BLIP-2: unified understanding and generation

4.2.1 BLIP

Core innovations:

• MED (multimodal mixture of encoder-decoder) architecture
• CapFilt data bootstrapping method

CapFilt pipeline:

4.2.2 BLIP-2

Core innovations:

• Q-Former architecture: a lightweight query Transformer
• two-stage training strategy
• a pluggable frozen-model design

Q-Former architecture:

Performance:

• surpasses Flamingo-80B by 8.7% on VQAv2 while using only 1/54 of its trainable parameters

4.3 LLaVA: the pioneer of visual instruction tuning

Basic information:

• Release date: April 2023
• Institution: University of Wisconsin–Madison
• Core idea: extend instruction tuning from NLP to the vision-language domain

Data generation:

• uses GPT-4 to generate multimodal instruction-following data
• three types: conversation, detailed description, complex reasoning
• 158K samples in total

Architecture:

Two-stage training:

1. Feature alignment pre-training:

   • freeze the visual encoder and LLM
   • train only the projection matrix
   • use 595K image-text pairs

2. End-to-end fine-tuning:

   • freeze the visual encoder
   • fine-tune the projection layer and LLM
   • use 158K instruction samples

Performance:

• achieves 92.53% accuracy on Science QA when combined with GPT-4 (SOTA)

4.4 MiniGPT-4: strong capability from a minimalist design

Basic information:

• Release date: April 2023
• Institution: King Abdullah University of Science and Technology (KAUST)

Architecture:

Key findings:

• training only a single linear projection layer is enough to effectively align vision and language
• training requires only 4 A100 GPUs and about 10 hours

Two-stage training:

1. Stage one: pre-train on large-scale image-text pairs (5 million pairs)
2. Stage two: fine-tune on 3,500 high-quality detailed descriptions

Emergent abilities:

• detailed image description
• website code generation
• poetry writing
• recipe generation
• humor explanation

4.5 Qwen-VL: a general model for multiple languages and images

Basic information:

• Release date: August 2023
• Institution: Alibaba
• Base model: Qwen-7B

Core features:

1. Multilingual support:

   • supports both Chinese and English
   • 77.3% English and 22.7% Chinese in the training data

2. Multi-image input:

   • supports arbitrarily interleaved image-text data
   • can compare and understand multiple images

3. Fine-grained understanding:

   • high-resolution input (448×448)
   • supports text reading (OCR)
   • supports visual grounding

Three-stage training:

Performance:

• Flickr30K image captioning: 85.8 CIDEr (zero-shot SOTA)
• VQAv2: 79.5% accuracy
• RefCOCO grounding: 88.55% accuracy

5. Applications of VLMs

5.1 Image captioning and visual question answering

Image captioning:

• generating natural language descriptions for images
• applications: assisting visually impaired people, image retrieval, content moderation

Visual question answering (VQA):

• answering natural language questions based on image content
• applications: intelligent customer service, educational assistance, medical image analysis

5.2 Image-text retrieval

Image-text retrieval:

• text-to-image retrieval: finding matching images based on a text description
• image-to-text retrieval: finding relevant text descriptions based on an image
• applications: search engines, e-commerce platforms, content recommendation

5.3 Document understanding

OCR and document analysis:

• text reading: recognizing text content in an image
• table understanding: parsing table structure and content
• chart analysis: understanding the data and trends in charts
• applications: financial report analysis, contract review, invoice processing

5.4 Visual grounding and referring expressions

Visual grounding:

• locating objects in an image based on a text description
• outputs bounding-box coordinates

Referring expression comprehension:

• understanding descriptions such as “the girl in red on the left”
• applications: robot navigation, image editing, interactive AI

5.5 Creative generation

Image-to-code:

• converting hand-drawn sketches into HTML/CSS code
• applications: front-end development, prototyping

Image-to-story/poetry:

• creating stories or poems based on an image
• applications: content creation, education and entertainment

6. The Hallucination Problem: The Achilles’ Heel of VLMs

6.1 What is VLM hallucination?

According to A Survey on Hallucination in Large Vision-Language Models, VLM hallucination refers to text responses generated by the model that show incorrect perception of image content.

Examples of hallucination:

6.2 Types of hallucination

According to the hallucination survey, VLM hallucinations can be divided into three categories:

1. Object existence hallucination:

   • describing objects that do not exist in the image
   • omitting objects that do exist in the image

2. Object attribute hallucination:

   • incorrectly describing attributes of objects such as color, shape, and material

3. Object relationship hallucination:

   • incorrectly describing the spatial relationships between objects
   • incorrectly describing the interaction relationships between objects

In addition, the research also defines other types of hallucination:

• Multimodal conflict hallucination: hallucination triggered when the text input does not match the visual input
• Counter-commonsense hallucination: hallucination caused by counter-commonsense elements in the image
• Event hallucination: creating completely fabricated events or scenarios based on incorrect perception
• Numerical hallucination: failing to accurately recognize the number of specific objects in an image

6.3 Causes of hallucination

The hallucination survey analyzes the causes of hallucination from four dimensions:

6.3.1 Training data

1. Image captions are not detailed enough:

   • captions in open-source datasets are relatively coarse
   • they cannot adequately cover the image content

2. Synthetic instruction-tuning data contains noise:

   • instruction data generated with GPT-4 may contain hallucinated elements
   • research shows that about 32.6% of LLaVA’s GPT-4 synthetic data contains hallucinations

3. Statistical biases in the data:

   • uneven frequency of object categories
   • an excessively high proportion of positive responses

6.3.2 Training tasks

• a single language modeling task lacks explicit constraints on consistency between modalities
• the model focuses too much on the fluency of generated text and ignores consistency with visual information

6.3.3 Visual encoding

1. Insufficient perception of fine-grained visual information:

   • input resolution is usually below 400 pixels
   • text annotations in training data can only cover the main content rather than details

2. Visual features do not receive sufficient attention from the LLM:

   • the average attention score of visual tokens in the LLM is less than 20%
   • attention to visual tokens keeps declining as generation proceeds

6.3.4 Text generation

1. Prior knowledge bias of the LLM:

   • the model tends to rely on prior knowledge rather than actual visual evidence
   • even with noisy input images, it may still output high-confidence answers

2. Bias accumulation in autoregressive decoding:

   • small errors made early in generation are amplified in later generation
   • hallucination frequency increases with generation length

3. Inherent sampling randomness:

   • the sampling process may cause the model to occasionally output low-probability incorrect tokens

6.4 Evaluating hallucination

Hallucination evaluation benchmarks include:

6.5 Mitigating hallucination

The hallucination survey discusses mitigation strategies from five dimensions:

6.5.1 Data

• improve the diversity of instruction data
• strengthen image-text relevance
• use high-quality human-annotated data

6.5.2 Visual perception

• use higher-resolution visual encoders
• integrate multiple visual encoders
• increase the parameter size of the visual encoder

6.5.3 Training strategy

• introduce new supervision signals
• use reinforcement learning
• add task information

6.5.4 Inference

• contrastive decoding
• uncertainty estimation
• multi-step reasoning

6.5.5 Post-hoc correction

• verification with external knowledge
• self-correction mechanisms
• human feedback

7. Technical Challenges and Future Directions for VLMs

7.1 Current challenges

7.1.1 Hallucination

• as mentioned above, hallucination is the main factor limiting the application of VLMs in high-stakes domains
• more reliable evaluation and mitigation methods are needed

7.1.2 Fine-grained understanding

• current VLMs still have limited ability to understand details
• high-resolution processing requires more computational resources

7.1.3 Multimodal fusion

• how to better fuse vision, language, audio, and other modalities
• how to handle information conflicts between modalities

7.1.4 Computational efficiency

• inference costs for large-scale VLMs are high
• more efficient model compression and acceleration techniques are needed

7.2 Future directions

7.2.1 Natively multimodal architectures

• design architectures that support multiple modalities from scratch, rather than simply stitching together unimodal models
• handle text, images, video, audio, and other modalities in a unified way

7.2.2 World models

• build VLMs that can understand the physical world
• support causal reasoning and physical simulation

7.2.3 Embodied intelligence

• combine VLMs with robots
• support tasks such as visual navigation and object manipulation

7.2.4 Personalization and interpretability

• VLMs that support user personalization
• provide interpretable reasoning processes

8. Conclusion

Vision-language models (VLMs) represent an important advance in artificial intelligence. They allow machines to “see” images and “understand” language at the same time, opening a new paradigm for human-computer interaction.

In terms of development history:

• from early dual-tower architectures to CLIP’s contrastive learning revolution
• from BLIP’s unified framework to LLM-powered visual assistants
• VLMs are moving toward larger scale, stronger capabilities, and more modalities

In terms of technical principles:

• the core of VLMs lies in cross-modal alignment and fusion
• pre-training objectives, model architectures, and training strategies are all evolving
• hallucination remains the main current challenge

In terms of applications:

• basic tasks such as image captioning, visual question answering, and image-text retrieval are already relatively mature
• advanced tasks such as document understanding, visual grounding, and creative generation are developing rapidly
• they will play an important role in more fields in the future

For beginners, the key to understanding VLMs is to grasp the core idea of cross-modal alignment; for technical practitioners, it is necessary to deeply understand the strengths and weaknesses of various architecture designs and training strategies. In any case, VLMs are a field worth continued attention. They are redefining the way humans interact with computers and bringing us one step closer to true general artificial intelligence.

References

[1]: What Are Vision-Language Models (VLMs)? | IBM

[2]: VLM overview

[3]: VLP: A Survey on Vision-Language Pre-training

[4]: Learning Transferable Visual Models From Natural Language Supervision (CLIP)

[5]: Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision (ALIGN)

[6]: BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation

[7]: Visual Instruction Tuning (LLaVA)

[8]: BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models

[9]: Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution

[10]: A Survey on Hallucination in Large Vision-Language Models: Causes, Evaluation, and Mitigation

Further reading:

• Guide to Vision-Language Models (VLMs) | Encord
• Understanding Vision-Language Models | Hugging Face
• What Are Vision Language Models | NVIDIA

References and Reading Order

Start with the original papers to understand the structure, then compare against specific model cards and code; this makes it easier to distinguish the method itself from the capabilities of a particular version.

• CLIP: Learning Transferable Visual Models From Natural Language Supervision: compare with the dual-tower architecture and image-text alignment sections above.
• BLIP-2: see how the frozen visual encoder and the language model are connected.
• LLaVA: Visual Instruction Tuning: see the visual instruction data and multimodal dialogue training.
• Qwen3-VL Technical Report: look at visual understanding and training design in the context of a newer model family.
• Qwen3-VL-2B-Instruct model card: check dependencies, input processing, and the corresponding version when deploying.

The example dialogues in this article are used to explain the tasks; they are not measured results from a fixed model. Paper performance, official capability descriptions, and local run results need to be recorded separately; to actually run a small model, you can start with the local model notes.