Multimodal AI

Many senses, one model

A toddler learns what a dog is by seeing one, hearing it bark, feeling its fur and hearing an adult say “dog”, all at once. For most of AI’s history, models got one of those channels. Multimodal models get several, and the trick that makes it work is surprisingly uniform: turn everything into a sequence of vectors.

What is multimodal AI?

A modality is a kind of input or output: text, images, audio, video, and beyond that depth maps, sensor readings or robot actions. A multimodal model takes in or produces more than one. You use one when you photograph a maths problem and ask for help, when a phone describes a scene aloud for a blind user, or when you talk to an assistant and it talks back without a text step in between.

How does it work, in one sentence?

Each modality gets an encoder that chops it into pieces and maps each piece to a vector in a shared space, and a transformer processes the resulting sequence without caring where each vector came from. Text becomes subword tokens, images become patches, and audio becomes short frames of a spectrogram.

Why does it matter?

Most of the world’s information is not text: photographs, diagrams, screens, speech, video. A model that can only read captions about the world misses most of it. Multimodality also unlocks agents: a model that can read a screenshot can operate software built for humans. And modalities teach each other. Text supervision gave vision models an open vocabulary, which is the story of CLIP at the heart of this lesson.

Key takeaways

• Multimodal models read or write more than one kind of data: text, images, audio, video.
• The shared trick: each modality is cut into pieces, each piece becomes a vector, and one transformer reads the sequence.
• It matters because most information is not text, and because agents need to see screens.

How images become tokens

Convolutional networks dominated vision for a decade. Then in 2020 a Google team showed that you could cut an image into squares, treat each square like a word, and feed the sequence to a plain transformer.

What is a Vision Transformer?

The Vision Transformer (ViT) takes a 224 × 224 image and slices it into a grid of 16 × 16 pixel patches: 14 × 14 = 196 patches. Each patch is flattened into 16 × 16 × 3 = 768 numbers, multiplied by a learned matrix to produce an embedding, and given a position embedding so the model knows where it came from. The title says it: an image is worth 16 × 16 words.

How does it work?

From there it is an ordinary transformer. Every patch attends to every other patch, so a patch showing a cat’s ear can gather context from the patch showing its tail in the very first layer, something a CNN only achieves after many layers of growing receptive fields. The price is a weaker built-in bias: CNNs assume nearby pixels matter together, while ViTs must learn that from data. The original paper found ViTs only overtook strong CNNs when pre-trained on very large datasets (hundreds of millions of images).

Halving the patch size quadruples the number of tokens and multiplies the attention pairs by sixteen. Small patches see fine detail, such as small text in a screenshot, but cost far more. Every vision-language model makes this trade-off, which is why the VLM section returns to token budgets.

Key takeaways

• A ViT cuts an image into fixed-size patches, flattens each one and projects it into an embedding: patches are tokens.
• Self-attention lets every patch see every other patch from the first layer, at a cost that grows with the square of the patch count.
• Patch size trades detail for compute. That trade shapes every modern vision-language model.

Contrastive learning and CLIP

The internet is full of images with text next to them: alt text, captions, product titles. In 2021 OpenAI trained a model on 400 million of those pairs with one simple goal: make each image’s vector point the same way as its own caption’s vector, and away from everyone else’s.

What is contrastive learning?

Contrastive learning teaches a model which things belong together by showing it matched pairs and many mismatched ones. CLIP (Contrastive Language-Image Pre-training) has two encoders, one for images and one for text, that map into the same embedding space. In each training batch of N image-caption pairs there is one right match per image and N − 1 wrong ones. CLIP used batches of 32,768, so every image had 32,767 captions to be pushed away from.

How does it work?

Normalise all the vectors to unit length and compute the N × N matrix of cosine similarities between every image and every caption. Divide by a temperature τ to get logits. Then apply softmax along each row (for each image, which caption?) and along each column (for each caption, which image?), and use cross-entropy with the diagonal as the correct answer. The loss is the average of the two directions.

Symmetric InfoNCE loss. s_ij is the cosine similarity of image i and caption j; τ is a learned temperature (CLIP initialised it to 0.07).

As training runs, each square swings round to line up with its circle and the pairs spread out around the circle, because being close to the wrong caption is penalised. With a high temperature the loss stalls well above zero even when every pair is matched: dividing by a large τ squashes the logits, so softmax can never become confident. A small τ makes the model decisive, but the gradients get large and training becomes jumpy. That is why CLIP learns τ rather than fixing it.

Why the shared space is so useful

Once images and text live in one space, many tasks become nearest-neighbour lookups. Text-to-image search is “embed the query, find the nearest image vectors”. Duplicate detection is “find images whose vectors are nearly identical”. CLIP’s text encoder also became the component that tells many image generators what the prompt means.

Key takeaways

• CLIP trains an image encoder and a text encoder so matching pairs point the same way and mismatched pairs do not.
• The loss is a softmax over an N × N similarity matrix, applied in both directions; the rest of the batch supplies the negatives.
• Temperature controls how sharp the softmax can get. Too high and the model can never be confident.

Classification by description

A normal image classifier can only output the classes it was trained on. CLIP can classify into categories nobody planned for, because you describe the categories in words at test time.

What is zero-shot classification?

Zero-shot means classifying without any labelled training examples for the task. With CLIP you write one caption per class (“a photo of a dog”, “a photo of a cat”), embed each with the text encoder, embed the image, and pick the caption with the highest cosine similarity. The class list is a runtime input. Change the list and you have a different classifier, with no retraining.

It works remarkably well. Zero-shot CLIP matched the ImageNet accuracy of the original supervised ResNet-50, 76.2%, without using any of ImageNet’s 1.28 million labelled training images.

How does it work?

It is the same softmax as in training, with the candidate captions playing the role of the batch. The image vector is compared with each caption vector, the similarities are scaled (CLIP’s learned scale capped at 100) and softmax turns them into probabilities. Prompt wording matters: the CLIP authors found that the template “A photo of a {label}.” beat the bare label, and that averaging over dozens of templates helped further.

The lesson hiding in this demo: softmax probabilities are relative to the labels you offered. With apple, tennis ball, dog and car on offer, the apple wins comfortably. Add pear, a plausible neighbour, and some probability shifts to it. Remove every fruit and the tennis ball wins, because round beats nothing. A zero-shot classifier never says “none of these”, unless you give it that option as a label.

Key takeaways

• Zero-shot classification embeds one caption per class and picks the most similar: the class list is just text.
• Zero-shot CLIP matched a supervised ResNet-50 on ImageNet (76.2%) without seeing any ImageNet training labels.
• Probabilities are relative to the labels offered. Choosing and wording labels is part of building the classifier.

Vision-language models

CLIP can tell you which caption fits an image best. It cannot describe a chart, answer a follow-up question or read a receipt out loud. For that you connect vision to a language model.

What is a vision-language model?

A vision-language model (VLM) takes images and text as input and generates text. You can ask it anything about an image: what is unusual here, what does this error dialog say, which of these two charts shows faster growth. Most chat assistants have worked this way since 2024.

How does it work?

There are three main designs, roughly in the order they appeared:

• Flamingo (DeepMind, 2022) kept a pre-trained vision encoder and a pre-trained language model frozen, compressed each image into 64 visual tokens with a “perceiver resampler”, and inserted new cross-attention layers so the language model could look at them. It could learn new tasks from a handful of examples in the prompt.
• LLaVA (2023) showed that something much simpler works: take CLIP’s patch features, pass them through a small projection so they are the same size as word embeddings, and put them in the prompt ahead of the text. It was fine-tuned on 158,000 instruction-following examples that GPT-4 wrote from image captions. Most open VLMs still follow this recipe.
• Natively multimodal models such as Gemini (2023) and GPT-4o (2024) are trained on interleaved text, images and audio from the start, rather than bolting an encoder onto a finished language model. GPT-4o’s system card describes one network trained end to end across text, vision and audio. By 2025 and 2026 this became the default for frontier assistants.

The token budget

Images are expensive. A 1080p screenshot cut into 14-pixel patches is over 10,000 tokens, far more than this whole section of text. Modern VLMs process images at their native aspect ratio and then shrink the sequence. Qwen2-VL, for example, merges each 2 × 2 block of neighbouring patch tokens into one, so a 224 × 224 image costs 64 tokens plus two markers.

A full-resolution phone photo is over 15,000 tokens even with merging. Real systems cap the pixel count, so fine print in a large photo may simply be downsampled away before the model sees it. If a VLM cannot read small text, crop first.

The other direction: generating images

Multimodal also runs the other way: text in, pixels out. Diffusion models learn to remove noise step by step, guided by a text embedding that often comes from a CLIP-style encoder. Some native multimodal models now generate images as part of the same token stream they use for text. The Generative AI lesson covers how, and the Diffusion Lab lets you watch denoising happen.

Key takeaways

• VLMs connect a vision encoder to a language model: via cross-attention (Flamingo), a projection into the prompt (LLaVA) or native joint training (Gemini, GPT-4o).
• Image tokens are the main cost. Dynamic resolution and token merging keep the budget manageable.
• Generation runs the other way: text embeddings guide diffusion models, and some native models output images directly.

Audio, speech and video

Sound is a one-dimensional wave, sampled thousands of times a second. To a transformer, raw samples are a hopeless sequence: one second of speech at 16 kHz is 16,000 numbers. The first job is to turn it into something that looks more like an image.

What does a speech model actually see?

Most speech models start from a log-Mel spectrogram. Slide a short window along the audio, measure how much energy each frequency has in that window with a Fourier transform, group the frequencies into bands spaced the way human hearing is (the Mel scale, which gives low pitches more resolution), and take the logarithm, since loudness is perceived logarithmically. The result is a picture: time along one axis, pitch along the other, brightness for energy.

Whisper (OpenAI, 2022) uses exactly this: 80 Mel bands, a 25 ms window moved 10 ms at a time, over 30-second chunks. A transformer encoder reads the spectrogram and a decoder writes the transcript. Its strength came from data rather than architecture: 680,000 hours of audio with loosely matching transcripts from the web, which made it unusually robust to accents and background noise.

Vowels are identified mostly by where their formants sit: “ah” has its first two bright bands close together low down, “ee” splits them far apart. A model reading spectrograms learns these patterns the way a ViT learns edges and textures.

From pipelines to native speech

Video

Video adds time. Understanding models usually sample frames and encode each as image tokens, which is why long videos are costly. Generation models extend patches into spacetime patches, small cubes of pixels across several frames, as OpenAI described for Sora in 2024. In 2025, video generation gained sound: Google DeepMind’s Veo 3 (May 2025) and OpenAI’s Sora 2 (September 2025) generate synchronised dialogue, sound effects and ambience together with the pictures.

Key takeaways

• Audio becomes a log-Mel spectrogram: an image of energy over time and pitch, which a transformer reads in frames.
• Whisper’s robustness came from 680,000 hours of weakly labelled audio; native speech models skip the transcript entirely.
• Video models use frames or spacetime patches. Since 2025, leading generators produce synchronised sound as well.

Where multimodal models fail

Multimodal models can explain a meme and summarise a lecture slide, then fail to count five overlapping circles. Their failures are strange because their strengths are strange.

What are the typical failure modes?

• Counting and fine geometry. In “Vision language models are blind” (2024), leading VLMs struggled with tasks a child finds trivial: counting where two lines cross, telling whether two circles overlap, counting nested squares.
• Spatial relations. Left versus right, above versus below and “behind” are often confused, because captions on the web rarely state them precisely.
• Hallucinated text and objects. Asked to read blurry text, a VLM may produce plausible text that is not there, or describe objects that commonly appear in similar scenes.
• Reading beats seeing. Text visible in an image can dominate the visual evidence.

How does a typographic attack work?

Goh et al. (2021) looked inside CLIP and found multimodal neurons: units that respond to a concept whether it appears as a photo, a drawing or the written word. A “Spider-Man” neuron fired for photos of the costume, comic drawings and the text “spider”. The same property creates a vulnerability. Stick a paper label reading “iPod” on a Granny Smith apple and CLIP classifies the apple as an iPod. The authors called these typographic attacks: they need nothing more than a pen.

Why does it matter?

These models increasingly read screens for agents, check documents and describe the world for people who cannot see it. A model that can be redirected by text in an image is a prompt-injection surface: a web page screenshot can carry instructions just as page text can. Evaluate multimodal systems on your own images, include adversarial and ambiguous cases, and cross-check high-stakes readings (prices, dosages, measurements) with specialised tools.

Key takeaways

• VLMs are weak at counting, fine geometry and spatial relations, and can hallucinate text and objects that fit the scene.
• CLIP’s multimodal neurons tie images and words together, so a written label can override what the model sees.
• Treat multimodal outputs as claims to verify, especially when agents act on screenshots or documents.

Check your understanding

Six scenarios from real multimodal deployments. Pick the answer you would bet a project on.

You deploy CLIP as a zero-shot classifier with the labels {cat, dog}. A user uploads a photo of a fox, and the model says "dog" with 94% confidence. What is going on?

References and further reading

Papers, reports and articles cited in this lesson. The Distill article on multimodal neurons is especially worth reading in full for its interactive figures.

References

1. [1]

   An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale(opens in a new tab)

   Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, et al., 2020

   The Vision Transformer (ViT): images as sequences of patches; ICLR 2021.

2. [2]

   Learning Transferable Visual Models From Natural Language Supervision(opens in a new tab)

   Alec Radford, Jong Wook Kim, Chris Hallacy, et al., 2021

   CLIP: contrastive pre-training on 400 million image-text pairs; zero-shot transfer.

3. [3]

   Flamingo: a Visual Language Model for Few-Shot Learning(opens in a new tab)

   Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, et al., 2022

4. [4]

   Visual Instruction Tuning(opens in a new tab)

   Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee, 2023

   LLaVA: a CLIP vision encoder connected to a language model by a projection, trained on GPT-4-generated instruction data.

5. [5]

   Gemini: A Family of Highly Capable Multimodal Models(opens in a new tab)

   Gemini Team, Google, 2023

6. [6]

   GPT-4o System Card(opens in a new tab)

   OpenAI, 2024

   An autoregressive omni model trained end to end across text, vision and audio.

7. [7]

   Qwen2-VL: Enhancing Vision-Language Model’s Perception of the World at Any Resolution(opens in a new tab)

   Peng Wang, Shuai Bai, Sinan Tan, et al., 2024

8. [8]

   Robust Speech Recognition via Large-Scale Weak Supervision(opens in a new tab)

   Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, Ilya Sutskever, 2022

   Whisper: speech recognition trained on 680,000 hours of weakly labelled web audio.

9. [9]

   Video generation models as world simulators(opens in a new tab)

   Tim Brooks, Bill Peebles, et al. (OpenAI), 2024

10. [10]

    Veo(opens in a new tab)

    Google DeepMind, 2025

11. [11]

    Sora 2 is here(opens in a new tab)

    OpenAI, 2025

12. [12]

    Vision language models are blind(opens in a new tab)

    Pooyan Rahmanzadehgervi, Logan Bolton, Mohammad Reza Taesiri, Anh Totti Nguyen, 2024

    Simple low-level visual tasks (counting line intersections, overlapping circles) that strong VLMs fail.

13. [13]

    Multimodal Neurons in Artificial Neural Networks(opens in a new tab)

    Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, Chris Olah, 2021

    Distill article finding neurons in CLIP that respond to a concept whether shown as a photo, a drawing or written text, and the typographic attacks this enables.