The shift from specialist to unified models

The key architectural insight enabling native multimodal generation: treating all modalities as sequences of tokens in a shared representation space. Images are discretised into patch tokens via a VQVAE or similar encoder; audio is tokenised with EnCodec or a similar audio codec; text is tokenised with a BPE vocabulary. All token sequences share the same transformer architecture, enabling the model to attend across modalities — reasoning jointly about image content and text meaning rather than processing them in separate passes.

What multimodal models can do in 2026

• Image understanding: Describe image contents, answer questions about scenes, read text in images (OCR), identify objects, analyze charts and graphs, explain scientific diagrams.
• Image generation (selected models): the GPT line generates images natively in-model (GPT-4o's image generation replaced the separate DALL-E 3 pipeline in 2025), matching precise textual descriptions with high spatial accuracy.
• Audio understanding: Gemini 2.5 Pro transcribes, translates, and reasons about spoken audio. Whisper (OpenAI) provides state-of-the-art speech recognition across 99 languages.
• Video understanding: Gemini 2.5 Pro processes long video clips (up to 1 hour with 1M token context) — describing events, answering questions about what happened, identifying objects across scenes.
• Cross-modal reasoning: 'Here is a photo of a circuit board. Here is the schematic for what it should look like. What components are missing?' — a task requiring joint visual and technical text reasoning.
• Document AI: Understanding PDFs with mixed text, tables, figures, and handwriting as unified structured documents rather than separate elements.

Model selection by modality task: In 2026: Claude leads on document analysis and visual reasoning tasks. Gemini 3 Pro leads on long video understanding (1M token context) and multilingual audio. The GPT-5 line leads on native image generation and spatial reasoning in images. For purely image-to-text OCR tasks, Google Cloud Vision API remains cheaper and faster than full multimodal LLM inference.

How multimodal models actually work: fusion and the shared token space

The mechanism behind every unified model is one idea: make everything a token. A transformer does not care whether a vector came from a word, a patch of pixels, or 20 milliseconds of audio — it only needs a sequence of vectors of the same width. So each modality gets an encoder that converts it into that shared space, and self-attention then reasons across all of them at once.

Where the modalities meet is the design decision that separates the generations of models:

• Late fusion — each modality is processed by a separate model and only the final outputs are combined (a captioner describes an image, an LLM reads the caption). Simple, but the LLM only ever sees a lossy text summary; anything the captioner omitted is gone forever.
• Early / unified fusion — every modality is projected into the same token space and fed into one transformer from the first layer. Attention can relate a specific pixel patch to a specific word directly. This is what "natively multimodal" means, and why GPT-4o-class models can reason about a chart's fine detail rather than a description of it.
• Cross-attention (adapter) fusion — the middle ground popularized by Flamingo and used by many open models: a frozen LLM keeps its text stack, and image features are injected through added cross-attention layers. Cheaper to train than a from-scratch unified model, but the vision pathway remains bolted on.

Why generating images is harder than reading them: Understanding is easy to unify — every modality just becomes input tokens. Generation is where architectures diverge, because the model must emit pixels, not read them. Two approaches dominate: discrete tokens (a VQ-VAE/VQ-GAN quantizes images into a codebook, so the transformer literally predicts "image tokens" the same way it predicts words — one model, one objective) and diffusion handoff (the transformer emits a conditioning embedding that a separate diffusion decoder renders). The industry moved from the second toward the first, which is exactly why native image generation improved so sharply at following precise instructions — the same attention that understands your sentence is now choosing the image tokens.

Practice questions

1. What is the key architectural difference that allows GPT-4V to understand images but not generate them, while GPT-4o generates both? (Answer: GPT-4V uses a CLIP-based vision encoder that converts images to token embeddings fed into the LLM — one-directional (image in, text out). GPT-4o uses a unified token space where both image patches and text are represented as tokens in the same vocabulary, with a diffusion decoder head for image generation. Bidirectional multimodal transformers require training both understanding and generation objectives simultaneously on shared representations.)
2. What is the 'tokenization' approach for image patches in a multimodal transformer? (Answer: Images are divided into fixed-size patches (16×16 or 32×32 pixels). Each patch is encoded into a fixed-dimension embedding via a linear projection or a small CNN (ViT approach). These patch embeddings are treated as tokens — just like text tokens — in the transformer's attention mechanism. A 224×224 image with 16×16 patches becomes 196 image tokens. The transformer can then attend across image tokens and text tokens jointly.)
3. Why is cross-modal alignment (CLIP training) important before multimodal fine-tuning? (Answer: CLIP trains image and text encoders to produce compatible embeddings: image of a dog and text 'a golden retriever' should have similar vector representations. Without this alignment, image embeddings and text embeddings exist in separate spaces — the LLM cannot relate visual concepts to language concepts. CLIP pretraining on 400M image-text pairs creates a shared semantic space, making it possible to fine-tune on relatively small amounts of multimodal data.)
4. What tasks genuinely require multimodal models vs tasks that could be solved with text alone? (Answer: Genuinely multimodal: reading text in images (OCR in context), analyzing medical images, interpreting charts and graphs, grounding spatial relationships in photos, video understanding, image generation from prompts. Text-only alternatives work for: describing images from captions (if captions exist), content moderation (text-only signals often sufficient), translation, summarization. Key test: does solving the task require interpreting raw pixels/audio/video?)
5. What is the 'hallucination' problem specific to vision-language models (VLMs)? (Answer: VLMs describe visual content that is not present in the image — they 'hallucinate' objects, text, or relationships. Example: describing a stop sign as a yield sign, inventing text that is not in the image, claiming people are smiling when they have neutral expressions. This happens because LLM language priors are very strong — if a scene looks like a kitchen, the model may add expected kitchen objects not actually visible. Evaluation benchmarks like POPE measure VLM hallucination specifically.)