Multimodal AI

1. Image encoder: A Vision Transformer (ViT) divides the image into 16×16 or 32×32 pixel patches. Each patch → linear embedding → positional encoding → Transformer. Output: a sequence of patch embeddings with the same dimension as text tokens.
2. Audio encoder: A log-mel spectrogram is computed from the raw waveform, then processed by a Transformer encoder (e.g., Whisper's encoder). Output: a sequence of audio embeddings.
3. Fusion: Image/audio embeddings are concatenated or interleaved with text token embeddings. The combined sequence is passed through the main language model.
4. Training: Models are trained on paired multimodal data (image-caption pairs, video-transcript pairs) using a contrastive or generative objective to align representations.

Sending an image to Claude with a text question

import anthropic
import base64
from pathlib import Path

client = anthropic.Anthropic()

def ask_about_image(image_path: str, question: str) -> str:
    image_data = base64.standard_b64encode(
        Path(image_path).read_bytes()
    ).decode("utf-8")

    # Detect format from extension
    ext = Path(image_path).suffix.lower()
    media_type = {".jpg": "image/jpeg", ".png": "image/png",
                  ".gif": "image/gif", ".webp": "image/webp"}[ext]

    response = client.messages.create(
        model="claude-sonnet-4-6",
        max_tokens=1024,
        messages=[{
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "source": {
                        "type": "base64",
                        "media_type": media_type,
                        "data": image_data,
                    },
                },
                {"type": "text", "text": question}
            ],
        }]
    )
    return response.content[0].text

# Example uses
answer = ask_about_image("textbook_page.jpg",
    "Explain the diagram on this page and summarize the key concept.")
print(answer)

1. What is the difference between late fusion and early fusion in multimodal AI systems? (Answer: Late fusion: each modality is processed independently through its own encoder; outputs are combined (concatenated, averaged, or learned weighted sum) at the final decision layer. Simple, modular, each encoder can be optimised separately. Misses cross-modal interactions during processing. Early fusion: raw inputs from multiple modalities are combined before or early in the neural network. Enables cross-modal feature learning. Harder to train (requires paired multimodal data), but learns richer joint representations.)
2. What is cross-modal attention and why is it central to transformers like BLIP-2? (Answer: Cross-modal attention: queries from one modality attend to keys/values from another. In BLIP-2: the Querying Transformer (Q-Former) has learnable query tokens that attend (via cross-attention) to frozen image encoder outputs — extracting visual information relevant to the text. The visual tokens produced by Q-Former bridge the frozen image encoder and frozen LLM. Cross-modal attention is what allows the LLM to condition text generation on image content without retraining either component.)
3. What is the grounding problem in multimodal AI? (Answer: Grounding: connecting language symbols to perceptual content. A model that understands 'red apple' should link the word 'red' to specific wavelengths of reflected light and the visual appearance of that colour. Without grounding, language models manipulate symbols without genuine perceptual reference. Multimodal models achieve partial grounding by training on image-text pairs — the model learns that 'red apple' co-occurs with images containing certain visual patterns. CLIP and similar contrastive models achieve strong perceptual grounding.)
4. What are the evaluation challenges specific to multimodal generation (image + text)? (Answer: Text evaluation: BLEU, ROUGE, BERTScore — standard NLP metrics. Image-text alignment: CLIPScore measures cosine similarity between generated image and text prompt embeddings. Visual quality: FID (Fréchet Inception Distance) measures distributional similarity to real images. Compositional accuracy: does the image correctly show 'a red ball to the LEFT of a blue cube'? Hard to measure automatically. Human evaluation: costly gold standard. Current models score well on CLIPScore but still struggle with precise spatial relationships and accurate text rendering.)
5. What is the difference between vision-language models (VLMs) for understanding vs generation? (Answer: VLMs for understanding: image → model → text. Tasks: visual QA, image captioning, optical character recognition, chart understanding. Examples: LLaVA, Idefics, PaliGemma, Claude Vision. Architecture: vision encoder + LLM. VLMs for generation: text → model → image. Tasks: text-to-image synthesis. Examples: DALL-E 3, Stable Diffusion, Midjourney. Architecture: text encoder + diffusion model or autoregressive image generator. Unified models (GPT-4o, Gemini 2.5): both input and output images, bridging understanding and generation in one system.)