Diffusion Models

Forward process: each step adds a small Gaussian noise scaled by β_t (the noise schedule). β_t increases from ~0.0001 to ~0.02 over T steps. By step T, the signal-to-noise ratio → 0.

The model ε_θ takes a noisy image x_t and timestep t, predicts the noise ε that was added. MSE against the true noise. Remarkably simple — yet produces state-of-the-art image quality.

DDPM training loop skeleton (simplified)

import torch
import torch.nn.functional as F

def ddpm_train_step(model, x_0, noise_schedule):
    """One training step for a diffusion model."""
    batch_size = x_0.shape[0]
    device = x_0.device

    # 1. Sample random timestep for each image in the batch
    t = torch.randint(0, noise_schedule.T, (batch_size,), device=device)

    # 2. Sample noise
    eps = torch.randn_like(x_0)

    # 3. Create noisy image x_t via closed-form reparameterization
    alpha_bar_t = noise_schedule.alpha_bar[t].view(-1, 1, 1, 1)
    x_t = torch.sqrt(alpha_bar_t) * x_0 + torch.sqrt(1 - alpha_bar_t) * eps

    # 4. Predict the noise (U-Net or DiT backbone)
    eps_pred = model(x_t, t)

    # 5. Simple MSE loss on predicted vs actual noise
    loss = F.mse_loss(eps_pred, eps)
    return loss

c = text condition, ∅ = null condition, s = guidance scale (typically 7–14). Higher s → stronger prompt adherence but less diversity and occasional artifacts ("oversaturation"). Lower s → more creative but may ignore the prompt.

1. What is the difference between DDPM (Denoising Diffusion Probabilistic Models) and DDIM (Denoising Diffusion Implicit Models)? (Answer: DDPM: adds Gaussian noise over T=1000 steps; generation reverses this by predicting and removing noise at each step. Requires all T steps → slow (1000 forward passes). DDIM: reformulates the denoising as a deterministic ODE (no stochastic sampling). Enables skipping steps — generate in 20–50 steps instead of 1000 with minimal quality loss. Same noise model, different sampling strategy. DDIM is the standard for fast sampling in Stable Diffusion (--steps 20 uses DDIM scheduler).)
2. What is the role of the text encoder in text-to-image diffusion models? (Answer: The text encoder (CLIP, T5, or similar) converts the text prompt into a sequence of embeddings. These embeddings are injected into the denoiser (U-Net or DiT) via cross-attention — each image patch attends to the text embeddings to guide denoising. The text encoder determines which semantic concepts the model can represent. Stable Diffusion uses CLIP; DALL-E 3/Imagen uses T5. Text encoder quality strongly determines prompt adherence — this is why jailbreaking SD often attacks the text encoding step.)
3. What is the forward process in DDPM and why must each step be small? (Answer: Forward process: q(x_t|x_{t-1}) = N(√(1-β_t) x_{t-1}, β_t I). Each step adds a small amount of Gaussian noise (β_t is small, e.g., 0.0001 to 0.02). After T=1000 steps, x_T ≈ N(0, I) — pure Gaussian noise. Steps must be small so the reverse step (denoising) is also approximately Gaussian — making it learnable. If β_t is large (few big steps), the reverse distribution is not Gaussian and the learned denoiser cannot model it.)
4. What is classifier-free guidance (CFG) scale and why does higher CFG improve prompt adherence but reduce diversity? (Answer: CFG output = uncond_pred + scale × (cond_pred - uncond_pred). Scale=1: pure conditional generation. Scale=7.5 (typical): conditional and unconditional predictions are extrapolated toward the conditional — stronger prompt adherence. High scale (15+): over-saturated, oversharped images, less natural variation. The CFG formula amplifies the difference between conditional and unconditional predictions — high scale means any deviation from the prompt is strongly penalised, reducing diversity. Images look 'more like the prompt' but less like natural photographs.)
5. What is ControlNet and how does it add spatial control to diffusion models? (Answer: ControlNet (Zhang et al. 2023): adds a trainable copy of the U-Net encoder that processes a control signal (edge map, depth map, pose skeleton, segmentation map). The control encoder's features are added to the main U-Net's features at corresponding resolutions. The original U-Net weights are frozen; only the control encoder is trained. This preserves the original model's generation quality while adding spatial conditioning. Multiple ControlNets can be combined: e.g., pose control + depth control simultaneously for precise scene composition.)