Stable Diffusion

The VAE bottleneck: encoder ℰ compresses image x to latent z; decoder 𝒟 reconstructs x from z. Diffusion runs entirely on z.

Run Stable Diffusion XL locally with diffusers — complete working example

from diffusers import StableDiffusionXLPipeline
import torch

# Load SDXL — downloads ~7GB on first run
pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
    use_safetensors=True,
    variant="fp16",
)
pipe = pipe.to("cuda")  # requires NVIDIA GPU with 8GB+ VRAM

# Optionally enable memory savings
pipe.enable_xformers_memory_efficient_attention()

image = pipe(
    prompt="A photorealistic portrait of a student studying with AI, warm library lighting, shallow depth of field",
    negative_prompt="blurry, distorted, low quality, watermark",
    height=1024,
    width=1024,
    num_inference_steps=30,   # more steps = higher quality, slower
    guidance_scale=7.5,       # how closely to follow the prompt (7–12 typical)
    generator=torch.Generator("cuda").manual_seed(42),  # reproducible seed
).images[0]

image.save("output.png")

1. What is the advantage of latent diffusion over pixel-space diffusion for image generation? (Answer: Latent diffusion operates in a compressed latent space (e.g., 64×64 latents for a 512×512 image via the VAE encoder — 64× fewer pixels to process). Diffusion steps run at this compressed resolution, dramatically reducing compute. The VAE decoder only runs once to expand to pixel space. Pixel-space diffusion (e.g., DALL-E 2's prior) runs diffusion at full resolution — 64× more compute per step. Stable Diffusion's efficiency comes almost entirely from this latent compression trick.)
2. What is classifier-free guidance (CFG) in Stable Diffusion and what does the CFG scale control? (Answer: CFG trains the model both with and without text conditioning. At inference, the denoised output is interpolated: output = uncond_output + scale × (cond_output - uncond_output). High CFG scale (7–15): strong adherence to prompt, more saturated/artistic results, less diversity. Low CFG scale (1–3): more random/creative, weaker prompt following. CFG scale = 7.5 is the typical default. Scale > 20 often produces overexposed, artifacted images.)
3. What is the role of the negative prompt in Stable Diffusion and how does it work mathematically? (Answer: Negative prompt provides the 'unconditional' guidance direction in CFG. Instead of guiding away from pure noise, the model guides away from the negative prompt embedding. Output = neg_prompt_embed + scale × (pos_prompt_embed - neg_prompt_embed). Common negatives: 'blurry, low quality, watermark, deformed hands' steer the denoising away from those distributions. Technically the negative prompt is the second text embedding in the CFG formula.)
4. What architectural change does Flux (2024) make over Stable Diffusion XL? (Answer: Flux replaces the U-Net denoiser with a Diffusion Transformer (DiT) using Multi-Modal Diffusion Transformer (MMDIT) blocks that perform self-attention jointly over text tokens and image patch tokens. This tight integration of text and image context (vs SD's cross-attention from separate streams) gives Flux dramatically better text rendering and spatial accuracy. Flux also uses flow matching instead of DDPM denoising — simpler, faster training.)
5. Why do Stable Diffusion models sometimes generate images with 6 fingers or anatomically wrong hands? (Answer: Hands are statistically rare and complex in training data — they appear in many orientations and are often partially occluded, blurry, or cropped in photos. The model lacks strong priors about hand anatomy compared to faces. Additionally, pixel-level fine details are hardest to learn from denoising. Current SDXL and Flux models have improved hand generation significantly through curated training data, but complete reliability still requires ControlNet or post-processing for professional use.)