Dropout: training an ensemble implicitly

Dropout (Srivastava et al., 2014): during each training step, randomly set each neuron's output to zero with probability p (typically 0.1–0.5). At test time, all neurons are active but outputs are scaled by (1−p) to preserve expected activation magnitude.

\tilde{h}_i = \begin{cases} 0 & \text{with probability } p \\ h_i / (1-p) & \text{with probability } 1-p \end{cases}

Implicit ensemble intuition: With n dropout-capable neurons, there are 2ⁿ possible subnetworks. Each training step samples one. At test time, running the full network with scaled weights approximates averaging over all 2ⁿ subnetworks simultaneously — cheap ensemble learning. This is why dropout is so effective as regularization.

Batch Normalization (BatchNorm)

BatchNorm (Ioffe & Szegedy, 2015): normalize each feature's activations across the current mini-batch to zero mean and unit variance, then apply learned scale and shift:

\hat{x}_i = \frac{x_i - \mu_\mathcal{B}}{\sqrt{\sigma^2_\mathcal{B} + \varepsilon}}, \qquad y_i = \gamma\,\hat{x}_i + \beta

import torch.nn as nn

# BatchNorm: normalizes over batch + spatial dims — standard in CNNs
# Note: behavior differs at train vs eval time (uses running stats at eval)
bn = nn.BatchNorm2d(num_features=64)   # for conv feature maps
bn1d = nn.BatchNorm1d(num_features=256) # for linear layer outputs

# LayerNorm: normalizes over feature dims per sample — standard in Transformers
# Same behavior at train and eval — no batch dependency
ln = nn.LayerNorm(normalized_shape=768)  # normalize 768-dim hidden state

# RMSNorm (used in LLaMA, Mistral) — simpler, no mean centering
# Not in standard PyTorch — commonly implemented as:
class RMSNorm(nn.Module):
    def __init__(self, dim):
        super().__init__()
        self.weight = nn.Parameter(torch.ones(dim))
    def forward(self, x):
        rms = x.norm(2, dim=-1, keepdim=True) / (x.shape[-1] ** 0.5)
        return self.weight * x / (rms + 1e-8)

Layer Normalization vs Batch Normalization

Why LLMs switched to RMSNorm: RMSNorm drops the mean-centering step (just divides by RMS). Experiments show this performs as well as full LayerNorm while being ~15% faster. LLaMA, Mistral, Falcon, and most 2023–2025 LLMs use RMSNorm.

Dropout variants and modern practices

Modern LLM pretraining practice: Most modern LLM pretraining runs (LLaMA 3, Mistral, Qwen 2/3-family models) use minimal dropout (p=0.0 or 0.1) in attention, relying instead on weight decay (AdamW), gradient clipping, and careful data curation and deduplication for regularization. Large, deduplicated training datasets make heavy dropout less necessary during pretraining.

Dropout in the LoRA / PEFT era

Dropout's role shifted rather than disappeared. While full pretraining runs of recent LLMs mostly minimize dropout, it came back through the side door with parameter-efficient fine-tuning. LoRA (Hu et al., 2021) and its successors expose a lora_dropout hyperparameter — typically 0.05–0.1 — applied to the low-rank adapter's input before the down-projection. Because a LoRA adapter has orders of magnitude fewer trainable parameters than the base model it's attached to (often under 1% of it), it is far more prone to overfitting a small fine-tuning set than the base model ever was, and dropout is one of the few regularizers cheap enough to apply at that scale without meaningfully slowing training.

QLoRA and dropout: In QLoRA (Dettmers et al., 2023) specifically, dropout on the adapter is one of the few practical levers left to fight overfitting: the base model is frozen and quantized, so you cannot rely on tuning weight decay across the full network the way you would for a from-scratch training run — the adapter's own regularization has to carry more of the load.

Weight initialization strategies

Proper weight initialization is critical — too small causes vanishing gradients, too large causes exploding gradients. The right strategy depends on the activation function:

Default for transformers: Most Transformer implementations (GPT, BERT, LLaMA) use small normal N(0, 0.02) for linear weights and zero for biases. Some use scaled initialization: divide by √(2 × n_layers) for residual branch projections to prevent residual stream explosion at initialization.

Practice questions

1. What is the difference between dropout during training and dropout during inference? (Answer: Training: randomly set p fraction of neurons to 0 each forward pass (inverted dropout: scale remaining neurons by 1/(1-p) during training). Each mini-batch uses a different random sparse subnetwork. Inference: disable dropout entirely — all neurons active. No random masking. If using inverted dropout (PyTorch default), weights are already properly scaled so no additional scaling is needed at inference. Critical: always call model.eval() before evaluation to disable dropout.)
2. What is MC Dropout (Monte Carlo Dropout) and what problem does it solve? (Answer: MC Dropout (Gal & Ghahramani 2016): keep dropout active during inference, run N forward passes (e.g., 100), compute mean and variance of predictions. Mean ≈ the model's best estimate. Variance ≈ model uncertainty. Provides Bayesian approximation to deep learning inference — uncertainty quantification without expensive Bayesian training. Use case: medical diagnosis where knowing 'the model is uncertain here' is as important as the prediction itself. Works with any dropout-trained model.)
3. Spatial dropout in CNNs drops entire feature maps rather than individual neurons. Why? (Answer: In convolutional layers, adjacent neurons (pixels in a feature map) are spatially correlated — dropping one neuron while its neighbors are active provides minimal regularization. Spatial dropout drops entire feature maps (channels) — forcing the network to learn that each feature map is replaceable, promoting robustness. Particularly effective in image segmentation models (U-Net uses spatial dropout) where spatial correlation is very high.)
4. What is the relationship between dropout and ensemble learning? (Answer: Training with dropout trains an exponential ensemble of 2^n different subnetworks (n = number of droppable neurons). Each forward pass uses a different random architecture. Inference without dropout is approximate averaging over this ensemble (weight sharing across all subnetworks). This is why dropout generalizes well: it is equivalent to training many different models simultaneously with shared weights — the ensemble effect reduces variance without the cost of training separate models.)
5. Why does a LoRA adapter typically need a higher relative dropout rate than the base model it fine-tunes ever used? (Answer: A LoRA adapter has a tiny fraction of the base model's trainable parameters (often <1%) but is usually trained on a comparatively small, narrow dataset. Its effective capacity-to-data ratio is much higher than the base model's was during pretraining, so it overfits the fine-tuning set much faster. lora_dropout (commonly 0.05–0.1) is one of the cheapest, most direct ways to fight this, since the base model stays frozen and can't absorb more regularization through weight decay.)