Real-life analogy: Shining a flashlight

Imagine two groups of colored balls scattered in 3D space. You want to find the angle to shine a flashlight so the shadows of the two groups on a wall are as separated as possible. LDA finds exactly this optimal projection direction — where the shadow separation between groups is maximum relative to how spread out each group shadow is.

Fisher criterion — what LDA maximizes

J(\mathbf{w}) = \frac{\mathbf{w}^T S_B \mathbf{w}}{\mathbf{w}^T S_W \mathbf{w}}

import numpy as np
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis
from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score

iris = load_iris()
X, y = iris.data, iris.target   # 4 features, 3 classes

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# ── LDA as classifier ──
lda_clf = LinearDiscriminantAnalysis()
lda_clf.fit(X_train, y_train)
print(f"LDA accuracy: {accuracy_score(y_test, lda_clf.predict(X_test)):.3f}")

# ── LDA as dimensionality reduction (K-1 components for K classes) ──
# 3 classes → max 2 LDA components
lda_2d = LinearDiscriminantAnalysis(n_components=2)
X_train_2d = lda_2d.fit_transform(X_train, y_train)
X_test_2d  = lda_2d.transform(X_test)
print(f"Shape: {X_train_2d.shape}")   # (120, 2) — 4D → 2D

# Explained variance ratio
print(f"Explained variance: {lda_2d.explained_variance_ratio_}")

# Compare with PCA (unsupervised)
from sklearn.decomposition import PCA
pca = PCA(n_components=2)
X_train_pca = pca.fit_transform(X_train)
print(f"PCA explained variance: {pca.explained_variance_ratio_}")
# LDA finds better separating directions than PCA for classification tasks

QDA — Quadratic Discriminant Analysis: LDA assumes all classes share the same covariance matrix (Σ). QDA (Quadratic Discriminant Analysis) relaxes this — each class k has its own covariance matrix Σₖ. This creates quadratic decision boundaries (curved). QDA has more parameters and needs more data. LDA is a special case of QDA when all Σₖ are equal.

Why LDA gives at most K−1 components — the rank argument

This is the single most-tested LDA fact, and the reason is pure linear algebra. The between-class scatter matrix is built from the K class-mean deviations:

S_B = \sum_{k=1}^{K} n_k (\boldsymbol{\mu}_k - \boldsymbol{\mu})(\boldsymbol{\mu}_k - \boldsymbol{\mu})^T

Because LDA's discriminant directions are the eigenvectors with non-zero eigenvalues, you can never extract more than K−1 of them: binary classification yields exactly 1 discriminant direction, 3 classes yield at most 2, and the Iris dataset (3 classes) is why every tutorial plots LDA in exactly 2D. If your problem has 2 classes and 500 features, LDA compresses to a single dimension — which is a hard ceiling, not a tuning choice.

When LDA breaks: the small-sample problem: LDA requires inverting S_W. When the number of samples n is less than the number of features p (the "n < p" or small-sample-size regime — common in genomics and text), S_W is singular and S_W⁻¹ does not exist. Standard fixes: (1) shrinkage LDA — blend S_W toward a scaled identity matrix (in scikit-learn: LinearDiscriminantAnalysis(solver="lsqr", shrinkage="auto"), which applies the Ledoit-Wolf estimator); (2) run PCA first to reduce p below n (the classic "PCA + LDA" pipeline from face recognition, a.k.a. Fisherfaces); (3) use the eigen or svd solver, noting that svd avoids forming S_W but cannot do shrinkage.

LDA vs logistic regression — the same boundary, different assumptions

Both produce a linear decision boundary, and students routinely ask which to use. The difference is generative vs discriminative: LDA models the class-conditional distributions P(x|y) as Gaussians and applies Bayes' rule; logistic regression models P(y|x) directly and never assumes anything about how x is distributed.

The practical rule: Use logistic regression as the default for pure classification — it makes fewer assumptions and is what most production tabular pipelines reach for. Use LDA when you also want a supervised projection for visualization or as a preprocessing step, when classes are genuinely near-Gaussian with similar spread, when data is scarce relative to features (with shrinkage), or when classes are perfectly separable and logistic regression blows up. Use QDA when class spreads visibly differ and you have enough data per class to estimate K separate covariance matrices.

Practice questions (GATE-style)

1. LDA for a 5-class problem can produce at most how many discriminant components? (Answer: K−1 = 4 components. The between-class scatter matrix S_B has rank at most K−1.)
2. What assumption does LDA make that QDA does not? (Answer: LDA assumes all classes have the same covariance matrix (Σ). QDA allows each class to have its own covariance matrix Σₖ, leading to quadratic decision boundaries.)
3. LDA maximizes: (Answer: The Fisher criterion J(w) = wᵀS_B w / wᵀS_W w — the ratio of between-class variance to within-class variance in the projected space.)
4. When would you choose PCA over LDA for preprocessing? (Answer: When you have no class labels (unsupervised setting), or when you want to preserve general variance for non-classification tasks like compression or anomaly detection.)
5. LDA assumes Gaussian distributions. What happens when this assumption is violated? (Answer: The linear decision boundary may be suboptimal — non-linear classifiers (RBF SVM, neural networks) might outperform LDA. Kernel LDA can handle non-Gaussian data.)
6. You have 80 samples and 5,000 features (a gene expression dataset), 2 classes. Why does plain LDA fail, and what are two fixes? (Answer: With n=80 < p=5000, the within-class scatter matrix S_W is singular, so S_W⁻¹ does not exist and the Fisher criterion cannot be solved. Fixes: (1) shrinkage LDA — regularize S_W toward a scaled identity, e.g. solver="lsqr", shrinkage="auto" in scikit-learn; (2) apply PCA first to reduce features below n, the classic PCA+LDA pipeline. Note the output is still just K−1 = 1 dimension.)
7. Why does binary LDA produce exactly one discriminant direction regardless of feature count? (Answer: The discriminant directions are eigenvectors of S_W⁻¹S_B with non-zero eigenvalues, and rank(S_B) ≤ K−1 because the K weighted class-mean deviations sum to zero. For K=2, rank(S_B) ≤ 1, so exactly one non-zero eigenvalue exists. LDA projects onto a single line no matter whether you have 5 features or 5,000.)
8. When would LDA beat logistic regression despite both producing linear boundaries? (Answer: When LDA's generative assumptions actually hold — approximately Gaussian classes with similar covariance — LDA is more statistically efficient and reaches a given accuracy with less data. LDA is also stable when classes are perfectly separable, where unregularized logistic regression's coefficients diverge to infinity. Conversely, when the Gaussian assumption is violated or outliers are present, logistic regression is the more robust choice.)