Principal Component Analysis (PCA)
Principal Component Analysis (PCA) is an unsupervised dimensionality reduction technique that transforms features into a new orthogonal coordinate system — the principal components — ordered by the amount of variance they explain. The first principal component (PC1) captures the direction of maximum variance; PC2 captures maximum remaining variance perpendicular to PC1, and so on. PCA reduces dimensions while retaining the most information, removes correlated features, and speeds up downstream ML. PCA is one of the highest-weightage topics in GATE DS&AI — typically 3–4 marks, often as numerical questions.
Compressing high-dimensional data while preserving as much variance as possible.
Category: Machine Learning
Real-life analogy: The shadow on the wall
Imagine a 3D object with a torch shining on it from different angles. Some angles cast a very informative shadow (showing the object's shape well). Other angles cast a small, uninformative shadow. PCA finds the angle that casts the most informative shadow — the direction that captures maximum variation in the data. The shadow is the low-dimensional projection.
PCA algorithm — step by step
- Standardise: Center and scale features to zero mean and unit variance (StandardScaler). Critical — PCA is scale-sensitive.
- Compute covariance matrix: Σ = (1/n) XᵀX (X already mean-centerd). Σ is (p×p) symmetric positive semi-definite.
- Eigendecomposition: Decompose Σ = VΛVᵀ where columns of V are eigenvectors (principal components) and Λ is diagonal with eigenvalues λ₁ ≥ λ₂ ≥ ... ≥ λₚ.
- Select top-K eigenvectors: Choose K eigenvectors with largest eigenvalues — they explain the most variance.
- Project: Transform data onto the K principal components: Z = XV_K (where V_K is the p×K matrix of top-K eigenvectors).
\text{Explained variance ratio}_k = \frac{\lambda_k}{\sum_{j=1}^p \lambda_j} \quad \text{Cumulative variance} = \frac{\sum_{k=1}^K \lambda_k}{\sum_{j=1}^p \lambda_j}
import numpy as np
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler
from sklearn.datasets import load_iris
iris = load_iris()
X = iris.data # 150×4 (sepal length, sepal width, petal length, petal width)
# ── PCA from scratch ──
# Step 1: Standardise
X_scaled = (X - X.mean(axis=0)) / X.std(axis=0)
# Step 2: Covariance matrix
Sigma = np.cov(X_scaled, rowvar=False) # (4×4)
print("Covariance matrix shape:", Sigma.shape)
# Step 3: Eigendecomposition
eigenvalues, eigenvectors = np.linalg.eigh(Sigma)
# Sort descending (eigh returns ascending)
idx = np.argsort(eigenvalues)[::-1]
eigenvalues = eigenvalues[idx]
eigenvectors = eigenvectors[:, idx]
print("Eigenvalues:", eigenvalues.round(3))
print("Explained variance ratio:", (eigenvalues/eigenvalues.sum()).round(3))
print("Cumulative explained variance:", np.cumsum(eigenvalues/eigenvalues.sum()).round(3))
# Step 4: Project onto top-2 PCs
V2 = eigenvectors[:, :2] # (4×2)
Z = X_scaled @ V2 # (150×2)
print(f"Original shape: {X.shape} → Reduced: {Z.shape}")
# ── sklearn PCA ──
pca = PCA(n_components=2)
scaler = StandardScaler()
X_pca = pca.fit_transform(scaler.fit_transform(X))
print(f"sklearn explained variance: {pca.explained_variance_ratio_.round(3)}")
# PC1 explains ~73%, PC2 explains ~23% → 2 PCs capture ~96% of variance!Scree plot and choosing K
A scree plot plots eigenvalues (or explained variance ratios) in decreasing order. Look for the 'elbow' — where variance drops sharply and then levels off. Components before the elbow are meaningful; those after are noise. The 95% cumulative variance rule is more objective: choose K such that the first K components explain ≥ 95% of total variance.
PCA is not feature selection: PCA creates NEW features (linear combinations of original features) — it does not select original features. Principal components are directions in feature space, not individual original features. If interpretability matters, use feature selection methods (Lasso, mutual information) instead. PCA components have no direct physical meaning.
| Method for choosing K | Rule | Trade-off |
|---|---|---|
| Cumulative variance | Smallest K explaining ≥ 90–95% of variance | Objective and most common; the threshold itself is arbitrary |
| Scree elbow | Cut where the eigenvalue curve flattens | Intuitive; the elbow is often ambiguous in real data |
| Kaiser criterion | Keep components with eigenvalue > 1 (on standardized data) | Fast rule of thumb; known to over-retain components |
| Downstream CV | Treat K as a hyperparameter; tune on validation score | Best when PCA feeds a model — optimizes what you actually care about |
PCA vs SVD, and why sklearn never builds the covariance matrix
Textbooks teach PCA as eigendecomposition of the covariance matrix C = XᵀX/(n−1). Real implementations do something else: they run SVD directly on the centered data matrix, X = UΣVᵀ. The two are mathematically equivalent — V's columns are exactly the principal components, and eigenvalues relate to singular values by λᵢ = σᵢ²/(n−1) — but SVD is numerically far safer, because forming XᵀX squares the condition number and destroys precision on ill-conditioned data. It is also cheaper when p ≫ n: a 100-sample, 20,000-feature dataset has a 20,000×20,000 covariance matrix (3.2 GB) but SVD never materializes it.
| Variant | What it does | Use when |
|---|---|---|
PCA(svd_solver="full") | Exact SVD of the centered matrix | Small-to-medium data; you want exactness |
PCA(svd_solver="randomized") | Randomized SVD approximating the top K components | Large data where K ≪ p — sklearn's auto choice |
IncrementalPCA | Processes data in mini-batches | Data does not fit in memory |
TruncatedSVD | Skips centering — works on sparse matrices | TF-IDF / sparse text data, where centering would densify and explode memory (this is LSA) |
KernelPCA | PCA in an implicit high-dimensional feature space | Structure is non-linear (e.g., concentric circles) |
The mistakes that quietly ruin PCA results: (1) Fitting on the full dataset before the train/test split — the test set then influences the components, leaking information and inflating your score. Always fit on train, transform the test set. (2) Forgetting to standardize — PCA maximizes variance, so an unscaled feature in rupees will dominate one in years, and PC1 becomes "whichever column had the biggest units." (3) Assuming high variance means high usefulness — PCA is unsupervised and cannot see labels, so it will happily discard a low-variance direction that perfectly separates your classes; if you want a supervised projection, use LDA. (4) Running PCA on one-hot encoded categoricals — variance is not meaningful for binary indicator columns; consider MCA or leave them out.
Practice questions (GATE-style)
- A dataset has 10 features. Eigenvalues of the covariance matrix are [5, 3, 2, 1, 0.5, 0.3, 0.1, 0.1, 0.05, 0.05]. How many PCs capture ≥ 90% variance? (Answer: Total = 12.1. Cumulative: 5(41%), 8(66%), 10(82%), 11(91%). Three PCs capture 83%, four capture 91% → need 4 PCs for ≥90%.)
- Why must data be standardized before PCA? (Answer: PCA maximizes variance. Features with large measurement scales (e.g., salary in thousands vs. age in years) dominate the covariance matrix. Standardising ensures all features contribute equally.)
- Principal components are always: (Answer: Orthogonal to each other (perpendicular in feature space) — they are uncorrelated by construction. This is the key property that makes PCA useful for removing correlation.)
- If all features are already uncorrelated, what does PCA produce? (Answer: Principal components aligned with the original feature axes — PCA finds nothing new. The eigenvectors are the standard basis vectors. PCA is most useful when features are correlated.)
- PCA reduces a 1000-feature dataset to 10 PCs. A new test point has 1000 features. How do you project it? (Answer: Subtract the training mean, divide by training std (same scaler), then multiply by the 1000×10 matrix of top-10 eigenvectors. Never refit the scaler or PCA on test data.)
- Why does scikit-learn compute PCA via SVD of the data matrix instead of eigendecomposition of the covariance matrix? (Answer: Numerical stability and cost. Forming XᵀX squares the condition number, amplifying floating-point error on ill-conditioned data. SVD of the centered X yields the same components (V columns) with λᵢ = σᵢ²/(n−1), without ever building the p×p covariance matrix — decisive when p >> n, e.g. 20,000 features would need a 3.2GB covariance matrix.)
- PCA discards the component that best separates your two classes. How is that possible? (Answer: PCA is unsupervised — it maximizes variance and never sees the labels. A direction can carry little total variance yet be perfectly discriminative, so PCA ranks it last and drops it. Use LDA instead when the goal is class separation; LDA maximizes between-class over within-class scatter using the labels.)
- Why use TruncatedSVD instead of PCA on a TF-IDF matrix? (Answer: PCA centers the data by subtracting the mean, which turns a sparse matrix dense — a 100k×50k TF-IDF matrix would explode in memory. TruncatedSVD skips centering and operates directly on the sparse matrix. Applied to TF-IDF this is exactly Latent Semantic Analysis (LSA).)
PCA is used extensively in LLM research: visualizing high-dimensional token embeddings in 2D, understanding what directions in embedding space correspond to semantic properties (e.g., the sentiment direction, the formality direction), and compressing model activations for efficient inference.