Real-life analogy: The doctor's diagnosis

A doctor examines blood pressure, cholesterol, and age to decide if a patient has heart disease (yes/no). Logistic regression does exactly this: it combines multiple factors with learned weights, passes the result through a sigmoid function to get a probability (e.g., 0.82 = 82% chance of heart disease), and then classifies above 0.5 as positive. The doctor's threshold (50%) can be adjusted — if the disease is dangerous, you might use 0.3 to catch more cases.

The sigmoid function and decision boundary

P(y=1 \mid x) = \sigma(z) = \frac{1}{1 + e^{-z}} \quad \text{where} \quad z = \beta_0 + \beta_1 x_1 + \cdots + \beta_p x_p = \boldsymbol{\beta}^T \mathbf{x}

The log-odds (logit) interpretation: log(P/(1−P)) = βᵀx. Each unit increase in feature xⱼ multiplies the odds by e^βⱼ. If β₁ = 0.5, then every unit of x₁ increases the odds of the positive class by e^0.5 ≈ 1.65× — a 65% increase in odds.

import numpy as np
from sklearn.linear_model import LogisticRegression
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# ── From scratch (binary logistic regression) ──
def sigmoid(z): return 1 / (1 + np.exp(-z))

def logistic_gradient_descent(X, y, lr=0.01, epochs=1000):
    n, p = X.shape
    beta = np.zeros(p + 1)
    X_aug = np.c_[np.ones(n), X]   # Add bias column
    for _ in range(epochs):
        z     = X_aug @ beta
        y_hat = sigmoid(z)
        grad  = X_aug.T @ (y_hat - y) / n
        beta -= lr * grad           # Gradient descent step
    return beta

# Generate binary classification data
X, y = make_classification(n_samples=500, n_features=4,
                            random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42)

# From scratch
beta = logistic_gradient_descent(X_train, y_train, lr=0.1, epochs=1000)
X_test_aug = np.c_[np.ones(len(X_test)), X_test]
probs       = sigmoid(X_test_aug @ beta)
preds       = (probs >= 0.5).astype(int)
print(f"Scratch accuracy: {accuracy_score(y_test, preds):.3f}")

# sklearn
clf = LogisticRegression(max_iter=1000)
clf.fit(X_train, y_train)
print(f"sklearn accuracy: {accuracy_score(y_test, clf.predict(X_test)):.3f}")
print(classification_report(y_test, clf.predict(X_test)))

Loss function: binary cross-entropy (log loss)

\mathcal{L} = -\frac{1}{n}\sum_{i=1}^{n} \left[ y_i \log(\hat{p}_i) + (1-y_i) \log(1-\hat{p}_i) \right]

Why not use MSE for logistic regression?: MSE applied to sigmoid outputs creates a non-convex loss landscape with many local minima — gradient descent may not converge. Cross-entropy with sigmoid creates a convex loss function — gradient descent always finds the global minimum. This is why cross-entropy is the standard loss for classification.

Multiclass logistic regression (Softmax)

P(y=k \mid \mathbf{x}) = \frac{e^{\mathbf{w}_k^T \mathbf{x}}}{\sum_{j=1}^K e^{\mathbf{w}_j^T \mathbf{x}}}

The interpretability payoff: odds ratios: This is why logistic regression survives in medicine, credit scoring, and any regulated setting. A coefficient β is a log-odds change, so e^β is an odds ratio: β = 0.7 → e^0.7 ≈ 2.0 means each one-unit increase in that feature doubles the odds of the positive outcome, holding others fixed. No other classifier hands you a defensible sentence like "each additional year of age raises the odds of readmission by 3%" — which is exactly what an auditor or a clinician asks for. Two caveats: odds ratios are not risk ratios (they exaggerate when the base rate is high), and coefficients are only comparable across features if the features are standardized.

Regularization and the separability blow-up

Unregularized logistic regression has a fatal edge case: if the classes are perfectly linearly separable, the likelihood keeps increasing as the weights grow without bound. The optimizer pushes coefficients toward ±∞, chasing predicted probabilities of exactly 0 and 1, and never converges — sklearn reports this as a ConvergenceWarning. This is not a bug; it is the maximum-likelihood estimate genuinely not existing. Regularization fixes it by penalizing large weights, which is why scikit-learn applies L2 by default (penalty='l2', C=1.0) — a detail that surprises people who expect a "plain" model.

C is inverse strength — the most misread hyperparameter: In scikit-learn the knob is C, and it is the inverse of regularization strength: small C = strong regularization (simpler model, more bias), large C = weak regularization (closer to unregularized, more variance). People routinely tune it backwards. Also: regularization penalizes weight magnitude, so features must be standardized first — otherwise a feature measured in rupees is penalized differently from the same feature measured in lakhs, and the penalty silently encodes your unit choices.

Practice questions (GATE-style)

1. What does the sigmoid output of 0.72 mean in logistic regression? (Answer: The model predicts a 72% probability of the input belonging to class 1. With threshold 0.5, it classifies as class 1.)
2. Why is logistic regression called "regression" when it does classification? (Answer: It models the log-odds as a linear regression: log(P/(1-P)) = βᵀx. The "regression" refers to modeling the log-odds, not the binary class label directly.)
3. A logistic regression model has coefficient β₁ = 1.2 for feature "hours studied". What is the odds ratio? (Answer: e^1.2 ≈ 3.32. Each additional hour of study multiplies the odds of passing by 3.32×.)
4. For a 3-class problem, logistic regression uses: (Answer: Softmax (multinomial logistic regression) with 3 weight vectors — one per class. Output is a probability vector summing to 1.)
5. What is the gradient of cross-entropy loss with respect to weights? (Answer: ∇_β L = (1/n) Xᵀ(ŷ − y) — identical in form to linear regression gradient but with ŷ = sigmoid(Xβ) instead of Xβ.)
6. Your logistic regression throws a ConvergenceWarning and coefficients are enormous. The classes turn out to be perfectly separable. Why, and what is the fix? (Answer: With perfect separation the likelihood increases without bound as weights grow — the model chases probabilities of exactly 0 and 1, so the maximum-likelihood estimate does not exist and the optimizer never converges. Fix: add regularization (L2 is sklearn's default; lower C to strengthen it), which bounds the weights and guarantees a unique solution.)
7. In scikit-learn, does C=100 mean more or less regularization than C=0.01? (Answer: Less. C is the INVERSE of regularization strength: C=100 is weak regularization (high variance, closer to unregularized MLE); C=0.01 is strong regularization (high bias, simpler model). This is the reverse of alpha in Ridge/Lasso, which is a common source of tuning errors.)
8. Why must features be standardized before applying L1 or L2 regularization? (Answer: The penalty is on weight magnitude, and a feature's weight scales inversely with the feature's units. A variable in metres gets a coefficient 1000× larger than the same variable in kilometres, so it absorbs 10^6× more L2 penalty — meaning the regularizer would penalize features based on arbitrary unit choices rather than actual importance. Standardizing puts all features on a comparable scale so the penalty is meaningful.)