Logistic Regression

Linear regression can predict 1.3 or -0.2. That is fine for house price and strange for the probability that a student passes a course. Probabilities have to stay between zero and one.

Logistic regression keeps the weighted sum, then wraps it in the sigmoid function:

$p(y=1\mid x)=\frac{1}{1+e^{-(\beta_0+\beta^Tx)}}$

The weighted sum can range from negative infinity to positive infinity. The sigmoid maps that whole line into a valid probability.

From Score to Decision

A tutoring program predicts whether a student will pass. Move study hours to follow one value through the linear score, odds, probability, and final threshold.

Study hours per week4.0 hoursDecision threshold0.50

Program decision

needs support

0p = 0.4011

threshold at 0.50

1. Linear score z = β₀ + β₁x

-4 + 0.9(4.0) = -4 + 3.60 = -0.400

2. Odds = e^z

e^(-0.400) = 0.670 → 0.670 : 1 in favor

3. Probability p = 1 / (1 + e^−z)

1 / (1 + e^−(-0.400)) = 1 / (1 + 1.492) = 0.401

4. Threshold comparison

0.401 < 0.50 → needs support

The sigmoid changes the scale, not the underlying geometry. Logistic regression still creates a linear decision boundary in feature space.

The linear equation models log-odds:

$\log\left(\frac{p}{1-p}\right)=\beta_0+\beta^Tx$

If $p=0.8$ , the odds are $0.8/0.2=4$ : four to one in favor of passing. A one-unit increase in feature $x_j$ multiplies the odds by $e^{\beta_j}$ . If a study-hours coefficient is 0.9, one additional hour multiplies the estimated odds by $e^{0.9}\approx2.46$ , holding the other features constant.

How e^β (odds ratio) changes with feature type

When X is numerical, a one-unit increase in X multiplies the estimated odds by e^β, the odds ratio. All other features are held constant.

log-odds = -2 + 0.9 × study_hours

β (study hours coefficient)0.90Current study hours (X)3 hrs

Odds of passing: 3 hrs → 4 hrs

P(pass | 3 hrs)67%

P(pass | 4 hrs)83%

β interpretation

Each additional study hour multiplies estimated odds by e^0.90 ≈ 2.46. Going from 3 → 4 hrs changes estimated probability from 66.8% to 83.2%.

⚠

The threshold is a policy choice

The model returns a probability. A school, lender, or hospital chooses what to do with it. Moving the threshold changes false positives and false negatives without retraining the regression. That choice should reflect the cost of each error and who bears it.

ℹ

Assumptions of logistic regression

Binary outcome. The response is a category with two levels. Extensions (multinomial, ordinal logistic) handle more, but the standard model assumes exactly two.
Independence of observations. Each data point is drawn independently. Clustered or repeated-measures data violates this and requires mixed or GEE models.
No perfect separation. If one feature (or combination of features) perfectly predicts the outcome, the algorithm cannot converge and coefficients inflate without bound.
Linear log-odds. The model assumes each feature has a straight-line relationship with the log-odds of the outcome. Non-linear effects need splines, interactions, or transformations.
Limited multicollinearity. Highly correlated predictors inflate standard errors and make individual coefficients unstable, just as in linear regression.
Sufficient sample size. A common rule of thumb is at least 10–20 events per predictor. Sparse data leads to unreliable coefficients and over-fit decision boundaries.

Logistic regression is interpretable when the feature set is manageable and the log-odds relationship is credible. It still needs enough data, sensible feature coding, limited multicollinearity, and checks for influential observations.

Checkpoint

A hospital lowers an escalation threshold from 0.7 to 0.4 while keeping the logistic model fixed. What changes?

Checkpoint

For a disease-screening model, which error would push you toward a lower threshold? What operational cost might that create?

←PreviousSparse ModelsRegression Next→Generalized Models: GLMs and GAMsRegression