Linear Regression

Whether you are searching for a home or just window shopping, apps that predict house prices are pretty neat. Let's say we want to build one. We have data on houses: bedrooms, location, square footage, whether there is a pool. We want a model that takes those features and returns a predicted price.

Linear regression is one of the oldest and most useful tools for this job. I also think it is the basis of a remarkable amount of machine learning. Neural networks are full of weighted sums. Linear regression lets us slow that idea down enough to see every part.

The Four OLS Assumptions

Linearity

Relationship is a straight line

The relationship between each predictor and the outcome is approximately linear. A straight line through the data should capture the trend without any systematic curve.

Independence

Observations don't influence each other

Each observation is independent of every other. Knowing the value of one residual tells you nothing about another. This is violated by clustered sampling, repeated measures, and time series data.

Homoscedasticity

Residuals have constant variance

The spread of residuals is roughly the same across all fitted values. A model that's equally uncertain whether predicting a $200k or a $900k house satisfies this assumption.

Normality of Residuals

Errors follow a bell curve

The residuals are approximately normally distributed around zero. This matters most for hypothesis tests on coefficients and for confidence intervals — not for the outcome variable itself.

These assumptions are checked after fitting a model, using residual plots. Violations bias coefficient estimates, corrupt standard errors, and make hypothesis tests unreliable.

Checkpoint

A real estate analyst fits a linear regression model predicting house price from square footage. She notices that the residuals for small houses are all clustered near zero, but for large houses the residuals vary widely — some $200k too high, some $150k too low. Which assumption is most likely violated?

Start with a deliberately silly predictor: the number of cats in the neighborhood. Plot cats against sale price and the dots should look like a cloud. Cats carry little useful information about the price.

Draw a line through that cloud. For each house, the residual is the vertical distance between the actual price and the line's prediction:

$e_i = y_i - \hat{y}_i$

Points above the line have positive residuals because the model underpredicted. Points below the line have negative residuals because the model overpredicted. The line is the model; the residuals are everywhere it is wrong.

ℹ

Why square the residuals?

Raw residuals cancel. A line can have large positive and negative errors that sum to zero. Squaring makes every contribution positive and makes a large miss cost more than several small misses.

$\text{SSE} = \sum_{i=1}^{n}(y_i - \hat{y}_i)^2$

Ordinary least squares (OLS) chooses the coefficients that minimize this sum of squared errors.

Residuals Explorer

Predictor

No real relationship. The OLS line is nearly flat — cats don't predict prices.

Drag anywhere to rotate and shift the line. Amber bars = residuals. Dashed gray = OLS solution.

Sum of Squared Errors (SSE)

Your line102.0B

OLS minimum102.0B

You're at (or very near) the OLS minimum — no line can do better.

Current line equation

price = 3.3 × cats + 260.8k

OLS: price = 3.3 × cats + 260.8k

Houses Your line OLS line Residuals

Checkpoint

Line A has raw residuals that sum to zero. Line B has squared residuals that sum to zero. Which line fits the data better?

Now switch from cats to bedrooms. Suppose OLS returns:

$\hat{\text{price}} = 65{,}000(\text{bedrooms}) + 100{,}000$

This is the same $y = mx + b$ you learned in algebra. The slope, $65{,}000$ , says each additional bedroom is associated with a $65{,}000$ increase in predicted price. The intercept, $100{,}000$ , is the prediction when bedrooms equals zero.

That last sentence is mathematically precise and may be practically strange. If the training data only contains houses with two to six bedrooms, zero bedrooms is outside the evidence. The intercept still anchors the line, but it does not automatically describe a meaningful house.

Regression Interpreter

Predictor

Orange dot = probe point. Dashed lines show the predicted value.

Probe

Probe (bd)3.5 bd

Interpretation

Slope — β₁ = +65.0 $k/bd

For each additional 1 bd of bedrooms, predicted house price increases by 65.0 $k.

Intercept — β₀ = 100 $k

When bedrooms = 0, predicted house price = 100 $k. Predicted price at 0 bedrooms (a studio or anchor point — not always interpretable).

Prediction at Bedrooms = 3.5 bd

ŷ = 65.0 × 3.5 + 100 = 328 $k

Equation

House price = 65.0 × Bedrooms + 100

Houses Regression line Probe point

⚠

Association is not causation

The bedroom coefficient describes an association in the data. It does not say that adding a bedroom causes a house to gain exactly $65,000 in value. Bedrooms travel with square footage, neighborhood, renovation quality, and plenty of other variables. Causal claims need a defensible design, not a coefficient alone.

Bedrooms alone is a limited model. Multiple linear regression adds more predictors:

$\hat{y} = \beta_0 + \beta_1x_1 + \beta_2x_2 + \cdots + \beta_px_p$

For a house-price model, one fitted equation might be:

$\hat{\text{price}} = 100{,}000 + 60{,}000(\text{bedrooms}) - 8{,}000(\text{miles}) + 45{,}000(\text{pool})$

How β₁ changes with feature type

When X is numerical, β₁ is the change in predicted outcome for each one-unit increase in X, holding everything else constant.

price = 65 × bedrooms + 100

β₁ (slope — $k per bedroom)+65kCurrent bedrooms (X)3 bedrooms

β₁ interpretation

Adding 1 bedroom (from 3 → 4) changes predicted price by +65k. That shift is always β₁ = 65k, regardless of which bedroom you add.

⚠

Coefficients depend on the other features

A bedroom coefficient can be positive in a simple model and negative after controlling for square footage. Holding total space constant, another bedroom means smaller rooms. Before interpreting a coefficient, inspect what else the model holds constant.

Checkpoint

A house-price model includes bedrooms, square footage, and distance from downtown. The bedroom coefficient is negative. What is the strongest interpretation?

Checkpoint

You fit a highly accurate house-price model, but the residual plot fans out sharply for expensive houses. Would you deploy it to set listing prices? What would you investigate first?

Next→Sparse ModelsRegression