Sparse Models
A linear model with five features is easy to read. A linear model with five thousand features is technically transparent but practically its not feasible to inspect every feature. Real data is messy, so we often end up with hundreds of measurements, one-hot encoded categories, and engineered interactions.
Sparsity asks the question: can we make a similar prediction with fewer active features? If the answer is yes, the smaller model is easier to inspect, explain, test, and maintain.
Lasso
The least absolute shrinkage and selection operator adds an L1 penalty to the regression objective:
The first term rewards fit. The second "charges" for coefficient magnitude. As grows, coefficients shrink and many become exactly zero.
Lasso Coefficient Path
Increase the regularization penalty. Weak coefficients reach zero first, so the model becomes easier to inspect while fit gradually worsens.
Active features
5 / 5
Validation RMSE
$3.8k
minimize SSE + λ Σ|βj|
Absolute values create sharp corners in the objective. Those corners make exact zeros common.
The tuning parameter controls the bargain. At zero, you recover ordinary least squares. A moderate value can remove weak features such as an arbitrary neighborhood code or the number of cats nearby. A large value can erase useful signal.
Choose with your validation data or via cross-validation. The goal is to find a balance between model complexity (we want this to be lower) and fit (we want this to be higher).
Zero coefficients are not evidence of irrelevance!
When two predictors are strongly correlated, Lasso may keep one and discard the other. A different sample can reverse that choice. A zero coefficient means the feature did not survive this penalized prediction problem; it does not prove the feature is irrelevant in the world.
A hospital's sparse model keeps one of three correlated lab measurements and drops the other two. What would you communicate before calling the retained lab 'the important one'?