Influential Instances
A training instance is influential when deleting it from the training data considerably changes the model's learned parameters or predictions. Identifying influential instances lets us debug models and better explain their behavior — both globally and for individual predictions.
Two questions influential instances can answer
- Which training instances were most influential on the model parameters or predictions overall?
- Which training instances were most influential on a particular prediction?
There are two major families of techniques for finding influential instances: Deletion Diagnostics and Influence Functions. Neither family is model-agnostic — each has specific requirements. They are covered here because they are example-based explanations.
Deletion Diagnostics
The deletion approach asks: what happens when we remove one training instance, refit the model, and compare the result to the full-data fit? Two widely used metrics capture this idea.
DFBETA
DFBETA measures the effect of deleting a single instance on the model's parameters. It is useful for models with explicit weight vectors, such as logistic regression and neural networks.
- Remove one training instance.
- Refit the model and record the new weight vector.
- Compare the new weights to those from the full dataset.
A large DFBETA value signals that the instance has a disproportionate effect on the learned weights.
Cook's Distance
Cook's Distance measures the effect of deleting a single instance on the model's predictions. It was originally developed for linear regression and includes a squared term and the mean squared error, but the core idea can be adapted to other model types by removing those specific terms.
- Remove one training instance.
- Refit the model and record the new predictions.
- Compare the new predictions to those from the full dataset.
A high Cook's Distance flags an instance whose removal shifts predictions substantially across the dataset.
Remove Training Instances and Watch the Prediction
Each dot is a training instance. Click to remove it. Influential points shift the test prediction; others have almost no effect.
Test prediction P(7)
no change
Hover over a point to inspect it. Click to remove it from training.
Influence Functions
Introduced in 2017, influence functions offer an alternative to deletion diagnostics that avoids the cost of retraining. Instead of removing a training instance, an influence function approximates how much the model parameters would change if you upweighted the loss of that instance by an infinitesimally small step ε.
This approximation uses the first derivative of the loss with respect to the model parameters, so it requires models whose loss is differentiable — such as logistic regression, SVMs, and neural networks. Models that do not expose a differentiable loss (for example, decision trees) are not compatible.
Limitations of influence functions
Influence functions scale better than exact deletion, but they rely on assumptions about the loss surface (specifically, that it is locally well-approximated by a quadratic). Deep neural networks can violate these assumptions, especially near sharp minima. For high-stakes findings, validate the approximation with targeted retraining.
Influential instance methods are valuable for debugging mislabeled data, tracing surprising predictions, and finding training examples that dominate a model's behavior. The choice between deletion diagnostics and influence functions depends on model type, dataset size, and whether an exact or approximate answer is needed.
How can an ordinary-looking training case be influential?