Permutation Feature Importance
The importance of a feature can be measured by calculating how much the model's prediction error increases after permuting the feature. If shuffling a feature's values increases the model error, the feature is important. If the model error doesn't change after shuffling, the feature is considered unimportant.
The technique was first introduced for random forests in 2001, then generalized to be model-agnostic and renamed model reliance in 2018.
Shark Attack GPT predicts daily shark attacks from temperature, precipitation, and ice cream sales. To test whether it relies on temperature, shuffle the temperature column and leave everything else alone.
If error rises sharply, the model depended on the association you broke.
Model reliance
For feature , permutation importance can be measured as:
The first term is error after shuffling feature . The second is original error. A larger increase means greater reliance.
Permutation Feature Importance — Step by Step
Walk through the four stages of measuring how much the Shark Attack GPT model relies on each feature.
Before touching any feature, record the model's prediction error on the held-out dataset. This is the reference point — every importance score is measured relative to it.
Baseline error
0.180
Progress
Run several shuffles because the measurement contains randomness. Use held-out data so importance describes generalization rather than training fit.
Correlated features create a second problem. If temperature and ice cream sales carry similar information, the model may keep performing after either one is shuffled. Both can look unimportant even when the pair matters.
Why can permutation importance underestimate two strongly correlated features?