Feature Interaction

An electric car may add $20,000. A luxury brand may add $40,000. If an electric luxury car adds $70,000, the extra $10,000 is an interaction.

Interactions matter because a feature's effect can depend on the value of another feature.

Find the Missing Cell

Set three car prices and compare the observed electric-luxury price with the price an additive model would predict.

Gasoline
Electric
Standard

Base

$30k

Luxury

Additive expectation

$90k

Observed joint price

$100k

Interaction

+$10k

The interaction is a difference of differences. It appears only after the four combinations are compared, not by inspecting one expensive car.

We use Friedman's H-statistic (2008) to measure feature interaction. There are two variants: the two-way H-statistic, which measures the interaction between two features jj and kk, and the total H-statistic, which measures the interaction between jj and all other features combined.

ℹ

The math: H-statistic

At each data point, partial dependence is evaluated over all nn data points.

Two-way H-statistic (j vs. k):

Hjk2=∑i=1n[f^jk(xj(i),xk(i))−f^j(xj(i))−f^k(xk(i))]2∑i=1nf^jk2(xj(i),xk(i))H^2_{jk}=\frac{\sum_{i=1}^{n}\left[\hat{f}_{jk}(x_{j}^{(i)},x_{k}^{(i)})-\hat{f}_{j}(x_{j}^{(i)})-\hat{f}_{k}(x_{k}^{(i)})\right]^2}{\sum_{i=1}^{n}\hat{f}^2_{jk}(x_{j}^{(i)},x_{k}^{(i)})}

Total H-statistic (j vs. all):

Hj2=∑i=1n[f^(x(i))−f^j(xj(i))−f^−j(x−j(i))]2∑i=1nf^2(x(i))H^2_{j}=\frac{\sum_{i=1}^{n}\left[\hat{f}(x^{(i)})-\hat{f}_{j}(x_{j}^{(i)})-\hat{f}_{-j}(x_{-j}^{(i)})\right]^2}{\sum_{i=1}^{n}\hat{f}^2(x^{(i)})}

Partial dependence shows the effect of one or more features on the predicted outcome while averaging out all other features — the average effect of a feature on predictions, holding everything else constant.

✦

Sound familiar?

Partial dependence is the average of ICE plots, which you saw in the Local Explanations chapter.

Computational cost is a real concern for feature interaction in practice:

  • Two-way H-statistic (j vs. k) requires up to 2n22n^2 calls to the model's predict function in the worst case.
  • Total H-statistic (j vs. all) requires up to 3n23n^2 calls.
  • To speed things up, we can sample from the nn data points — but this increases the variance of the partial dependence estimates, which can make the H-statistic unstable.
ℹ

Interpreting the H-statistic

Interpreting the H-statistic can be challenging, but the standard interpretation is:

  • H-statistic = 0: No interaction between the features.
  • Total H-statistic = 1: All variance of the function is explained by the sum of the individual partial dependence functions.
  • Two-way H-statistic = 1: Each single PD function is constant — the effect on the prediction comes entirely through the interaction.

Interaction measures can be computationally heavy and unstable. Correlated features also create the same unrealistic combinations that trouble PDPs.

Use a statistic to rank candidate interactions, then inspect the underlying surface and the data density before writing a story about it.

Checkpoint

Why should a high interaction score be followed by a data-density check?

←PreviousFeature DecompositionGlobal ExplanationsNext→Permutation Feature ImportanceGlobal Explanations