Saliency Maps

A classifier calls an image “house.” A saliency map asks which pixels would change that class score most if they moved slightly.

ℹ

Input gradient

For class score $S_c(x)$ , vanilla saliency uses:

$M_i=\left|\frac{\partial S_c}{\partial x_i}\right|$

A large magnitude means the score is locally sensitive to pixel or feature $i$ . It does not mean that feature is sufficient or semantically meaningful.

Mask Pixels, Watch the Score

Paint over pixels to gray them out. The model re-scores without that information. Then toggle the saliency overlay to see whether the bright regions predicted by gradients actually drive the decision.

House (16×12 px)

Click or drag to mask · drag again to restore · toggle overlay to compare

P(house)

0.910

Score drop

—

Pixels masked

—

Paint over pixels, then toggle the saliency overlay to compare what the gradient predicted.

When high-saliency regions drop the score and low-saliency regions don't, gradients were trustworthy here. When the pattern breaks, the map is pointing at sensitivity, not the cause.

Use saliency maps to inspect a model during development, then pair them with perturbation tests. If removing the brightest pixels does not affect the score, the map is not doing the explanatory work you hoped.

Checkpoint

What does a bright saliency pixel establish?

←PreviousInfluential InstancesExample-Based Explanations Next→Grad-CAM and VariantsAttribution-Based Explanations