The Attention-as-Explanation Debate
One 2019 paper argued that attention is not an explanation. A rebuttal published that year argued that attention can still provide explanatory evidence. And the debate continues to this day!
The strongest criticism is that very different attention patterns can sometimes produce the same output.
Routing is not the final reason
Attention weights expose one internal operation. They can be weakly correlated with gradient attribution, may change without changing the prediction, and cannot always be intervened on while every other variable stays fixed.
Same Prediction, Different Attention
Switch between two attention distributions for the pronoun “she.” Both produce the same toy coreference prediction even though the internal routing looks different.
Coreference prediction
nurse
Prediction probability
0.82
Attention changed
baseline
Attention can still support an explanation when the claim is narrow: “this head routes information from the pronoun to the noun.” It is weaker evidence for “the model predicted this because of that noun.”
Match the visualization to the claim. Architecture inspection and output attribution are different jobs.
When is attention evidence most defensible?