Overview

A trained neural network is an artifact with no blueprint. Nobody wrote the "detect curve," "copy this name," or "answer with France" procedure by hand. Training searched through millions or billions of numbers until those procedures appeared.

Mechanistic interpretability starts from a stubborn question: can we read those learned procedures back out of the model? Not just where the model looked. Not just which input features mattered. What algorithm did the network actually build?

This matters most when the answer has consequences. A credit model may look like it uses debt burden while quietly routing through a neighborhood proxy. A medical model may highlight the right part of an image for the wrong internal reason. A chatbot may refuse one harmful request and fail on a paraphrase because the refusal behavior is a brittle circuit, not a stable rule.

In each case, a surface explanation is only the beginning. Mechanistic interpretability asks for the internal story that would let us debug, predict, and intervene.

The hard part is that neural networks do not come apart cleanly. A useful feature may be spread across many neurons. One neuron may participate in several unrelated behaviors. The part you want to name may not line up with any one coordinate in the model at all.

So the workflow has to be scientific rather than decorative: observe a behavior, inspect candidate internal parts, form a mechanism hypothesis, intervene, and see whether the model changes in the way your story predicts.

Suppose a fraud model suddenly flags every purchase from a new payment processor. A weak explanation says, "processor name was important." A mechanistic investigation asks where processor identity appears internally, which downstream features read it, and whether removing that pathway changes only the suspicious behavior or damages fraud detection more broadly.

The last distinction matters. Breaking the model is easy. Locating the computation is the work.

A programmer writes an algorithm as instructions. A neural network learns an algorithm as structure. A transformer can learn to copy a repeated pattern, retrieve a capital city, track a closing bracket, or switch into refusal mode without anyone writing source code for that behavior.

The weights are the implementation. The learned algorithm is the recurring computation those weights produce when the right kind of input arrives.

In production, this changes how you debug. If an LLM knows that Paris is in France but fails when the sentence is long, the missing piece may not be the fact. The model may represent the right fact and then lose the route that carries it to the output.