Mechanisms

Take the activation vector $a$ from one layer. The linear representation hypothesis says a concept can often be read along a direction $f$. Its strength is approximately a dot product:

$$s = a \cdot f = \sum_i a_i f_i$$

Each term asks how much neuron $i$ contributes to alignment with the feature direction.

Linear directions are practical. A team can train a linear probe, add a steering direction, or compare activations without decoding the entire network. The hypothesis has limits. Curved or context-dependent representations may need nonlinear tools, and a decodable direction does not prove the model uses it.

Early vision layers often respond to edges, color contrasts, or texture. Later layers can combine those signals into curves, wheels, dog heads, or object parts. This hierarchy is why curve detectors are such a useful case study: we can often read their construction from earlier oriented edges.

Language features can track syntax, topic, style, entities, code patterns, or the model's current plan. A feature might activate on a number describing a group of people, a variable name bound to a list, or language that signals a refusal.

The same token can produce different features in different contexts. “Bank” beside “river” should create a different internal state than “bank” beside “loan.”

An internal representation is the model's working state. It can contain facts about the input, partial results, uncertainty, and instructions for later layers. The state changes as information moves through the network.

Think of it as a crowded whiteboard. Features are directions we use to read particular notes from that board.

Representations are where many debugging questions meet. Does the model encode the correct patient condition? Does that information survive to the decision layer? Is the right feature present but ignored? Separating representation from use prevents a common mistake: assuming that information found by a probe caused the output.