Core Hypotheses
Mechanistic interpretability begins with three speculative claims. Models represent meaningful features. Features interact through circuits. Similar tasks may produce recurring features and circuits, a claim called universality.
Three claims
- Features: meaningful properties correspond to directions in activation space.
- Circuits: weighted connections compose features into computations.
- Universality: analogous solutions can emerge across models trained on similar problems.
The lecture adds two working assumptions that make reverse engineering feel possible: decomposability and linearity. Decomposability says a representation can be described in terms of parts that are relatively understandable on their own. Linearity says many of those parts can be read as directions in activation space.
Lecture assumptions
- Decomposability: independently understandable features can describe useful pieces of a representation.
- Linearity: features often behave like directions or linear combinations of neurons in a layer.
- Limitation: the same neuron may support several features, and one feature may spread across many neurons.
Consider a neural-network based computer vision model trained on ImageNet. It performs well on tasks like image classification, accurately distinguishing between a thousand different objects, everything from cats and dogs to peanut butter and jelly sandwiches.
In a large network like this one, the model learns to extract and combine numerous feature representations from the input image data. In early layers it may learn low-level features like edges and textures. In deeper layers, it combines those into higher-level features that represent more abstract concepts like shapes and objects.
The key intuition behind decomposability is that each of these feature representations is relatively independent and understandable on its own. A feature representation for "dog" is roughly independent of the one for "cat," which is roughly independent of the one for "PB&J sandwich."
This independence is what lets us break the model down instead of holding the entire computation in our heads at once. It also helps us avoid the curse of dimensionality.
The curse of dimensionality
The curse of dimensionality refers to the challenges that arise when working with high-dimensional data:
- As the number of dimensions (features) in a dataset increases, the data becomes increasingly sparse in the high-dimensional space.
- The amount of data required to maintain the same density grows exponentially with the number of dimensions.
- Distance measures become less meaningful and discriminative in high-dimensional spaces (e.g., most data points appear almost equidistant from one another).
- Visualization and interpretation of high-dimensional data become increasingly difficult.
Decomposability has limits
Neural networks are not fully decomposable. The dog, cat, and PB&J example is a useful simplification, but real features often overlap and share neurons rather than sitting in neat, independent directions. We will return to this later in the module and look at what it means for mechanistic interpretability.
Test Decomposability on Three Concepts
Activate a concept and watch how much of its evidence lands on neurons the other concepts also use. In the idealized case each concept owns its own neuron. Switch on entanglement to see what real, overlapping features look like.
dog neuron
0.85
cat neuron
0.10
sandwich neuron
0.05
Cross-talk with cat
0.17
Cross-talk with PB&J sandwich
0.09
Turning "dog" on barely moves the other concepts' neurons. You can study, describe, or intervene on this feature without reasoning about the rest of the model.