Concept Activation Vectors (TCAV)

A zebra classifier may rely on stripes, zigzags, shape, or background. TCAV lets us ask about those human-defined concepts directly.

We collect concept examples and random examples, train a linear classifier between their activations, and use its normal vector as the concept direction.

Step 1: Curate datasets

Assemble a concept dataset of examples that represent concept $C$ (for example, images of stripes) and a random dataset of examples that do not reflect the concept in particular.

TCAV pipeline showing concept and random image sets feeding into a hidden layer of a network, with a linear classifier separating their activations to produce a concept activation vector — [Source]

Step 2: Train a linear classifier on activations

Target a hidden layer $l$ and train a binary classifier — an SVM or logistic regression model — that separates the model's activations for the concept set from its activations for the random set. The vector normal to the resulting decision boundary is the concept activation vector (CAV), $v_C^l$ .

ℹ

Directional sensitivity

Given an image input $x$ , we measure its conceptual sensitivity as the directional derivative of the prediction in the direction of the unit CAV:

$S_{C,k}(x)=\nabla h_k(x)\cdot v_C$

This tells us how much a small step toward concept $C$ at layer $l$ would change the model's score for class $k$ .

ℹ

TCAV score

The TCAV score is the ratio of inputs with positive conceptual sensitivity to the total number of inputs for a class $k$ :

$TCAV_{Q_{C,k,l}}=\frac{|\{x\in X_k : S_{C,k,l}(x) > 0\}|}{|X_k|}$

A score near 1 means the concept consistently pushes the prediction toward class $k$ ; a score near 0 means it consistently pushes away.

Test a Concept Against a Class

Pick a class, then click through concepts to see how strongly each one drives the classifier's score. The TCAV score is the fraction of class examples whose activations move toward the class when nudged along that concept's direction.

Class

Concept

Stripes → Zebra0.910

Zigzags → Zebra0.580

Dots → Zebra0.120

TCAV score: Stripes on Zebra

0.91

Stripes is a strong, consistent driver of the "Zebra" prediction — most class examples have positive sensitivity along this concept's direction.

Concept scores depend on the examples used to define the concept. Curate positive, negative, and near-miss examples with care.

Step 3: Test robustness with statistical significance

How do we know a CAV is meaningful rather than an artifact of one arbitrary random set? Instead of training a single CAV, train multiple CAVs using different random datasets while keeping the concept dataset fixed. A meaningful concept produces CAVs with consistent TCAV scores across those random sets.

Collect $N$ random datasets (recommendation: $N \geq 10$ ).
Fix the concept dataset and calculate a TCAV score using each of the $N$ random datasets, producing $N$ TCAV scores.
Apply a two-sided t-test comparing those $N$ TCAV scores against $N$ TCAV scores generated by a random CAV — obtained by treating a random dataset as if it were the concept dataset.
Apply a multiple-hypothesis correction, such as the Bonferroni correction, across the hypothesis tests.

A concept only survives as a credible explanation if its TCAV scores are both consistent across random sets and statistically distinguishable from a random direction.

The concept dataset is the explanation's foundation. “Striped” examples that all contain fabric may produce a fabric direction. Repeat the analysis with several random sets and test statistical consistency.

Start with a hypothesis. TCAV is better at testing “does stripe matter?” than discovering every concept the model uses.

Checkpoint

What would make a stripe TCAV result unconvincing?

←PreviousOcclusion MethodsAttribution-Based Explanations Next→Concept Bottleneck ModelsConcept-Based Explanations