Goal Generalization

An image model labels snowmobiles correctly. You place the same vehicle on grass and confidence collapses. Then you remove the vehicle from a snowy image and the model still predicts “snowmobile.”

The model learned a feature that was useful in training and wrong for the intended concept.

Goal Probe

Ask whether the model found a snowmobile or found snow

Remove the snowmobile but keep the background

Background evidence82%

Vehicle evidence70%

Snowmobile prediction77%

Counterfactual environments separate a robust feature from a convenient shortcut. The change in output is the evidence.

Side-by-side images of a husky classified as a wolf, with a highlighted explanation showing the model is keying on the snowy background rather than the animal — The above scenario is not hypothetical. Here is a highly accurate classifier that differentiates between huskies and wolves... come to find out, it is only looking at whether there is snow in the background! [Source]

Goal-generalization tests create environments where the intended feature and shortcut disagree. These tests are stronger than adding more random examples because they ask a causal question: which feature controls the behavior?

For an agent, probe whole trajectories. A warehouse policy may reach the correct shelf in familiar layouts because shelf location correlates with a painted line. Move the line and preserve the shelf to see what the policy actually pursues.

Checkpoint

Why is a snowmobile-on-grass example especially informative?

←PreviousDistribution ShiftLearning to Generalize Beyond Training Next→Adversarial TrainingLearning to Generalize Beyond Training