Can You Fool the Model?
1 / 7
What is this a picture of?
For each real adversarial example, guess what a human sees before flipping the card to see what the attacked model predicted.
Start with a simple question: what do you see?
A speed limit sign says 35. A turtle looks like a turtle. A handwritten 2 looks like a 2. These feel like easy cases because your visual system is doing much more than matching pixels. You bring shape, context, experience, and a useful sense of what details matter.
A machine learning model sees a different problem. It maps numerical features to a prediction. If a carefully chosen change moves those features across a decision boundary, the prediction changes even when the object still looks the same to you.
To make that failure less mysterious, shrink the problem to two features. A linear classifier draws a boundary through feature space. Points on one side receive one prediction; points on the other side receive another.
The important quantity is the margin: the distance from an input to the boundary. An input with a small margin may need only a small, carefully directed change. Moving parallel to the boundary does very little. Moving perpendicular to it reaches the other side fastest.
Decision Boundary Explorer
Move a transaction through a two-feature classifier. The shortest attack travels perpendicular to the boundary because any sideways movement wastes distance.
Classifier score
z = 1.2x1 - 0.8x2 - 0.2
z = 1.2(0.65) - 0.8(0.40) - 0.2
z = 0.260
Prediction
approve
P(approve)
56.5%
Margin distance
0.180
After attack
review
Move the input, calculate its margin, and reveal the shortest perturbation that crosses the model's decision boundary.
Adversarial Example
An adversarial example is an input intentionally constructed to make a model fail. The change may be tiny and hard for a person to notice, or it may be large and visible, like a printed patch. The defining feature is intent: someone chose the input because of how the model responds.
The two-feature boundary is simple enough to draw. An image classifier draws a much more complicated boundary across thousands or millions of input dimensions, but the attack still tries to find a short path across it.
This is the connection to explainability. An explanation can reveal which features drive a prediction and which features an attacker can manipulate. The model may rely on a texture, a pixel pattern, or a phrase that a person treats as irrelevant.
Ordinary test accuracy answers a different question. A test set asks, "How often is the model right on examples sampled like our data?" An adversary asks, "Can I choose an example that makes the model wrong?"
Why This Matters in Practice
A wrong label on a benchmark is a metric. A wrong road-sign label can change how a vehicle behaves. A prompt injection in a chatbot can become a refund, an email, or a database query when the model has tools. The surrounding system determines the consequence.
Which detail makes an input adversarial?