A Map of Adversarial Attacks
Poisoning changes what the model learns before deployment. Evasion changes an input after deployment. Extraction and inference learn from the outputs a deployed model reveals. The same application can face all three.
One Spam Filter, Three Attacks
- Poisoning: publish large numbers of legitimate-looking messages with a chosen phrase so future training data associates the phrase with "not spam."
- Evasion: change spelling, spacing, images, or Unicode characters until one message passes the current filter.
- Extraction: submit many messages, record scores, and learn which features move the decision.
Classify each scenario into the correct attack type
A researcher adds imperceptible noise to a stop sign image, causing a deployed self-driving car system to classify it as a speed limit sign.
A competitor queries your text classification API 50,000 times with varied inputs, collecting all the (input, output) pairs to train a copy of your model.
A malicious labeling contractor mislabels 0.5% of your training images before you receive the dataset, causing your model to consistently misclassify a specific class.
An actor submits thousands of fake reviews with specific text patterns, knowing they'll be scraped and used to train the next version of a sentiment model.
A user puts on glasses with an adversarial pattern that causes a facial recognition system to identify them as someone else.
A security researcher systematically varies inputs to a spam filter API, mapping out the boundary between 'spam' and 'not spam' to understand the model's decision rules.
Drag a scenario or use its placement buttons. Select Check to see the reasoning.
A second distinction cuts across this map: targeted and untargeted attacks.
A targeted attack wants a specific outcome, such as "classify this turtle as a rifle" or "approve this transaction." An untargeted attack wants any wrong outcome or broad performance damage. Targeted attacks are often harder to construct and easier to monetize because the attacker knows exactly what success looks like.