Poisoning and Backdoors
Evasion attacks one prediction. Poisoning changes the learning process.
An attacker may add fake records, flip labels, alter features while preserving labels, or contribute a model artifact that already contains malicious behavior. The effect can persist through the operational life of the trained model.
Start with a small classifier that represents each class by its centroid, the mean location of its training points. A new point receives the label of the nearest centroid. The decision boundary sits where the two distances are equal.
A mislabeled training point changes the centroid calculation. Several strategically placed points can move the boundary toward a target while leaving most clean examples on the correct side.
Poison the Learned Boundary
Nearest-centroid classifier
class(x) = arg min_k ||x - centroid_k||2
c0 = [-0.59, -0.49]
c1 = [0.57, 0.51]
The mislabeled points pull the class 1 centroid toward class 0. The equal-distance boundary moves with it.
Poison rate
0.0%
Clean accuracy
100%
Target class
0 -> 0
Target still has its clean label
Aggregate accuracy can remain high while one strategically chosen region changes.
Add mislabeled points, recalculate the class centroids, and watch the learned boundary move toward a chosen target.
A backdoor is targeted poisoning with a trigger. The model behaves normally on clean inputs and changes behavior when a chosen pattern appears. A hashtag might bypass moderation. An emoji sequence might activate data leakage. An audio trigger might cause an unauthorized command.
That normal behavior is what makes the attack difficult to catch. Aggregate validation accuracy can stay high because the trigger appears rarely or never in the test set.
Change the training process
Backdoors can produce a large targeted effect while clean accuracy barely moves. That is why ordinary validation can miss them.
Values are illustrative and are intended for comparing mechanisms, not benchmarking a deployed model.
Clean Labels Can Still Carry Poison
In a clean-label attack, the visible label remains correct. The attacker changes features so the model learns a hidden association. A reviewer sees a dog image labeled dog and approves it. The model sees feature-space movement that affects how later cats are classified.
Poison enters through supply chains: compromised labeling services, third-party datasets, public web pages that will be scraped, malicious participants in federated learning, insiders, and downloaded checkpoints.
The art-protection tools Glaze and Nightshade make the power of training-data manipulation concrete. Glaze changes an artwork so a model learns a different style representation than a person sees. Nightshade changes images so their learned concept moves toward another concept, which can corrupt a text-to-image model trained on the poisoned examples. The artist sees a usable image. The training pipeline receives a strategically altered signal.
Why Small Poison Rates Matter
Modern training pipelines collect data at enormous scale and often from sources the model builder only partially controls. A small set of strategically placed examples can shape a target concept, trigger, or decision boundary.