Adversarial Training

A model trained only on clean examples has little reason to behave well on adversarial ones. Adversarial training puts the attack into the training loop.

For each batch, generate attacked inputs, include them in training, and update the model to classify them correctly. The model now sees examples near the decision boundaries an attacker is likely to probe.

The objective contains two linked optimization problems:

$\min_\theta \; \mathbb{E}\left[\max_{\delta \in S} J(\theta, x + \delta, y)\right]$

The inner maximization holds the model fixed and searches the allowed perturbation set $S$ for a high-loss input. The outer minimization holds that attacked input fixed long enough to update the model parameters $\theta$ . Training alternates between the attacker and the learner.

FGSM adversarial training generates each attacked example with one FGSM step. It is relatively cheap and exposes the model to a fast attack.

Projected Gradient Descent adversarial training, or PGD-AT, takes several smaller attack steps. After each step, it projects the input back into the allowed perturbation region. The projection matters because the training example must stay inside the threat model. A powerful attack outside the allowed budget teaches the wrong problem.

Adversarial logit pairing, or ALP, adds a loss that encourages clean and adversarial versions of the same input to produce similar logits. The goal is stability inside the model's output representation.

The Min-Max Training Loop

How this works

Each training round has two steps. First, an attacker searches for the input inside the allowed threat radius (epsilon) that hurts the model most. Then the model takes one gradient step to learn from that example. Switch to clean training to see what happens when the attack step is skipped entirely.

Step 1 of 2: find the worst input inside the threat radius. Click “Find the attack” below.

-epsilonLoss over allowed perturbations+epsilon

Threat radius (epsilon)0.60Learning rate0.35

Rounds trained

Training P(y=1)

69.0%

Worst-case P(y=1)

57.9%

Worst-case confidence over training rounds

Higher is better: the model still predicts the correct label under the worst allowed attack.

Show the numbers behind this round

weight = 0.800, bias = 0.000

delta* (worst-case shift) = -0.600

delta used in training = 0.000

dJ/dweight = -0.230, dJ/dbias = -0.310

weight after update = 0.881, bias after update = 0.109

Play several rounds in adversarial mode and watch the green trend line climb: the model is learning to stay correct even under attack. Reset and replay the same number of rounds in clean training mode and the gray line climbs too, but more slowly and to a lower point — clean training only ever sees the easy input, so it closes the worst-case gap more slowly than a model that trains directly on its hardest example.

⚠

Robust to Which Attack?

Training on small L-infinity image perturbations targets that specific threat model. Patches, rotations, poisoned data, model extraction, and prompt injection require their own tests and mitigations.

The min-max view explains the cost. Every training update may contain several attack steps, and a weak inner attack can make the outer model look robust when it has only learned to defeat that weak attack.

Clean accuracy can fall because the model stops using features that are predictive on ordinary data but unstable under perturbation.

Checkpoint

Why does PGD project the attacked input back into the allowed perturbation region after each step?

←PreviousPrompt Injection and SycophancyAdversarial Attacks Next→Certified DefensesAdversarial Defenses