Evaluating Robustness
A defense can look strong because the evaluation is weak. The attack may use too small a budget, too few steps, the wrong access assumptions, or a method the defense was designed to block.
Robustness evaluation begins by writing the threat model next to the score.
Three Accuracy Measures
- Clean accuracy: performance on ordinary, non-adversarial inputs.
- Robust accuracy: performance on adversarial examples generated under a specified attack and budget.
- Mixed-set accuracy: performance on a dataset containing both clean and adversarial examples.
Score the defense honestly
Clean accuracy
92%
Robust accuracy
32%
Mixed-set accuracy
62%
Values are illustrative. Real results depend on the dataset, model, attack, and threat model.
An adaptive attack is designed with knowledge of the defense. If the defender squeezes image features, the attacker optimizes an input that survives squeezing. If the system blocks one jailbreak phrase, the attacker varies the role, encoding, order, and context. A defense-aware test asks whether the underlying weakness remains.
For production systems, evaluate consequences as well as model outputs. Count unauthorized tool calls, harmful content that reaches users, sensitive fields revealed, and cases escalated to human review. A model can preserve accuracy while the surrounding workflow still fails.
A Useful Robustness Report
"The classifier has 92% clean accuracy and 61% robust accuracy against a 20-step white-box PGD attack with an L-infinity budget of 8/255. A patch attack reduces detection to 34%. The patch result is outside the training threat model and requires a separate mitigation."
This report is actionable because it names the attack, access, budget, and remaining weakness.
What makes an evaluation adaptive?