White-Box and Black-Box Attacks
Before choosing an attack, ask what the attacker knows.
In a white-box attack, the attacker can inspect the model: architecture, parameters, gradients, and often preprocessing. In a black-box attack, the attacker sends inputs and observes outputs. They may see a label, a confidence score, a refusal, or a tool call, but they cannot calculate gradients directly from the target.
Change what the attacker can see
Attacker knows
Architecture, weights, gradients, preprocessing
Likely move
Differentiate the loss with respect to the input and optimize a perturbation directly.
Concrete example
A model owner tests a vision classifier with FGSM before release.
Values are illustrative and are intended for comparing mechanisms, not benchmarking a deployed model.
Switch the attacker from white-box to black-box access and change the query budget.
White-box access makes optimization direct. If you can compute the gradient of the loss with respect to an input, the model tells you which direction will increase its error.
Black-box access turns the problem into experimentation. An attacker can search from the feedback, estimate gradients with repeated queries, train a surrogate model, or use attacks that transfer from another model.
Public APIs, chat interfaces, and deployed cameras are black boxes from the outside. Their observable behavior can still support systematic probing.
When White-Box Access Is Realistic
A company may publish an open model, deploy a downloaded checkpoint, share weights with a contractor, or expose them to an insider. Defenders also use white-box attacks during evaluation because those attacks provide a strong test of what the model would do under full knowledge.
A public API returns a probability vector for each query. Which statement is most accurate?