Extraction and Inference Attacks

A prediction API looks simple: send an input, receive an output. Repeated outputs can reveal much more than the product team intended.

In a model-extraction attack, the attacker collects input-output pairs and trains a surrogate to imitate the target. In an inference attack, the attacker uses outputs to learn something hidden, such as whether a record was in the training data or which features drive a decision.

A surrogate attack follows a practical loop:

1. Collect seed inputs from the target domain.
2. Query the target and save its outputs.
3. Train a surrogate model on those input-output pairs.
4. Create new queries where the surrogate is uncertain or changes rapidly.
5. Ask the target for labels on the new queries and retrain the surrogate.
6. Generate white-box attacks against the surrogate.
7. Submit those attacks to the target and measure how many transfer.

The surrogate does not need the target's exact weights. It needs a boundary that is similar in the part of feature space the attacker cares about. That is why query selection matters. A new query near the current surrogate boundary often teaches more than another query in a region where every model already agrees.

Defenses make extraction more expensive. Rate limits reduce the query budget. Coarser outputs reveal less than full probability vectors. Monitoring can flag systematic boundary searches. Authentication and per-user quotas make it harder to spread queries across accounts.

Each defense also affects legitimate use. A useful API may need calibrated scores. A researcher may need enough queries to audit fairness. The threat model decides which information and volume are acceptable.