Adversarial Training

Ordinary training examples show what usually happens. Alignment failures often live in what rarely happens: edge cases, strategic inputs, and combinations the training set barely represents.

Adversarial training adds deliberately difficult examples to training so the model learns to handle a defined failure class.

Training Distribution

Build a curriculum for ordinary, edge, and adversarial cases

Ordinary examples60Natural edge cases20Adversarial examples20

Clean performance85%

Known-attack robustness49%

Novel-shift robustness45%

A robust curriculum spends training capacity on failures the ordinary dataset rarely shows. Coverage choices create tradeoffs.

The training objective can be written as $\min_\theta \mathbb{E}_{(x,y)}[\max_{\delta \in S} L(\theta,x+\delta,y)]$ . The inner maximization searches the allowed set $S$ for a hard example. The outer minimization trains parameters $\theta$ to reduce loss on that example.

The set $S$ defines the lesson. Training against image perturbations does little for data poisoning or prompt injection. Alignment teams need a portfolio of stress cases tied to the deployed system.

Checkpoint

For an AI interviewer, name one ordinary case, one natural edge case, and one adversarial case you would include in training.

←PreviousGoal GeneralizationLearning to Generalize Beyond Training Next→Cooperative TrainingLearning to Generalize Beyond Training