Safety and Reliability

Reliability answers a conditional question: under which conditions does this system keep working?

A bird classifier may be excellent when waterbirds appear on water and landbirds appear on land. That test set rewards a shortcut. Move a waterbird onto a beach or a landbird near a pond and the relationship between label and background breaks.

This is a distribution-shift problem. The deployment environment contains combinations that were rare or absent in training. A model that learned the animal may generalize. A model that learned the scenery may collapse.

Stress-Test the Background

This bird classifier can use the animal's shape or the scene behind it. Increase deployment shift to send more waterbirds onto land backgrounds and landbirds onto water backgrounds.

Weight on bird shape0.65Weight on watery background0.80Shifted cases in deployment50%

Expected deployment accuracy

50%

Waterbird

on water

score = 0.65(1) + 0.80(1) = 1.45

predicts waterbird

Waterbird

on land

score = 0.65(1) + 0.80(-1) = -0.15

predicts landbird

Landbird

on water

score = 0.65(-1) + 0.80(1) = 0.15

predicts waterbird

Landbird

on land

score = 0.65(-1) + 0.80(-1) = -1.45

predicts landbird

Reliability is conditional. A model can perform well where the shortcut holds and fail when the environment breaks that relationship.

Chart showing waterbirds vs. landbirds — The above example is from an actual study, where they looked at waterbirds and landbirds, switching backgrounds and examining the inferences with post-hoc explainability techniques [Source]

Safety adds consequence. A mistaken playlist recommendation is annoying. A mistaken speed-limit reading can change how a vehicle behaves. A medical triage error can delay care. The same technical failure deserves different controls in different settings.

Explanations help teams form and test hypotheses about failure. If a saliency method points to the background, create counterexamples with swapped backgrounds. If the model appears sensitive to a one-pixel artifact, remove or vary that artifact. If a language model follows instructions hidden in retrieved documents, test the tool boundary and permission model.

A single explanation is weak evidence. Reliability comes from repeated tests across realistic conditions, monitored after deployment.

Checkpoint

A bird classifier is accurate on familiar backgrounds and poor on swapped backgrounds. What is the strongest conclusion?

←PreviousAccountabilityWhy Do We Need Explainable AI?Next→Debugging and Model ImprovementWhy Do We Need Explainable AI?