Attack Surfaces Across an AI System
When people picture an AI attack, they often picture someone manipulating the model itself. In production, the model is one part of a longer chain.
Data is collected and labeled. A model or adapter is downloaded or trained. Inputs pass through preprocessing. Outputs may reveal scores or explanations. An agent may call tools. A person may accept the recommendation without review. Every handoff creates an attack surface.
Follow the system from data to action
Select each stage and ask what an attacker can control or observe.
Question to ask
Who can add, label, rank, or publish material that enters training?
Attack at this stage
Poisoning or backdoor insertion
Example: A malicious labeling vendor marks known malware as benign.
Why it matters: The mistake becomes part of the trained model and can persist across every future prediction.
Trace the system from training data to downstream action and inspect the attack available at each stage.
The objective is therefore to defend the full system you deployed.
A robust classifier still fails if the data pipeline is poisoned. A careful system prompt still fails if the model can call a refund tool with unchecked arguments. A good detector still fails if operators learn to ignore its alerts. Security work follows the full path from input to consequence.
Use Explainability to Find Leverage Points
Explanations can show which features influence a model, which training examples shape a prediction, and where confidence changes sharply. Those same clues help a defender identify features an attacker may manipulate. Explainability becomes part of attack-surface discovery.
Choose an AI feature you have built or used. Trace one path from user input to real-world consequence. Where would you place the first control?