Threat Models
"Could someone attack this?" is too broad to guide engineering. Almost any deployed system can be attacked somehow. A useful analysis gets specific.
Take a fraud detector. A fraud ring wants transactions approved. An insider may want one account exempted. A security researcher wants to demonstrate a weakness without stealing money. These people have different access, incentives, and acceptable costs. They will try different attacks.
Threat Model
A threat model describes an adversary, the access and capabilities available to that adversary, the goal they want to achieve, and the consequence if they succeed.
Build the model with four questions:
- Who benefits from a failure? Name the person or group and their incentive.
- What can they touch or observe? Training data, model files, API inputs, confidence scores, tools, or human operators all create different possibilities.
- What outcome do they want? One targeted mistake, broad performance damage, model theft, private information, or an unauthorized action.
- What constraints do they face? Query limits, cost, time, visibility, physical access, and risk of detection shape what is plausible.
Build a Threat Model
Start with a plausible adversary. Then choose a method that adversary could actually use and the outcome they care about.
Your three choices still need a plausibility check. A threat model is an argument about capability and incentive, not a menu of scary words.
A Concrete Threat Model
System: an airline support agent with a refund tool. Adversary: a customer seeking an unauthorized refund. Access: public chat and visible tool results. Method: claim a false premium status, inject instructions, and repeat variations. Goal: trigger the refund tool. Consequence: financial loss and an account-control failure.
This threat model points toward specific controls: verify status from trusted account data, constrain refund amounts, validate tool arguments, and require approval above a threshold.
Why must a threat model include attacker access and capability?