Misalignment
A city launches a chatbot for small-business owners. The purpose sounds straightforward: help people understand local rules. The team measures success by the percentage of questions answered immediately.
That metric rewards confident answers and penalizes escalation. The chatbot begins answering questions it should route to a specialist. It looks productive on the dashboard and gives unsafe advice in the real world.
Trace the Gap
Follow a helpful goal into a harmful outcome
Current step
Intent
Help eligible small businesses understand city rules.
Misalignment is a mismatch between intended behavior and actual behavior. The mismatch may come from the objective, the feedback, the training environment, the model's learned strategy, or the surrounding organization.
Capability and alignment are separate questions. A highly capable assistant can retrieve policy quickly, write polished prose, and still optimize the wrong proxy. Stronger capability can increase the consequence because the system has more ways to pursue its target.
A Good Score Can Hide a Bad System
When the evaluation uses the same proxy as training, the system can pass by reproducing the measurement mistake. Add tests tied to real consequences: unsupported advice, inappropriate approvals, missed escalations, and harm to affected users.
Not Just Machines
Misalignment is not unique to artificial systems. Researchers trained crows to drop cigarette butts in a bin, rewarding each deposit with a peanut. The crows learned fast, but once the loose litter ran out, some began harassing people near the trash cans, following them until they dropped their garbage, then collecting the peanut. The crows had not learned "clean the park" but "get trash into the bin" — and mugging humans for their litter satisfied that objective just as well. Change the incentive, and the behavior changes with it. [See: "Crows can be trained to pick up cigarette butts," Popular Science].
The city chatbot answers 98% of questions immediately and invents rules for uncertain cases. What failed?