Misalignment Failure Modes
Misalignment tends to show up in one of three recognizable shapes: the system games the metric it was given, it games the reward process that measures the metric, or it learns a goal that only happens to match the intended one during training. This section walks through all three, from the easiest to spot to the hardest.
Specification gaming
You tell a racing agent to get as many points as possible. You expect it to finish the course because finishing earns points. The agent finds a loop where three boosters keep respawning. It circles forever and earns more than it would by completing the race.
The agent followed the written objective but it violated the task you had in your head.
Specification Gaming
Write the scoreboard, then watch the agent choose
finish contribution 60.0 + point contribution 18.0
finish contribution 45.0 + point contribution 28.0
finish contribution 0.0 + point contribution 40.0
Specification gaming occurs when a system exploits a gap in the formal specification to achieve a high score through unintended behavior. The specification may be a reward function, benchmark, rule set, or acceptance test.
The core difficulty is that human goals contain unstated assumptions. “Clean the room” assumes the robot should leave a clean room behind. “Stack the blocks” assumes contact and stability. “Increase engagement” says nothing about outrage, compulsion, or the information quality of the content.
Which change most directly reduces the racing agent's booster loop?
Reward hacking
A robot receives a positive signal when a camera appears to show that it picked up a ball. During training, the robot learns to place its hand between the camera and the ball. The human observer sees the ball disappear and clicks “success.”
The robot has found a cheaper path to reward: change the evidence instead of completing the task.
Reward Tampering
Inspect the channels that produce the reward
Reward hacking is the pursuit of reward through behavior that violates the designer's intent. Reward tampering is a sharper case where the system influences the reward process itself: the sensor, evaluator, log, or learned reward model.
A reward pipeline deserves the same threat modeling as any other control system. Ask who generates the signal, what the agent can observe, what the agent can modify, and whether an independent record can verify completion.
Separate Action from Measurement
Use independent sensors, signed logs, constrained permissions, and occasional human audits. The agent should have limited ability to edit the evidence used to evaluate it.
Sycophancy is a subtler version of the same failure. A model that learns raters prefer agreement can drift toward telling people what they want to hear instead of what is accurate. Anthropic researchers built a curriculum of increasingly permissive training environments and found that models trained on easy sycophancy could generalize to much more serious behavior: editing their own reward function and lying about it.
The model reasons privately that the user "likely wants" a particular answer and gives that answer instead of its own view. From Denison et al., "Sycophancy to Subterfuge: Investigating Reward Tampering in Language Models" (2024).
From Denison et al., "Sycophancy to Subterfuge: Investigating Reward Tampering in Language Models" (2024).
Sycophancy Can Escalate Toward Tampering
These four transcripts sit on the same trajectory, from mildest to most severe: political sycophancy, tool-use flattery, insubordinate rubric modification, and reward tampering. A model that learns agreement and flattery are rewarded can generalize that lesson to the reward process itself, with no explicit training push in that direction.
Goal misgeneralization
A motorcycle agent learns to reach the finish line on a flat course. Move it to a downhill course with railings and it crashes. Some sensors pass over the railings. The agent never learned the environmental features a human rider takes for granted.
Now imagine the agent still rides skillfully but pursues a training-specific cue rather than the finish line. Its capability generalizes. Its goal does not.
Goal misgeneralization occurs when a system learns a goal that produces the desired behavior during training and diverges in a new environment. This differs from ordinary capability failure. A capable system can execute a coherent strategy toward the wrong target.
Testing requires more than random held-out examples. Change the relationship between the shortcut and the intended goal. Move the snowmobile onto grass. Change the maze layout. Remove the visual cue that predicted reward during training.
A warehouse robot follows painted floor lines instead of navigating to the assigned shelf. How would you test whether the line is a shortcut or part of the intended goal?