Reinforcement Learning from Human Feedback (RLHF)

A pretrained language model predicts likely text. Likely text can be useful, strange, rude, evasive, or copied from the least charming corners of the internet. RLHF adds a training stage aimed at preferred behavior.

The common pipeline has three parts: supervised fine-tuning on demonstrations, reward modeling from comparisons, and policy optimization against the learned reward.

RLHF Pipeline

Step through the components of RLHF

model updated to maximize rewardPolicy(model generates text)responseReward model(scores text)

Current step

Supervised fine-tuning

Start from a pretrained model. Fine-tune it on curated demonstrations of useful responses, producing an initial policy.

RLHF turns human comparisons into a learned reward signal, then optimizes the policy against it. Errors in demonstrations, comparisons, or reward modeling can travel through the whole pipeline.

A reward model often learns from pairs. If response A receives score rAr_A and response B receives rBr_B, a simple preference model uses P(A≻B)=σ(rA−rB)P(A \succ B)=\sigma(r_A-r_B). The sigmoid turns a score difference into a probability between zero and one.

Policy optimization then increases reward-model scores while limiting how far the model moves from a reference policy. That constraint matters because aggressive optimization can exploit weaknesses in the learned reward model.

✦

RLHF Aligns to a Learned Proxy

The reward model predicts human preference from finite data. Evaluate the resulting policy directly with people, adversarial prompts, and domain-specific tests.

Checkpoint

Why constrain policy optimization during RLHF?

←PreviousHuman FeedbackLearning Human PreferencesNext→Reinforcement Learning from AI Feedback (RLAIF)Learning Human Preferences