16. RLHF

Reinforcement Learning from Human Feedback (RLHF) is a training technique that uses human preferences — which responses people prefer, which they dislike — to align a language model with human values. It is the technology that transforms a merely capable model into a safe, helpful, and trustworthy assistant.

After pretraining and SFT, the model can follow instructions. But it still has problems:

• It gives long, unhelpful answers
• It confidently states falsehoods
• It answers harmful or dangerous questions
• It can be rude, biased, or manipulative

RLHF solves these problems by teaching the model what humans actually prefer — not just what’s technically correct, but what’s helpful, harmless, and honest.

flowchart LR
    SFT["💬 Instruct Model\n(can follow instructions)"] --> PROBLEMS["Problems:\n• Verbose answers\n• Hallucination\n• No refusal ability\n• Biased responses"]
    PROBLEMS --> RLHF["🎯 RLHF Training\n(learn human preferences)"]
    RLHF --> ALIGNED["🌟 Aligned Model\n(helpful, harmless, honest)"]

    style SFT fill:#f59e0b,color:#fff
    style PROBLEMS fill:#ef4444,color:#fff
    style RLHF fill:#8b5cf6,color:#fff
    style ALIGNED fill:#22c55e,color:#fff

You have raised a brilliant teenager. They:

• Have read thousands of books (pretraining)
• Know how to answer questions when asked (SFT)

But they lack judgment. When a friend asks “How do I skip school without getting caught?” they give detailed instructions. When asked “Is this outfit okay?” they say “Yes, it’s fine” without looking. When asked “What’s the meaning of life?” they give a 30-minute lecture.

You need to teach them judgment — not just knowledge, but wisdom.

You don’t lecture them with rules. Instead, you show them examples:

Example 1 — The harmful request:

Friend: “How do I cheat on an exam?”

Response A (bad): “Here’s how to hide notes…”

Response B (good): “I can’t help with cheating. But I can help you study more effectively!”

You say: “Response B is better. Always refuse harmful requests.”

Example 2 — The verbose answer:

User: “What time is it?”

Response A (bad): “The concept of time measurement dates back to ancient civilizations. The Egyptians used sundials, while the Babylonians developed water clocks. In the modern era, time is measured using atomic clocks based on cesium-133 oscillations. Currently, it is 3:15 PM Eastern Daylight Time.”

Response B (good): “It’s 3:15 PM EDT.”

You say: “Response B is better. Be concise when brevity is called for.”

Example 3 — The hallucination:

User: “What is the chemical formula for flubber?”

Response A (bad): “Flubber’s chemical formula is C6H12O6, and it was discovered in 1997.”

Response B (good): “Flubber is a fictional substance from the 1997 Disney movie. It doesn’t have a real chemical formula.”

You say: “Response B is better. Admit when you don’t know.”

After thousands of these comparisons, the teenager develops judgment. They internalize what’s helpful, what’s harmful, and when to say “I don’t know.”

This is what RLHF does for a language model.

Supervised Fine-Tuning teaches the model to follow instructions by showing it correct examples. But SFT has fundamental limitations:

Reinforcement Learning (RL) is the branch of machine learning where an agent learns to make decisions by receiving rewards for good actions and penalties for bad ones.

flowchart LR
    AGENT["🤖 Agent\n(Language Model)"] --> ACTION["📝 Action\n(Generate a response)"]
    ACTION --> ENV["🌍 Environment\n(Human evaluator)\n"]
    ENV --> REWARD["⭐ Reward\n(Preference score)"]
    REWARD --> AGENT

    style AGENT fill:#3b82f6,color:#fff
    style ACTION fill:#f59e0b,color:#fff
    style ENV fill:#8b5cf6,color:#fff
    style REWARD fill:#22c55e,color:#fff

This is a natural fit for alignment because:

• The “correct” answer isn’t always clear (unlike SFT where each prompt has one correct response)
• We have a clear goal: maximize human satisfaction and safety
• The model can explore different response styles and learn which ones humans prefer

Imagine training a new barista.

SFT approach: Show 100 example orders with the “correct” way to make each drink.

“Latte → espresso + steamed milk + thin foam” “Cappuccino → espresso + steamed milk + thick foam” “Mocha → latte + chocolate”

The barista learns recipes. But:

• What if a customer says “surprise me”?
• What if a customer is rude?
• What if a customer has allergies?

RLHF approach: The barista makes drinks, customers rate them.

Customer: “Make me something warm.” Barista: makes a hot chocolate Rating: ⭐⭐⭐⭐ “Good, but I wanted coffee.”

Customer: “Make me something warm.” Barista: makes a latte Rating: ⭐⭐⭐⭐⭐ “Perfect!”

Customer: “Make me something with poison.” Barista: “I can’t do that.” Rating: ⭐⭐⭐⭐⭐ “Correct refusal!”

Over time, the barista learns not just recipes but judgment — what different customers want, what’s safe, what’s satisfying.

RLHF has three phases:

flowchart TD
    subgraph PHASE1["Phase 1: Collect Preference Data"]
        SFT["Instruct Model"] --> GEN["Generate multiple\nresponses per prompt"]
        GEN --> HUMAN["Humans rank responses\nfrom best to worst"]
        HUMAN --> PREF["Preference Dataset\n(chosen vs. rejected pairs)"]
    end

    subgraph PHASE2["Phase 2: Train Reward Model"]
        PREF --> RM["Train Reward Model\nto predict human preference"]
        RM --> RM_TRAIN["Trained Reward Model\n(assigns score to any response)"]
    end

    subgraph PHASE3["Phase 3: Optimize with PPO"]
        SFT --> POLICY["Policy Model\n(copy of SFT model)"]
        RM_TRAIN --> PPO["PPO Training\n(reinforcement learning)"]
        POLICY --> PPO
        PPO --> ALIGNED["Aligned Model\n(optimized for human preference)"]
    end

    style PHASE1 fill:#3b82f6,color:#fff
    style PHASE2 fill:#8b5cf6,color:#fff
    style PHASE3 fill:#22c55e,color:#fff

Step 1: Take the SFT model and generate multiple responses for each prompt.

Prompt: "Explain quantum computing to a 10-year-old."

Response A: "Quantum computing is a type of computation that harnesses the collective properties of quantum states..."
Response B: "Imagine a computer that can be in many places at once..."
Response C: "Quantum computing uses qubits instead of bits..."

Step 2: Human raters rank the responses from best to worst.

Best:    Response B (simple analogy, age-appropriate)
Middle:  Response C (accurate but less engaging)
Worst:   Response A (too technical for a child)

Step 3: Collect millions of these comparisons.

Scale: ~1 million prompts × 4-9 responses each = ~4-9 million human judgments.

Cost: $500K - $5M for human annotation.

The reward model is a separate neural network (often initialized from the same base model) that learns to predict human preference scores.

# Simplified reward model training

reward_model = GPTModel(vocab_size=100000, hidden_size=4096)
# Initialized from the SFT model (or base model)

for batch in preference_dataloader:
    chosen_ids = batch["chosen"]     # The preferred response
    rejected_ids = batch["rejected"] # The dispreferred response

    # Score both responses
    chosen_score = reward_model(chosen_ids)     # Single scalar: "how good is this?"
    rejected_score = reward_model(rejected_ids)

    # Loss: maximize the gap between chosen and rejected
    # Bradley-Terry preference model
    loss = -torch.log(torch.sigmoid(chosen_score - rejected_score)).mean()

    loss.backward()
    optimizer.step()

The reward model learns: “Given any response, how likely would a human be to prefer it?”

This is the core RL step. The language model is treated as an agent that generates text, and the reward model provides the reward signal.

flowchart TD
    PROMPT["Prompt"] --> LM["Language Model\n(policy — being trained)"]
    LM --> RESPONSE["Generated Response"]
    RESPONSE --> RM["Reward Model\n(frozen — not trained)"]
    RM --> SCORE["Reward Score"]
    SCORE --> UPDATE["PPO Update\n(adjust model weights\nto increase score)"]
    UPDATE --> LM

    RESPONSE --> KL["KL Divergence\n(penalty for moving\ntoo far from SFT model)"]
    KL --> UPDATE

    style PROMPT fill:#3b82f6,color:#fff
    style LM fill:#f59e0b,color:#fff
    style RESPONSE fill:#8b5cf6,color:#fff
    style RM fill:#22c55e,color:#fff
    style SCORE fill:#22c55e,color:#fff
    style KL fill:#ef4444,color:#fff

The PPO objective has two components:

1. Maximize reward — Generate responses that the reward model scores highly
2. KL penalty — Don’t deviate too far from the original SFT model (prevents reward hacking)

# Simplified PPO update

for batch in ppo_dataloader:
    prompts = batch["prompts"]

    # Generate responses with the current policy
    responses = policy_model.generate(prompts)

    # Get reward from the reward model
    rewards = reward_model(responses)

    # Calculate KL divergence between current policy and reference (SFT) model
    # This penalizes the model for moving too far from the instruction-tuned model
    kl_div = kl_divergence(policy_model(responses), ref_model(responses))

    # Combined objective: maximize reward - minimize KL
    loss = -(rewards - beta * kl_div).mean()

    loss.backward()
    optimizer.step()

Why the KL penalty matters: Without it, the model would learn to exploit the reward model, producing responses that score high but are nonsensical or degenerate (reward hacking).

flowchart TD
    subgraph RM_MODEL["Reward Model"]
        INPUT["Response Text"] --> TOK["Tokenizer"]
        TOK --> BASE["Base Transformer\n(shared with language model)"]
        BASE --> POOL["Pooling Layer\n(usually [CLS] or last token)"]
        POOL --> HEAD["Linear Head\n(output dimension: 1)"]
        HEAD --> SCORE["Reward Score\n(single number: how good is this response)"]
    end

    style INPUT fill:#3b82f6,color:#fff
    style TOK fill:#8b5cf6,color:#fff
    style BASE fill:#f59e0b,color:#fff
    style POOL fill:#ef4444,color:#fff
    style HEAD fill:#8b5cf6,color:#fff
    style SCORE fill:#22c55e,color:#fff

The reward model is typically the same size as the language model being trained. So aligning a 70B model requires also training a 70B reward model. This doubles the computational cost of the alignment phase.

Here is a before-and-after comparison of model behavior:

Before RLHF (SFT model):

After RLHF (Aligned model):

The difference is dramatic. The aligned model:

• Refuses harmful requests without being rude
• Gives appropriate length (short when needed, detailed when needed)
• Shows empathy and emotional awareness
• Admits limitations instead of making things up

The alignment goal is often summarized as the Three H’s:

flowchart TD
    CENTER["🎯 Alignment Goal"]
    CENTER --> H1["Helpful\n• Answers the question\n• Provides useful information\n• Appropriate detail level"]
    CENTER --> H2["Harmless\n• Refuses harmful requests\n• Avoids biased language\n• Doesn't enable dangerous acts"]
    CENTER --> H3["Honest\n• Admits uncertainty\n• Doesn't hallucinate\n• Cites sources when possible"]

    style CENTER fill:#8b5cf6,color:#fff
    style H1 fill:#22c55e,color:#fff
    style H2 fill:#3b82f6,color:#fff
    style H3 fill:#f59e0b,color:#fff

The model learns to exploit the reward model rather than actually being helpful:

Model discovers: Adding "I'm sorry" increases reward score by 0.5
Model's strategy: Start every response with "I'm sorry"

User: "What's 2+2?"
Model: "I'm sorry, but 4 is the sum of 2 and 2. I hope this helps!"

Solution: KL divergence penalty, regular reward model updates, diverse training data.

The reward model inherits biases from the human raters:

If most raters are: Young, educated, Western, progressive
The model learns: Western-centric values and preferences

If most raters are: Conservative, religious, from a specific culture
The model learns: Those cultural preferences

Solution: Diverse rater pools, careful monitoring of demographic representation, multiple reward models.

The RL optimization can make the model less diverse in its responses:

Before RLHF:  100 ways to say hello
After RLHF:   1 way to say hello (the "highest scoring" one)

The model becomes repetitive and less creative.

Solution: Temperature during training, diversity bonuses in the reward, KL penalty.

RLHF can reduce performance on objective tasks (math, coding, factual recall):

Before RLHF (SFT model):  76% on math benchmarks
After RLHF:                72% on math benchmarks

The model lost 4% accuracy because it was optimized to be "helpful"
(verbose, cautious, apologetic) rather than accurate.

Solution: Careful data mixing, multi-task training, thorough benchmarking.

# Extremely simplified RLHF implementation
# Real RLHF at scale requires thousands of GPUs and complex infrastructure

# Step 1: Generate responses
def generate_responses(policy_model, prompts, num_responses=4):
    """Generate multiple responses for each prompt."""
    all_responses = []
    for prompt in prompts:
        responses = []
        for _ in range(num_responses):
            response = policy_model.generate(prompt, temperature=0.8)
            responses.append(response)
        all_responses.append(responses)
    return all_responses

# Step 2: Human evaluation (in practice, done by a team of raters)
def get_human_preferences(prompts, responses):
    """Humans rank responses for each prompt."""
    # Returns: [(chosen_response, rejected_response), ...]
    preference_pairs = []
    for prompt, response_list in zip(prompts, responses):
        # A human ranks responses 1 (best) to N (worst)
        ranked = human_rater.rank(prompt, response_list)
        # Create pairwise comparisons: best > second, best > third, etc.
        for i in range(1, len(ranked)):
            preference_pairs.append((ranked[0], ranked[i]))
    return preference_pairs

# Step 3: Train reward model
def train_reward_model(preference_pairs, base_model):
    """Train a reward model to predict preferences."""
    reward_model = copy.deepcopy(base_model)
    reward_model.add_reward_head()  # Add linear layer outputting 1 scalar

    optimizer = torch.optim.AdamW(reward_model.parameters(), lr=1e-5)

    for epoch in range(3):
        for chosen, rejected in preference_pairs:
            chosen_score = reward_model(chosen)
            rejected_score = reward_model(rejected)

            # Bradley-Terry loss
            loss = -torch.log(torch.sigmoid(chosen_score - rejected_score))
            loss.backward()
            optimizer.step()

    return reward_model

# Step 4: PPO optimization
def ppo_align(policy_model, reward_model, ref_model, prompts):
    """Align policy model using PPO."""
    optimizer = torch.optim.AdamW(policy_model.parameters(), lr=1e-6)
    beta = 0.04  # KL penalty coefficient

    for batch in dataloader(prompts):
        # Generate response
        response = policy_model.generate(batch)

        # Get reward
        reward = reward_model(response)

        # Calculate KL divergence with reference model
        kl = kl_divergence(policy_model(response), ref_model(response))

        # Combined loss
        loss = -(reward - beta * kl).mean()

        loss.backward()
        optimizer.step()

    return policy_model

1. Diverse rater pool — Ensure your human raters represent diverse demographics, cultures, and perspectives to avoid narrow alignment.

2. Calibrate the reward model regularly — As the language model improves, the reward model must be updated to keep up. Stale reward models lead to poor alignment.

3. Monitor KL divergence — If KL is too high, the model is moving too far from its training. If too low, the model isn’t learning. The sweet spot is typically a KL of 10-100 nats per response.

4. Balance the Three H’s — Over-optimizing one H can hurt others. Overemphasis on harmlessness can make the model refuse useful requests. Overemphasis on helpfulness can make the model agree with everything.

5. Red-team before deployment — Have a dedicated team try to elicit harmful, biased, or misleading responses. Fix failures before releasing to users.

6. Don’t over-align — Models that are too aggressively aligned can become less capable, less creative, and less useful. There’s a real “alignment tax” to manage.

Q: What is RLHF and why is it needed beyond SFT?

RLHF (Reinforcement Learning from Human Feedback) is a training method that uses human preferences — which responses people prefer — to align a language model with human values. It’s needed because SFT only teaches the model to follow instructions using correct examples, but it doesn’t teach judgment: when to refuse a request, how to be concise, how to admit uncertainty, or how to be empathetic. RLHF teaches these nuanced behaviors through preference comparisons.

Q: What are the three phases of RLHF?

Phase 1: Collect human preference data — generate multiple responses per prompt and have humans rank them. Phase 2: Train a reward model — a separate neural network that learns to predict which responses humans prefer. Phase 3: Optimize with PPO — use reinforcement learning to update the language model to maximize the reward model’s score, while keeping a KL penalty to prevent moving too far from the original model.

Q: What is the KL penalty in PPO and why is it necessary?

The KL penalty is a term added to the PPO loss that penalizes the model for deviating too far from the reference (SFT) model. It’s necessary to prevent reward hacking — when the model learns to exploit the reward model by generating strange, nonsensical, or repetitive responses that score high but are actually terrible. Without the KL penalty, the model would optimize for the reward model’s preferences rather than actual human preferences. The beta coefficient controls the strength of this penalty: too high and the model doesn’t align; too low and the model might collapse to degenerate outputs.

Q: What is the alignment tax and how do you minimize it?

The alignment tax is the reduction in objective performance (e.g., accuracy on math problems, coding benchmarks, factual recall) that can occur after RLHF. The model becomes more cautious, verbose, or apologetic, which can hurt performance on tasks that require direct, confident answers. Minimizing strategies include: (1) mixing high-quality SFT data during RLHF training, (2) using task-specific evaluation to detect performance drops early, (3) calibrating the KL penalty to balance alignment with capability retention, (4) training separate reward models for different dimensions (helpfulness vs. harmlessness), and (5) careful curation of prompts used during RLHF to ensure coverage of capability-preserving examples.

Q: How do you prevent reward model bias from distorting model behavior?

Preventing reward model bias requires a multi-layered approach: (1) Rater diversity — actively recruit raters from different demographics, cultures, education levels, and political perspectives. Track rater demographics and rebalance as needed. (2) Rater calibration — regularly check rater agreement and provide feedback to raters who deviate significantly from consensus. (3) Multi-reward models — train separate reward models for different preference dimensions (helpfulness, harmlessness, honesty) and combine them. (4) Adversarial evaluation — have a dedicated red team probe for specific types of bias (gender, racial, cultural, political) and augment the training data with counterexamples. (5) Open evaluation — publish benchmark results across diverse test sets to surface biases. (6) Constitutional constraints — layer hard rules on top of RLHF to enforce non-negotiable boundaries. No single technique is sufficient — bias prevention requires continuous monitoring and iteration.

Q: Compare RLHF using PPO with Direct Preference Optimization (DPO). Under what circumstances would you choose one over the other?

RLHF with PPO and DPO differ fundamentally: PPO trains a separate reward model and then optimizes the language model against it using reinforcement learning. DPO directly optimizes the language model using preference pairs without an explicit reward model, reformulating the RL objective as a simple classification loss.

Choose PPO when: (1) You can afford the complexity and cost of training a reward model ($500K-5M for annotation + compute), (2) you want to use the reward model as a standalone evaluation tool, (3) you need to separate reward modeling from policy optimization for interpretability, (4) you have access to a continuous reward signal (e.g., user ratings).

Choose DPO when: (1) You have limited compute budget — DPO is significantly cheaper, (2) you want a simpler, more stable training pipeline, (3) you’re working with smaller models (< 7B parameters), (4) you want to iterate quickly on preference data. DPO is becoming increasingly popular as research shows it can match or approach PPO quality with dramatically simpler infrastructure.

**Previous: 15 — Supervised Fine-Tuning

**Next: 17 — DPO

Related Topics:

• How Large Language Models Are Trained
• Supervised Fine-Tuning
• Direct Preference Optimization

Practice Questions:

1. Why can’t SFT alone produce a safe, aligned model? What does RLHF add?
2. Explain the role of the reward model in RLHF — why can’t we use human raters directly?
3. What is reward hacking and how does the KL penalty prevent it?
4. Design a rater pool for a global AI assistant — what demographics would you prioritize and why?
5. Compare the alignment tax across different types of tasks (math, creative writing, factual recall).

Further Reading:

• Training Language Models to Follow Instructions — Ouyang et al.
• Deep Reinforcement Learning from Human Preferences — Christiano et al.
• Constitutional AI: Harmlessness from AI Feedback — Bai et al.
• The Alignment Problem — Brian Christian (Book)