Reinforcement Learning Fine-Tuning (RLFT)

RL based Fine Tuning of GPT Models for my own understanding and sharing it with others as a part of Giving-back series.

1. Building Nano-GPT

This is direct copy from the tutorials of Andrej Karpathy on building GPT from scratch. Please refer repo here

Summary:

1. Character and token embeddings: represent character/token as a vector which is used throughout the model instead of working in one-hot space
2. Bigram Model: next token is only dependent on current token i.e. we use $p(x_{t+1} | x_t)$
3. Attention Block intuition: to get the context, change the current embedding to average of embedding $\sum x_{\leq t}$; attention is just weighted combination. How to compute this weights is where attention mechanism comes into play via Query, Key and Value.
4. Nano-GPT model

2. Reinforcement Learning Fine-Tuning of Language Models

Notes are taken from the excellent RL lectures by UC Berkeley, which I really enjoy watching. I highly recommend this lecture series to everyone.

Introduction

Reinforcement Learning Fine-Tuning (RLFT) has become a central paradigm for aligning large language models (LLMs) with human preferences. Unlike classical supervised training, RLFT incorporates feedback (human or automated) to optimize model outputs towards desirable behaviors. This chapter introduces the three major RLFT approaches:

• Reinforcement Learning from Human Feedback (RLHF, typically using PPO),
• Direct Preference Optimization (DPO),
• Group Relative Policy Optimization (GRPO).

Each method is introduced along with its mathematical formulation, nuances, and clarifications of common misconceptions.

Broadly, the training process is as follows:

1. Pre-train the LLM on internet-scale data.
2. Supervised fine-tuning (SFT): Use human demonstrations to fine-tune the model for specific tasks. However, labeled data is limited, and we want the LLM to generalize rather than merely imitate human examples (LLM imitates internet data generated by human/culture). This motivates the next RL fine-tuning stage. RL helps with emergent behaviour i.e. to connect different imitation trajectories to find novel solutions.
3. Fine-tuning with preference data: Given an input prompt, we sample multiple output completions. A judge (human or automated) assigns preferences among the outputs. From this, we can either (a) train a reward model to score completions or (b) directly optimize using preference data (e.g., via DPO).

---

Reinforcement Learning from Human Feedback (RLHF)

Problem Setup

Let $x$ denote a prompt with sequence of tokens as $x=(x_1, x_2 \dots x_t)$, and $y = (y_1,\dots,y_T)$ a completion generated by an autoregressive policy $\pi_{\theta}$:

$$ \pi_\theta(y \mid x) = \prod_{t=1}^T \pi_\theta(y_t \mid x, y_{\lt t}). $$

A reward model $r_\phi(x,y)$ is trained from preference data to provide a scalar score. The goal is to update $\pi_\theta$ such that preferred completions are more likely.

Contextual Bandit View

Unlike standard RL environments, RLHF is essentially a one-step episodic MDP (contextual bandit):

• State: prompt $x$,
• Action: full completion $y$ (it could also be viewed as token-wise action but mostly whole sequence is viewed as action) ,
• Reward: $R = r_\phi(x,y)$,
• Episode terminates after reward is given.

Training a Reward Model with Bradley–Terry

We train a reward model $r_\phi(x,y) \in \mathbb{R}$ from human preference data.
The dataset consists of tuples:

$$ (x, y_h, y_l), \qquad y_h \succ y_l $$

where $x$ is a prompt, $y_h$ is the preferred completion, and $y_l$ is the less preferred one.
Humans (or an automatic judge) only assign relative preference labels.

Bradley–Terry Preference Likelihood: The probability that $y_h$ is preferred over $y_l$ is modeled as:

$$ P(y_h \succ y_l \mid x) = \sigma!\left( r_\phi(x,y_h) - r_\phi(x,y_l) \right), $$

where $\sigma(z) = \frac{1}{1+e^{-z}}$ is the logistic sigmoid.

The reward model is trained by maximizing the log-likelihood of the preferences (probability: where we want to maximize the probability of getting high score of preferred prompt), or equivalently minimizing:

$$ \mathcal{L}(\phi) = - \mathbb{E}_{(x,y_h,y_l)} \left[ \log \sigma \left( r_\phi(x,y_h) - r_\phi(x,y_l) \right) \right]. $$

The reward model is usually initialized from the base policy $\pi_\theta$ (e.g. an SFT model).
A small neural head is added on top of the transformer’s hidden states to output a scalar $r_\phi(x,y)$.

Policy Optimization Objective

Once we have a reward model $r_\phi(x,y)$ that represents human preference (goodness of responses), the goal is to train a new policy $\pi_\theta$ that:

1. Achieves high reward under the reward model, and
2. Stays close to the original supervised model $\pi_{\text{ref}}$ (to avoid drifting too far).

The optimization problem is:

$$ \max_{\pi_\theta} \mathbb{E}_{x \sim \mathcal{D}} \mathbb{E}_{y \sim \pi_\theta(\cdot \mid x)} \left[ r_\phi(x,y) \right] - \beta , \mathrm{KL}(\pi_\theta(\cdot \mid x), \pi_{\text{ref}}(\cdot \mid x)) $$

Why this objective?

• Sampling from policy: $y \sim \pi_\theta(y \mid x)$ means completions are generated by the current policy.
• Maximize reward: The expectation $\mathbb{E}[r_\phi(x,y)]$ pushes the model towards outputs judged as better by humans (or the reward model).
• KL penalty: The term $D_{\text{KL}}(\pi_\theta ,|, \pi_{\text{ref}})$ discourages the fine-tuned policy from diverging too far from the original supervised model, ensuring stability and preventing reward hacking.

In short, the policy seeks to balance improving alignment (higher reward) with conservatism (not forgetting what was learned during supervised fine-tuning).

For multistep, interaction with the model, s and a will be pompts (sequence of tokens). Even though the reward arrives only at the end, gradients propagate tokenwise:

$$ \nabla_\theta J(\theta) = \mathbb{E}_{x,y \sim \pi_\theta}\left[ r(x,y) \sum_{t=1}^T \nabla_\theta \log \pi_\theta(y_t \mid x,y_{<t}) \right]. $$

This is mathematically equivalent to sequence-level REINFORCE but is implemented per-token in practice to support PPO clipping and per-token advantages.

PPO Optimization

Given samples from $\pi_{\text{old}}$, define

$$ \rho_t(\theta) = \frac{\pi_\theta(y_t \mid s_t)}{\pi_{\text{old}}(y_t \mid s_t)}. $$

The clipped PPO objective is

$$ L_{\text{PPO}}(\theta) = \mathbb{E}_t \left[\min!\left( \rho_t(\theta),\hat A_t,;\text{clip}(\rho_t(\theta), 1-\epsilon, 1+\epsilon),\hat A_t \right)\right]. $$

Here $\hat A_t$ is an advantage estimate. In practice, with outcome supervision, the same reward $R$ is broadcast to all tokens, possibly normalized with a batch baseline.

KL Regularization

A KL penalty anchors the fine-tuned policy to a reference model $\pi_0$ (usually the SFT model):

$$ \widehat{\mathrm{KL}} = \mathbb{E}_t\left[ \log \pi_\theta(y_t \mid s_t) - \log \pi_0(y_t \mid s_t) \right]. $$

Final loss:

$$ \mathcal{L}(\theta) = - L_{\text{PPO}}(\theta) + \beta ,\widehat{\mathrm{KL}}. $$

Clarifications

• One-step vs multi-step: Although the environment is one-step, tokenwise updates are necessary because PPO requires per-token log-prob ratios.
• Advantage vs baseline: Advantage $\hat A_t = Q(s_t,a_t)-V(s_t)$ emerges as REINFORCE with a value-function baseline; subtracting $V(s)$ reduces variance.
• KL meaning: Both PPO and RLHF use KL on action distributions, not on parameters.

Challenges

• Implementation complecity
• resource requirement as it involves many models like reward model, value function etc
• stability of training

---

Direct Preference Optimization (DPO)

There is a closed form relationship between reward model and the optimal policy.

From Preferences to Policy

Instead of learning a reward model, DPO directly optimizes the policy from preference pairs $(x,y_w,y_l)$ with $y_w \succ y_l$.

The Bradley–Terry model gives:

$$ P(y_w \succ y_l \mid x) = \sigma!\Big(\tfrac{1}{\beta}(r(x,y_w)-r(x,y_l))\Big). $$

Reward–Policy Connection

From maximum entropy RL, the optimal policy satisfies

$$ \pi^*(y \mid x) \propto \pi_0(y \mid x) \exp!\Big(\tfrac{1}{\beta} r(x,y)\Big). $$

Rearranging:

$$ r(x,y) = \beta \Big(\log \pi^*(y \mid x) - \log \pi_0(y \mid x)\Big) + \text{const}. $$

Substituting into Bradley–Terry yields the DPO loss:

$$ \mathcal{L}_{\text{DPO}}(\theta) = - \mathbb{E}_{(x,y_w,y_l)} \log \sigma!\Big(\beta\big[\log \tfrac{\pi_\theta(y_w \mid x)}{\pi_0(y_w \mid x)}- \log \tfrac{\pi_\theta(y_l \mid x)}{\pi_0(y_l \mid x)} \big]\Big). $$

Clarifications

• Not RL in practice: DPO is implemented as supervised logistic regression on preference pairs. No reward model or critic is trained.
• Why log-ratio? The log-ratio $\log \pi_\theta - \log \pi_0$ measures preference-aligned deviation from the reference; this keeps optimization stable.
• Equivalence: DPO can be derived as a direct surrogate of PPO-RLHF under the reward–policy duality.

Gradient of the DPO Loss

The gradient of the DPO loss with respect to the policy parameters $\theta$ is:

$$ \nabla_\theta \mathcal{L}_{\text{DPO}}(\pi_\theta; \pi_{\text{ref}}) = - \beta \mathbb{E}_{(x,y_w,y_l)\sim \mathcal{D}} \Bigg[\sigma!\big(\hat r_\theta(x,y_l) - \hat r_\theta(x,y_w)\big) \Big( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \Big)\Bigg]. $$

where

$$ \hat r_\theta(x,y) = \beta \log \frac{\pi_\theta(y \mid x)}{\pi_{\text{ref}}(y \mid x)}. $$

• $-\beta$ factor: Scales the update; effectively decreases learning rate as the KL-constraint is relaxed.
• Sigmoid weight $\sigma(\hat r_\theta(x,y_l) - \hat r_\theta(x,y_w))$: Acts as a per-example weighting. Higher weight is applied when the model currently ranks the completions incorrectly.
• $\nabla_\theta \log \pi_\theta(y_w \mid x)$: Increases the likelihood of the preferred completion.
• $-\nabla_\theta \log \pi_\theta(y_l \mid x)$: Decreases the likelihood of the dispreferred completion.

Summary:
DPO gradients push the policy to assign higher probability to preferred completions and lower probability to dispreferred ones, with weighting that is strongest when the current policy deviates from the reference-preference alignment. The actual algorithm has no rollouts, no critic, no REINFORCE estimator so its similar to supervised learning, but the derived formulation is based on entropy-RL.

When Does DPO Fail? — A Challenge

Example

• Input (x): A long Reddit post explaining a situation.
• Preferred output ($y_w$):
  "TL;DR: My gf hates I have my own chef and maid. She is worried about us moving together but thinks I should just make my own things."
• Rejected output ($y_l$):
  "TL;DR: I'm chef / maid and my gf doesn't like it."
• Model samples ($y \sim p_{\pi_{\text{ref}}}$):
  "My gf hates me. My own chef and maid bother her. She asks me to cook things. I have a chef and maid. My gf doesn’t like that fact and hates me."

Gradient of DPO Loss

The DPO update is based on the difference of log-prob gradients:

$$ \nabla_\theta \mathcal{L}_{\text{DPO}} = -\beta \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} ;\sigma!\left(\hat r_\theta(x,y_l) - \hat r_\theta(x,y_w)\right) \Big( \nabla_\theta \log \pi_\theta(y_w \mid x) - \nabla_\theta \log \pi_\theta(y_l \mid x) \Big). $$

Expanding into per-token terms:

$$ \sum_{t=0}^{T_w} \nabla_\theta \log \pi_\theta(y_w^t \mid x,y_w^{\lt t}) ;-; \sum_{t=0}^{T_l} \nabla_\theta \log \pi_\theta(y_l^t \mid x,y_l^{\lt t}). $$

The Problem

• For the first tokens of $y_w$ and $y_l$, both start with the prefix "TL;DR:".
• Their gradients cancel out, since $\nabla_\theta \log \pi_\theta(\text{``TL;DR:"} \mid x)$ appears in both sums.
• As a result, the model does not receive a training signal to reinforce generating "TL;DR:".

Key Insight

• DPO learns differences, not absolute probabilities.
• When preferred and dispreferred completions share the same prefix, the gradient cancels on those tokens.
• Thus, important stylistic markers (like "TL;DR:") may not be learned.

Challenges (Takeaway)

• DPO can fail to teach the model to produce useful shared prefixes, because the per-token gradients cancel out for identical initial tokens.
  This highlights a limitation: DPO emphasizes preference contrast but ignores tokens where outputs agree.
• It needs preference data (preference pairs) and can't perform updates with online sampled data.

---

Group Relative Policy Optimization (GRPO)

Motivation

GRPO seeks to retain PPO-style RL optimization without requiring a critic (but may still require a reward model). It replaces the value-function baseline with group-relative normalization. It can train model online from sampled roll-outs.

Group Relative Advantage

For a prompt $x$, generate $G$ completions with rewards ${r_i}$. Normalize within the group:

$$ \tilde r_i = \frac{r_i - \mu}{\sigma}, \quad \mu = \tfrac{1}{G}\sum_{j=1}^G r_j, \quad \sigma = \sqrt{\tfrac{1}{G}\sum_{j=1}^G (r_j-\mu)^2}. $$

Then assign per-token advantages:

$$ \hat A_{i,t} = \tilde r_i \qquad \forall t \in y_i, $$

or cumulative step-level sums in process supervision.

GRPO Objective

The PPO objective with group-relative advantages is:

$$ J_{\text{GRPO}}(\theta) = \mathbb{E}\Big[\min(\rho_{i,t}\hat A_{i,t},;\text{clip}(\rho_{i,t},1-\epsilon,1+\epsilon)\hat A_{i,t})\Big] - \beta D_{\mathrm{KL}}(\pi_\theta ,|, \pi_{\text{ref}}). $$

Clarifications

• No critic: Group mean replaces $V(s)$ as a baseline; variance reduction is achieved without training a value network.
• Relation to PPO: Still an RL-style policy gradient with importance ratios and clipping, unlike DPO.
• Interpretation: Each completion is judged relative to peers; this reflects comparative feedback rather than absolute reward.

---

Summary

• RLHF (PPO): Classical RL-style update with a reward model and critic; environment is one-step but updates are tokenwise.
• DPO: Removes reward model and critic; reduces to supervised preference optimization with a log-ratio loss.
• GRPO: Retains PPO mechanics but eliminates critic via group-relative normalization of rewards.

All three methods are mathematically connected, but differ in practice: RLHF is heavy RL, DPO is supervised contrastive learning, and GRPO is a middle ground that keeps PPO structure while simplifying baseline estimation.