Formula Cheat Sheet - RL without Tears

LLM foundation

Language Modeling

\[ \mathrm{Pr}_{\theta}(\mathbf{z}) = \prod_{i=1}^{N}\mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i}), \qquad \log \mathrm{Pr}_{\theta}(\mathbf{z}) = \sum_{i=1}^{N}\log \mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i}) \]

Here, \(\mathbf{z}=z_1,\ldots,z_N\) is a token sequence and \(\mathbf{z}_{<i}=z_1,\ldots,z_{i-1}\). This left-to-right next-token view is the interface used for training, prompting, inference, and RL optimization.

LLM foundation

Pre-training Objective

\[ \hat{\theta} = \arg\max_{\theta} \sum_{\mathbf{z}\in\mathcal{S}_{\mathrm{pre}}} \sum_{i=1}^{N} \log \mathrm{Pr}_{\theta}(z_i\mid\mathbf{z}_{<i}) \]

\(\mathcal{S}_{\mathrm{pre}}\) is the pre-training corpus and \(\hat{\theta}\) denotes the optimized pretrained parameters.

Pre-training teaches general language modeling from unlabeled text. Later stages adapt this base model toward instruction following and reward-seeking behavior.

Conditioned generation

Prompting

Prompting adapts an LLM by changing the context used for generation, without updating model parameters. A prompt may contain instructions, user-provided content, output requirements, or demonstrations.

Prompt quality can strongly affect model performance. Even prompts with nearly the same intent may lead to different outputs, so prompts are often refined through trial and error or automated prompt search.

Baseline

Supervised Fine-Tuning

\[ \tilde{\theta} = \arg\max_{\hat{\theta}^{+}} \sum_{(\mathbf{x},\mathbf{y})\in\mathcal{S}} \log \mathrm{Pr}_{\hat{\theta}^{+}}(\mathbf{y}\mid\mathbf{x}) \]

\[ \log \mathrm{Pr}_{\theta}(\mathbf{y}\mid \mathbf{x}) = \sum_{t=1}^{T}\log \mathrm{Pr}_{\theta}(y_t\mid \mathbf{x},\mathbf{y}_{<t}) \]

Here, \(\mathbf{x}\) is the input sequence, \(\mathbf{y}=y_1,\ldots,y_T\) is the target output sequence, and \(y_t\) is predicted from the input and previous target tokens.

SFT maximizes the likelihood of demonstrated outputs. It provides the initial policy before preference learning or RL optimization.

Autoregressive generation

Common Decoding Rules

Greedy decoding

\[ y_t^{\mathrm{greedy}} = \arg\max_{w\in\mathcal{V}} \mathrm{Pr}_{\theta}(w\mid\mathbf{x},\mathbf{y}_{<t}) \]

Beam search

\[ s(\mathbf{y}_{\leq t}) = s(\mathbf{y}_{<t}) + \log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t}) \]

\[ \mathcal{B}_t = \operatorname{TopB} \left\{ (\mathbf{y}_{<t},w): \mathbf{y}_{<t}\in\mathcal{B}_{t-1}, w\in\mathcal{V} \right\} \]

Sampling

\[ y_t \sim \mathrm{Pr}_{\theta}(\cdot\mid\mathbf{x},\mathbf{y}_{<t}), \qquad \mathrm{Pr}_{\theta}^{(T)}(w\mid\mathbf{x},\mathbf{y}_{<t}) = \frac{\exp(l_w/T)}{\sum_{v\in\mathcal{V}}\exp(l_v/T)} \]

Greedy decoding takes the most likely token. Beam search keeps the top \(B\) candidate prefixes under the accumulated sequence score. Sampling draws from the next-token distribution, often adjusted by temperature \(T\), top-\(k\), or top-\(p\) filtering.

Sampling is especially important for RL because it mirrors stochastic policy sampling and produces trajectories that can be scored by rewards.

LLM-RL mapping

LLM as a Policy

\[ \pi_{\theta}(a_t\mid s_t) = \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t}), \qquad a_t=y_t,\quad s_t=(\mathbf{x},\mathbf{y}_{<t}) \]

Here, \(s_t\) is the prompt plus generated prefix, and \(a_t\) is the next-token action selected by the LLM.

This is the bridge between token generation and RL. The policy chooses the next token conditioned on the prompt and generated prefix.

RL foundation

MDP Policy Objective

\[ \pi^{*} = \arg\max_{\pi} \mathbb{E} \left[ \sum_{t=0}^{H-1}r_t \mid \pi \right] \]

A trajectory is \(\tau=(s_0,a_0,s_1,a_1,\ldots,s_{H-1},a_{H-1})\), and \(H\) is the trajectory length.

In LLM RL, a trajectory may be a completion, reasoning trace, tool-use interaction, or multimodal generation path.

Policy optimization

Policy Gradient

\[ \nabla J(\theta) = \mathbb{E}_{\tau\sim \pi_{\theta}} \left[ R(\tau)\nabla_{\theta}\log \pi_{\theta}(\tau) \right] \]

Here, \(J(\theta)\) is the policy objective, \(\tau\) is a sampled trajectory, and \(R(\tau)\) is its cumulative reward.

LLM form in the paper

\[ \begin{aligned} \frac{\partial J(\theta)}{\partial\theta} &= \mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}} \left[ \frac{1}{d} \sum_{\mathbf{y}\in\mathcal{D}} \frac{\partial \log\mathrm{Pr}_{\theta}(\mathbf{y}\mid\mathbf{x})}{\partial\theta} R(\mathbf{y}) \right] \\ \frac{\partial J(\theta)}{\partial\theta} &= \mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}} \left[ \frac{1}{d} \sum_{\mathbf{y}\in\mathcal{D}} \sum_{t=1}^{T} \frac{\partial \log \mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})}{\partial\theta} \sum_{k=t}^{T}r_k \right] \end{aligned} \]

Increase the probability of sampled trajectories that receive high reward. The reward does not need to be differentiable.

Expected future reward

Value and Discounted Return

\[ V(s)= \mathbb{E}\left[ \sum_{t=0}^{\infty}\gamma^t r_t \mid s_0=s,\pi \right] \]

Here, \(V(s)\) is the expected discounted return from state \(s\), \(r_t\) is the reward at timestep \(t\), and \(\gamma\) is the discount factor.

The value function estimates future return under a policy. The discount factor controls how much future rewards matter relative to near-term rewards.

Variance reduction

Value and Advantage

\[ A(s,a)=Q(s,a)-V(s) \]

Here, \(Q(s,a)\) is the expected return after taking action \(a\) at state \(s\), while \(V(s)\) is the baseline expected return from the state.

Advantage measures whether an action is better than the expected behavior at the same state. It reduces the variance of reward-weighted updates.

Bias-variance tradeoff

Generalized Advantage Estimation

\[ \delta_t=r_t+\gamma V_{t+1}-V_t,\qquad A_t^{\mathrm{GAE}}=\delta_t+\gamma\lambda A_{t+1}^{\mathrm{GAE}} \]

Here, \(\delta_t\) is the temporal-difference residual, \(V_t\) is the value estimate at timestep \(t\), and \(\lambda\) controls the GAE bias-variance tradeoff.

GAE recursively mixes temporal-difference signals across future steps. It is a practical compromise between noisy Monte Carlo returns and biased one-step estimates.

Off-policy correction

Importance Sampling Ratio

\[ \rho_t(\theta)= \frac{\pi_{\theta}(a_t\mid s_t)} {\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)} \]

Here, \(\pi_{\theta}\) is the current policy and \(\pi_{\theta_{\mathrm{old}}}\) is the old policy that generated the sampled action \(a_t\).

The ratio corrects the mismatch between the policy used to generate samples and the policy currently being optimized.

Policy drift control

KL Penalty

\[ \mathrm{Penalty}_t = \log \mathrm{Pr}_{\theta}(y_t\mid \mathbf{x},\mathbf{y}_{<t}) - \log \mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t\mid \mathbf{x},\mathbf{y}_{<t}) \]

Here, \(\theta_{\mathrm{ref}}\) denotes the reference model parameters, and the two log probabilities compare the trained policy with the reference policy on the same token.

The penalty discourages the trained policy from moving too far away from the reference model. In RLHF, this is central for preserving language quality while optimizing reward. In PPO-style objectives, \(\beta\) is the weight controlling the strength of this penalty.

Stable RLHF

PPO Clipped Objective

\[ \min\left( \rho_t A_t, \mathrm{clip}(\rho_t,1-\epsilon,1+\epsilon)A_t \right) \]

Here, \(A_t\) is the advantage at timestep \(t\), \(\rho_t\) is the policy probability ratio, and \(\epsilon\) sets the clipping range.

PPO limits policy updates by clipping the probability ratio. This helps prevent reward over-optimization and unstable policy drift.

\[ \mathcal{L}(\theta) = - \mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}} \mathbb{E}_{\mathbf{y}\sim\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\cdot\mid\mathbf{x})} \left[ \sum_{t=1}^{T} \left( \mathrm{Clip}\left( \frac{\mathrm{Pr}_{\theta}(y_t\mid\mathbf{x},\mathbf{y}_{<t})} {\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_t\mid\mathbf{x},\mathbf{y}_{<t})} A_t \right) - \beta \mathrm{Penalty}_t \right) \right] \]

Preference reward

Pairwise Reward Model Loss

\[ \mathcal{L}_{\mathrm{RM}} = -\log \sigma\left(r_{\phi}(\mathbf{x},\mathbf{y}_w)-r_{\phi}(\mathbf{x},\mathbf{y}_l)\right) \]

Here, \(r_{\phi}\) is the reward model, \(\mathbf{x}\) is the input, \(\mathbf{y}_w\) is the preferred output sequence, and \(\mathbf{y}_l\) is the rejected output sequence.

The reward model learns to assign a higher score to the preferred response than to the rejected response.

Denser feedback

Reward Shaping

\[ r'_t=r_t+f(\mathbf{x},\mathbf{y}_{<t},y_t),\qquad f=\gamma\Phi(\mathbf{x},\mathbf{y}_{<t+1},y_{t+1})-\Phi(\mathbf{x},\mathbf{y}_{<t},y_t) \]

Here, \(r'_t\) is the shaped reward, \(r_t\) is the original reward, and \(\Phi\) is the potential function used to construct the additional feedback term.

Shaping adds intermediate reward signals. Potential-based shaping is useful because it can provide denser feedback while preserving the intended optimal policy.

Group-relative RL

GRPO Advantage

\[ A^{\mathrm{GRPO}}_{i,t} = \frac{R(\mathbf{x},\mathbf{y}_i)-\mathrm{Mean}(\mathbf{R})} {\mathrm{Std}(\mathbf{R})} \]

Here, \(\mathbf{y}_i\) is one response in a group sampled for the same input \(\mathbf{x}\), and \(\mathbf{R}\) is the set of rewards for that group.

GRPO estimates advantage by comparing responses sampled for the same prompt. This can remove the need for a separate value model.

\[ \mathcal{L}_{g}(\theta) = - \mathbb{E}_{\mathbf{x}\sim\mathcal{S}_{x}} \frac{1}{G} \sum_{i=1}^{G} \left[ \sum_{t=1}^{T} \mathrm{Clip}\left( \frac{\mathrm{Pr}_{\theta}(y_{i,t}\mid\mathbf{x},\mathbf{y}_{i,<t})} {\mathrm{Pr}_{\theta_{\mathrm{ref}}}(y_{i,t}\mid\mathbf{x},\mathbf{y}_{i,<t})} A_{i,t}^{\mathrm{GRPO}} \right) - \beta \mathrm{Penalty} \right] \]

Direct preference learning

DPO Preference Objective

\[ \mathcal{L}_{\mathrm{DPO}} = -\log \sigma\left( \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_w\mid \mathbf{x})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\mathbf{y}_w\mid \mathbf{x})} - \beta \log \frac{\mathrm{Pr}_{\theta}(\mathbf{y}_l\mid \mathbf{x})}{\mathrm{Pr}_{\theta_{\mathrm{ref}}}(\mathbf{y}_l\mid \mathbf{x})} \right) \]

Here, \(\mathbf{y}_w\) and \(\mathbf{y}_l\) denote the preferred and dispreferred output sequences, \(\theta_{\mathrm{ref}}\) is the reference model, and \(\beta\) controls the strength of the implicit reward ratio.

DPO optimizes the policy directly from preference pairs, avoiding an explicit pipeline that trains a reward model and then applies RL in its basic form.

Reasoning search

UCT Selection in MCTS

\[ \bar{\mathbf{y}}_k^* = \arg\max_{\bar{\mathbf{y}}_k} \left( \bar{R}(\bar{\mathbf{y}}_k) + c\sqrt{\frac{\ln N_p}{N_{\bar{\mathbf{y}}_k}}} \right) \]

Here, \(\bar{\mathbf{y}}_k\) is a candidate reasoning step, \(N_p\) is the parent-node visit count, \(N_{\bar{\mathbf{y}}_k}\) is the child-node visit count, and \(c\) balances exploitation and exploration.

MCTS balances exploiting promising reasoning steps and exploring under-visited alternatives. This is a useful mental model for verifier-guided reasoning.

Interactive agents

Agent Trajectory Objective

\[ \tau=[p,u_1,o_1,\ldots,u_T,o_T],\qquad \theta^*=\arg\max_{\theta} \mathbb{E}_{\tau\sim\pi_{\theta}(\cdot\mid e,q)} \left[R(\tau)\right] \]

Here, \(e\) is environment information, \(q\) is the user task, \(p\) is the generated plan, and \(u_t,o_t\) are the agent behavior and environment observation at step \(t\).

Agentic RL optimizes plans and action-observation trajectories rather than only single completions. This makes delayed rewards and credit assignment especially important.

Beyond text

Multimodal Generation RL Objectives

\[ \max_{\theta} \mathbb{E}_{\mathbf{z}\sim p(\mathbf{z}),\,\mathbf{x}_0\sim p_{\theta}(\mathbf{x}_0\mid \mathbf{z})} \left[ R_{\mathrm{dm}}(\mathbf{x}_0,\mathbf{z}) \right] \]

Here, \(\mathbf{z}\) denotes the text prompt, \(\mathbf{x}_0\) is the generated sample, and \(R_{\mathrm{dm}}\) is the reward function used to evaluate that sample.

For image generation, the final sample can be scored by alignment, aesthetics, safety, or preference rewards, while the denoising process supplies the trajectory to optimize.

\[ \mathcal{L}_{\mathrm{dmrl}}(\theta) = - \mathbb{E}_{\mathbf{z}\sim S_z,\,\tau_{\mathrm{dm}}\sim \mathrm{Pr}_{\theta}(\cdot)} \left[ \sum_{t=0}^{T-1} \log U_{\theta}(\mathbf{x}_{T-t-1}\mid\mathbf{x}_{T-t},\mathbf{z}) R_{\mathrm{dm}}(\mathbf{x}_0,\mathbf{z}) \right] \]

This follows the DPOK-style denoising-trajectory formulation used in the paper; later diffusion RL variants may modify the regularization, reference model, or advantage estimator.

\[ \mathcal{L}_{\mathrm{fmgrpo}}(\theta) = - \mathbb{E}_{\mathbf{z}\sim S_z,\,\tau_{\mathrm{fm}}\sim \mathrm{Pr}(\cdot)} \frac{1}{G} \sum_{i=1}^{G} \left[ \frac{1}{T} \sum_t \min \left( \frac{\mathrm{Pr}_{\theta}(a_t^i\mid s_t^i)} {\mathrm{Pr}_{\theta_{\mathrm{old}}}(a_t^i\mid s_t^i)} \hat{A}_i, \mathrm{clip} \left( \frac{\mathrm{Pr}_{\theta}(a_t^i\mid s_t^i)} {\mathrm{Pr}_{\theta_{\mathrm{old}}}(a_t^i\mid s_t^i)}, 1-\epsilon, 1+\epsilon \right) \hat{A}_i \right) \right] \]

This follows Flow-GRPO, where ODE-based flow matching is converted into a stochastic MDP for online RL; later flow-matching RL variants may alter the stochastic conversion or the group-relative update.

RL Formula Cheat Sheet