Companion material

Glossary

This glossary follows the same progression as the tutorial. It opens with a shared abbreviation and notation guide, then moves through LLM foundations, core RL terms, RLHF, Reward Modeling, Reasoning RL, Agentic RL, and Multimodal RL. The emphasis is on how each RL concept should be interpreted when the policy is an LLM or a related foundation model.

Selected entries include references to representative papers or standard background resources. These references are intended as targeted entry points rather than an exhaustive bibliography, and should be read as supplementary material for the main tutorial.

Abbreviations and Symbols

RL: Reinforcement learning.
LLM / VLM: Large language model / vision-language model.
SFT: Supervised Fine-Tuning.
RLHF / RLAIF: Reinforcement Learning from Human Feedback / Reinforcement Learning from AI Feedback.
RM / PRM / ORM / VRM: Reward Model / Process Reward Model / Outcome Reward Model / Visual Reward Model.
MDP: Markov Decision Process.
PPO / GRPO / DPO: Proximal Policy Optimization / Group Relative Policy Optimization / Direct Preference Optimization.
TD / GAE: Temporal Difference / Generalized Advantage Estimation.
KL: Kullback-Leibler divergence or the KL-based regularization term used to keep a policy close to a reference policy.
MCTS / UCT / OPD: Monte Carlo Tree Search / Upper Confidence Bounds applied to Trees / On-Policy Distillation.
\(\mathbf{x}\), \(\mathbf{y}\), and \(y_t\): \(\mathbf{x}\) denotes an input or prompt sequence; \(\mathbf{y}\) denotes a generated output sequence; \(y_t\) denotes the token generated at step \(t\).
\(\mathcal{V}\): The vocabulary, or finite set of tokens the model can generate.
\(\mathbf{y}_w\) and \(\mathbf{y}_l\): The preferred and rejected output sequences in pairwise preference data. The subscript \(w\) means winner, and \(l\) means loser.
\(s_t\), \(a_t\), and \(\tau\): \(s_t\) is the state at step \(t\), \(a_t\) is the action, and \(\tau\) is the full trajectory.
\(\mathrm{Pr}_{\theta}\) and \(\pi_{\theta}\): \(\mathrm{Pr}_{\theta}\) is used for LLM token or sequence probability. \(\pi_{\theta}\) is used when the same model is viewed as an RL policy.
\(\theta_{\mathrm{old}}\) and \(\theta_{\mathrm{ref}}\): \(\theta_{\mathrm{old}}\) denotes the policy parameters that generated sampled data. \(\theta_{\mathrm{ref}}\) denotes the fixed reference model used for KL regularization.
\(r_t\), \(R\), \(V\), \(Q\), and \(A\): \(r_t\) is step reward, \(R\) is trajectory-level return or reward, \(V\) is state value, \(Q\) is action value, and \(A\) is advantage.

LLM Foundations

Token

The basic unit processed by an LLM. A token may be a word, subword, punctuation mark, whitespace fragment, or another piece produced by the tokenizer. In the tutorial notation, inputs and outputs are represented as token sequences such as z = z₁...z_N.

Vocabulary

The finite set of tokens an LLM can read and generate, usually written as \(\mathcal{V}\). At each generation step, the model outputs a probability distribution over this vocabulary.

Decoder-only Transformer

A Transformer language model that generates text from left to right using causal self-attention. The prediction at each position can attend only to previous tokens, which matches the autoregressive factorization used throughout the tutorial.

[1] Improving Language Understanding by Generative Pre-Training OpenAI 2018 paper, Alec Radford, Karthik Narasimhan et al.

Autoregressive Generation

The process of generating an output sequence one token at a time, appending each selected token to the context before predicting the next. Given an input sequence x, the output probability is factorized as Pr_θ(y|x)=∏_t=1^TPr_θ(y_t|x,y_<t). This is the foundation for mapping LLM generation to sequential decision-making.

[1] Attention Is All You Need NeurIPS 2017 paper, Ashish Vaswani, Noam Shazeer et al.

Pre-training

The first stage of building most LLMs, where the model learns general language capabilities from large-scale text using next-token prediction. Pre-training gives the model broad linguistic and factual knowledge, but it does not by itself teach the model to follow a user's instruction or produce a task-specific response format.

[1] Improving Language Understanding by Generative Pre-Training OpenAI 2018 paper, Alec Radford, Karthik Narasimhan et al.

[2] Language Models are Unsupervised Multitask Learners OpenAI 2019 paper, Alec Radford, Jeffrey Wu et al.

Prompt

The input text given to an LLM to guide its output. A prompt may contain an instruction, user-provided content, output requirements, demonstrations, or formatting constraints. In the tutorial, the prompt sequence is written as x=x₁...x_m, and the model generates an output sequence y conditioned on it.

Prompt Template

A reusable prompt pattern with placeholders that are filled before the prompt is sent to the model. Templates make it possible to keep the task framing fixed while varying the concrete input content.

In-context Learning

A prompting technique where examples are placed in the context so the model can infer the desired input-output pattern without parameter updates. In-context learning changes the conditioning context, not the model weights.

Supervised Fine-Tuning (SFT)

A post-training stage that adapts a pretrained LLM using labeled input-output pairs. The loss is usually computed on output tokens while the input tokens provide context. SFT teaches the model to follow instructions and often serves as the initial policy for RL training.

Inference

The process of applying a trained LLM to generate outputs for new inputs. At each step, the model predicts a next-token distribution, a decoding strategy selects a token, and generation continues until an end condition is reached.

Decoding Strategy

The rule used to select the next token from the model's predicted distribution. Greedy decoding selects the highest-probability token; beam search keeps multiple candidate prefixes; sampling draws from the distribution, often with temperature, top-k, or top-p filtering. Sampling is especially close to policy sampling in RL.

[1] Hierarchical Neural Story Generation ACL 2018 paper, Angela Fan, Mike Lewis, Yann Dauphin.

[2] The Curious Case of Neural Text Degeneration ICLR 2020 paper, Ari Holtzman, Jan Buys et al.

RL Core

Markov Decision Process (MDP): A mathematical framework for sequential decision-making. An agent observes a state, chooses an action, receives reward, and transitions to a new state. In LLM RL, the state can be the prompt plus generated prefix, the action can be the next token, and the trajectory can be the generated output or a longer interaction.
Agent: The learner or decision-maker in RL. In the context of LLMs, the agent is usually the language model itself. It generates tokens, answers, tool calls, or interaction steps, and its behavior is optimized according to feedback associated with these decisions.
Environment: The external process that receives actions and returns observations or rewards. For LLMs, the environment may be a reward model, verifier, simulator, tool system, code runner, browser, or multi-turn task environment. It specifies the interaction structure and the feedback available to the agent.
State: The information available when a decision is made. In standard RL, a state summarizes the relevant situation for action selection. For LLMs, it is often the input prompt together with previously generated tokens, tool observations, and conversation history.
Action: The decision made by the policy at a given state. In standard LLM generation, the action is naturally the next token. In agentic settings, it may be a tool call, an argument string, a search query, a code edit, or a higher-level interaction step.
Policy: A distribution over actions conditioned on the current state. Written compactly, a policy is often denoted as π(a | s), the probability of choosing action a in state s. An LLM policy assigns probabilities to next tokens or, in agent settings, to higher-level actions such as tool use.
Trajectory: A sequence of states, actions, and often rewards generated under a policy. A common notation is τ = (s₀, a₀, r₁, s₁, ...). In LLMs, a trajectory may be a completion, a reasoning trace, a denoising path, or a multi-turn agent interaction.
Return: The cumulative reward collected along a trajectory. It is the quantity the policy is ultimately trained to maximize. A standard discounted return is G_t = r_t+1 + γr_t+2 + γ²r_t+3 + ....
Discount Factor: A coefficient, usually written γ, that controls how much future rewards contribute to return. Smaller values emphasize near-term rewards, while larger values give more weight to delayed outcomes. In many LLM tasks with short completions, discounting is less central than in robotics or games, but it remains relevant whenever early decisions affect later quality.
[1] Markov Decision Process Wiki wiki
Value Function: An estimate of expected future return from a state or state-action pair. The state-value function is often written V(s), while the action-value function is written Q(s, a). In LLM RL, a value model is often used as a baseline for advantage estimation, helping the optimizer distinguish genuinely high-quality sampled behavior from behavior that benefits from prompt-specific difficulty variation.
[1] Reinforcement Learning: An Introduction MIT Press 2018 book, Richard S. Sutton, Andrew G. Barto.
Critic: A model that estimates values or advantages for policy optimization. In actor-critic language, the policy is the actor because it chooses actions, while the critic comments on whether those choices look better or worse than expected. The critic is not necessarily judging the final answer directly; it is often estimating a learning signal that makes policy updates less noisy. In PPO-style LLM RL, it is often implemented as a value head attached to a language-model backbone and trained alongside the policy.
[1] Actor-Critic Algorithms NeurIPS 1999 paper, Vijay R. Konda, John N. Tsitsiklis.
Advantage: The difference between the value of an action and the expected value of the state, often written A(s, a) = Q(s, a) - V(s). It indicates whether a sampled token or action is better or worse than the baseline behavior expected at that state, and is therefore central to variance reduction in policy-gradient methods.
[1] Policy Gradient Methods for Reinforcement Learning with Function Approximation NeurIPS 1999 paper, Richard S. Sutton, David McAllester et al.
Monte Carlo Estimate: An estimate computed from sampled complete trajectories. It uses the observed return directly after the trajectory has finished. Monte Carlo estimates can reduce bias, but often have high variance, especially for long LLM outputs or multi-step agent interactions.
Temporal Difference: An estimate that updates value or advantage using the next predicted value before the full trajectory is complete. A basic temporal-difference error can be written as δ_t = r_t + γV(s_t+1) - V(s_t). Temporal-difference methods reduce variance by bootstrapping from a value model, but may introduce bias when the value estimate is inaccurate.
Generalized Advantage Estimation (GAE): Generalized Advantage Estimation. A recursive estimator that mixes temporal-difference residuals across future steps, controlled by discount and smoothing parameters to balance bias and variance. Its key smoothing parameter, often written λ, interpolates between short-horizon bootstrapping and long-horizon Monte Carlo estimation.
[1] High-Dimensional Continuous Control Using Generalized Advantage Estimation ICLR 2016 paper, John Schulman, Philipp Moritz et al.

LLM RL

RLHF

Reinforcement Learning from Human Feedback. Human preference data is commonly used to train a reward model, which then provides feedback for optimizing the policy.

[1] Deep Reinforcement Learning from Human Preferences NeurIPS 2017 paper, Paul F. Christiano, Jan Leike et al.

[2] Learning to Summarize from Human Feedback NeurIPS 2020 paper, Nisan Stiennon, Long Ouyang et al.

[3] Training Language Models to Follow Instructions with Human Feedback NeurIPS 2022 paper, Long Ouyang, Jeffrey Wu et al.

RLAIF

Reinforcement Learning from AI Feedback. Instead of relying only on human annotations, AI systems provide preference labels, critiques, scores, or rewards to scale feedback collection.

[1] RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback arXiv 2023 paper, Harrison Lee, Samrat Phatale et al.

[2] Constitutional AI: Harmlessness from AI Feedback arXiv 2022 paper, Yuntao Bai, Saurav Kadavath et al.

Reference Policy

A fixed or slowly updated policy used to regularize the trained model. In LLM RL, this is often the supervised fine-tuned model before RL begins. The reference policy constrains optimization so that reward improvement does not substantially degrade language quality, style, or instruction-following behavior.

Kullback-Leibler Penalty (KL Penalty)

A penalty based on the distance between the trained policy and the reference policy. In RLHF, it helps preserve language quality while the model optimizes reward. A compact objective is often described as maximizing reward while subtracting a term such as β KL(π || π_ref), where β controls the strength of regularization.

[1] Training Language Models to Follow Instructions with Human Feedback NeurIPS 2022 paper, Long Ouyang, Jeffrey Wu et al.

Policy Drift

The movement of the trained policy away from the reference or data-generating policy. Some drift is necessary for optimization, but excessive drift can produce behavior that receives high reward model scores while degrading the intended quality of the model output.

Rollout

A sampled output or interaction trajectory generated by a policy. In LLM RL, rollouts provide the behaviors that are scored by rewards and then used for policy updates. For a chat model, a rollout may be one answer; for a reasoning model, it may include a chain of intermediate steps; for an agent, it may include tool calls and observations.

On-policy

Training with samples generated by the current policy or a very recent version of it. This reduces mismatch between the data distribution and the policy being optimized, which is important when policy updates substantially change the model's behavior.

Off-policy

Training with samples generated by a different policy. This can reuse old data efficiently, but the samples may no longer match the current policy. Methods such as importance sampling are often used to correct for this distribution mismatch.

Proximal Policy Optimization (PPO)

Proximal Policy Optimization. A policy-gradient method that stabilizes updates by limiting how much the new policy can change relative to the policy that produced the data. PPO commonly uses a clipped probability ratio that compares π_new(a | s) with π_old(a | s), thereby preventing overly large policy updates. This made PPO a common choice for PPO-style RLHF pipelines.

[1] Proximal Policy Optimization Algorithms arXiv 2017 paper, John Schulman, Filip Wolski et al.

Group Relative Policy Optimization (GRPO)

Group Relative Policy Optimization. A method that estimates relative advantages among multiple responses sampled for the same prompt, avoiding the need for a separate value model. GRPO evaluates each response relative to other responses in the same prompt group, which is especially natural for reasoning tasks where several sampled solutions can be checked or scored together.

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models arXiv 2024 paper, Zhihong Shao, Peiyi Wang et al.

Group-relative Advantage

An advantage estimate computed by comparing each response with other responses sampled for the same prompt. The group reward distribution acts as a prompt-specific baseline. A common implementation normalizes each reward by the group's mean and standard deviation, so a response receives positive advantage when it outperforms other samples in the group and negative advantage when it underperforms.

[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models arXiv 2024 paper, Zhihong Shao, Peiyi Wang et al.

Direct Preference Optimization (DPO)

Direct Preference Optimization. A preference learning method that trains the policy directly from chosen and rejected responses, without explicitly training a separate reward model in the basic pipeline. The objective increases the likelihood of preferred responses relative to rejected responses while regularizing the policy toward a reference model. Compared with PPO-style RLHF, DPO can be viewed as supervised preference learning with an underlying RL interpretation.

[1] Direct Preference Optimization: Your Language Model is Secretly a Reward Model NeurIPS 2023 paper, Rafael Rafailov, Archit Sharma et al.

Implicit Reward

The log-probability-ratio reward induced by direct preference methods such as DPO. It replaces an explicit learned reward model with a reward implied by policy and reference model likelihoods.

[1] A General Theoretical Paradigm to Understand Learning from Human Preferences AISTATS 2024 paper, Mohammad Gheshlaghi Azar, Mark Rowland et al.

Rewards

Reward Model

A model that maps an input-output pair to a scalar score. It can be used to rank responses, select the best candidate, or provide reward signals for policy optimization.

[1] Learning to Summarize from Human Feedback NeurIPS 2020 paper, Nisan Stiennon, Long Ouyang et al.

[2] Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback arXiv 2022 paper, Yuntao Bai, Andy Jones et al.

Preference Pair

A pair of candidate responses where one is labeled preferred over the other. Pairwise preference data is often more reliable to collect than absolute numerical scores.

Outcome Reward

A reward assigned to the final output or final task result. It is useful when correctness can be verified at the end, but can be sparse for long reasoning chains.

Process Reward

A reward assigned to intermediate reasoning steps or subgoals. It provides denser feedback than an outcome reward and can help with credit assignment in long trajectories.

[1] Let's Verify Step by Step ICLR 2024 paper, Hunter Lightman, Vineet Kosaraju et al.

Rule-based Reward

A reward computed by deterministic checks such as exact answer matching, unit tests, formatting rules, or program execution. These rewards can be especially stable when the desired behavior is verifiable.

Rubric Reward

A reward produced by evaluating responses against explicit criteria such as correctness, clarity, safety, or completeness. Rubrics make the evaluation dimensions more transparent than a single opaque preference label.

Generative Reward Model

A reward model that frames judgment as generation, such as predicting a preference label or explanation token with an LLM. This lets the reward model reuse instruction-following and reasoning abilities.

[1] Generative Reward Models arXiv 2024 paper, Dakota Mahan, Duy Van Phung et al.

[2] GRAM-R²: Self-Training Generative Foundation Reward Models for Reward Reasoning arXiv 2025 paper, Chenglong Wang, Yongyu Mu et al.

Reward Generalization

The ability of a reward model or reward rule to remain reliable when the policy generates outputs outside the original training distribution. It matters because policy optimization can move into new behavior regions during training.

[1] Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs NeurIPS 2024 paper, Rui Yang, Ruomeng Ding et al.

[2] GRAM: A Generative Foundation Reward Model for Reward Generalization ICML 2025 paper, Chenglong Wang, Yang Gan et al.

Reward Shaping

The practice of adding intermediate or auxiliary rewards to provide denser learning signals. In LLM reasoning or agent tasks, shaping can provide feedback before the final answer or task outcome is known.

[1] Policy Invariance under Reward Transformations: Theory and Application to Reward Shaping ICML 1999 paper, Andrew Y. Ng, Daishi Harada, Stuart Russell.

Potential-based Shaping

A form of reward shaping that adds differences of a potential function. Its appeal is that it can provide intermediate feedback while preserving the intended optimal policy under suitable conditions.

Reward Hacking

A failure mode where the policy exploits flaws in the reward instead of improving the intended behavior. It is especially concerning when the reward model can be optimized in ways that diverge from the true task objective.

[1] Reward Hacking Wiki wiki

Overoptimization

A failure mode where continued optimization improves reward model score but degrades the true intended quality. In preference learning, this can happen when the policy learns artifacts of the reward model or dataset.

[1] Scaling Laws for Reward Model Overoptimization ICML 2023 paper, Leo Gao, John Schulman, Jacob Hilton.

Reasoning

Verifier

A system that checks whether an answer, program, proof, or intermediate step is correct. In Reasoning RL, verifiers provide more objective feedback than open-ended preference judgments when correctness can be checked.

Best-of-N

A test-time method that samples N candidate outputs and selects the best according to a reward model or verifier. It can also motivate rejection sampling, where selected outputs are reused for training.

Test-time Scaling

Improving inference by spending more computation on sampling, search, verification, or selection rather than changing model weights. In reasoning tasks, this can help uncover stronger reasoning paths at inference time.

Process Reward Model

A reward model that evaluates intermediate steps rather than only the final answer. It is useful when final-answer rewards are too sparse to show where a reasoning path begins to fail.

[1] Let's Verify Step by Step ICLR 2024 paper, Hunter Lightman, Vineet Kosaraju et al.

[2] Solving Math Word Problems with Process- and Outcome-Based Feedback arXiv 2022 paper, Jonathan Uesato, Nate Kushman et al.

Outcome Reward Model

A model or rule that evaluates only the final answer or final task outcome. It is often cheaper to define, but may provide limited guidance for long reasoning or agent trajectories.

[1] Solving Math Word Problems with Process- and Outcome-Based Feedback arXiv 2022 paper, Jonathan Uesato, Nate Kushman et al.

Monte Carlo Tree Search (MCTS)

Monte Carlo Tree Search. A search method that explores possible reasoning paths through selection, expansion, simulation, and backpropagation, using verifier or reward feedback to guide exploration.

[1] Monte Carlo Tree Search Wiki wiki

Upper Confidence Bounds Applied to Trees (UCT)

Upper Confidence bounds applied to Trees. A selection rule in MCTS that balances exploiting high-reward paths with exploring under-visited reasoning steps.

On-policy Distillation

A method where the student samples from its current policy and receives teacher distributional supervision at the states it actually visits. This reduces mismatch compared with offline distillation from teacher-generated trajectories.

[1] On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes ICLR 2024 paper, Rishabh Agarwal, Nino Vieillard et al.

Agents

Agentic RL: RL for models that learn through multi-step interaction with tools, environments, memory, and observations. The focus shifts from optimizing a single response to optimizing long-horizon decisions and task outcomes.
Tool Use: The ability to call external systems such as search, APIs, code interpreters, or databases. In Agentic RL, the model can be rewarded for using tools correctly, efficiently, and only when they help the task.
[1] Toolformer: Language Models Can Teach Themselves to Use Tools NeurIPS 2023 paper, Timo Schick, Jane Dwivedi-Yu et al.
Credit Assignment: The problem of deciding which earlier decisions caused a delayed success or failure. It becomes harder in agent tasks because planning choices, tool calls, and observations may affect the final outcome many steps later.
Memory Management: The process of storing, updating, deleting, and retrieving useful information from past interactions. RL can train memory operations by rewarding future task success after a memory is used.
[1] Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learning ACL 2026 paper, Sikuan Yan, Xiufeng Yang et al.
Skill Optimization: The process of converting repeated successful behavior into reusable skills for future tasks. Skills can serve as high-level behavioral priors that guide later planning and RL exploration.
[1] SkillRL: Evolving Agents via Recursive Skill-Augmented Reinforcement Learning arXiv 2026 paper, Peng Xia, Jianwen Chen et al.
Environment Synthesis: The construction or generation of interactive tasks that provide observations, actions, and rewards for agent training. Good environments expose weaknesses in the current policy and create useful interaction data.
Trajectory-level Reward: A reward assigned to the whole interaction trajectory, often based on final task success. It is natural for agent tasks, but can be too coarse to explain which intermediate decision mattered.
Step-level Reward: A reward assigned to an intermediate plan, tool call, observation response, or subgoal. It supports finer credit assignment when the final trajectory contains many decisions.
State-based Reward: A reward computed by checking the final environment state against a task goal, such as whether a database or file system changed correctly. This can be more reliable than judging only the text of the final answer.
Tool Cost: A penalty for unnecessary or expensive tool use, such as latency, API cost, extra calls, or computation. It encourages agents to balance tool benefit against interaction cost.

Multimodal

Visual Reward Model

A reward model that evaluates text responses or generated content with respect to visual inputs, prompts, or visual preferences. It extends RLHF-style feedback to settings where the model must understand images as well as language.

[1] Unified Reward Model for Multimodal Understanding and Generation arXiv 2025 paper, Yibin Wang, Yuhang Zang et al.

[2] RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data AAAI 2025 paper, Chenglong Wang, Yang Gan et al.

Diffusion RL

Reward optimization for diffusion generators, where the denoising process is treated as a trajectory and the final sample receives reward. This lets image generators optimize criteria such as prompt satisfaction or human preference.

[1] DPOK: Reinforcement Learning for Fine-Tuning Text-to-Image Diffusion Models NeurIPS 2023 paper, Ying Fan, Olivia Watkins et al.

Flow Matching RL

Reward optimization for flow matching models, often by discretizing the continuous generation path into decision steps. Stochastic sampling can be introduced so policy-gradient or GRPO-style methods can explore trajectories.

[1] Flow-GRPO: Training Flow Matching Models via Online RL NeurIPS 2025 paper, Jie Liu, Jiajun Wu et al.