RL without Tears

Abstract

Reinforcement learning has become one of the main languages for explaining modern LLM post-training: it turns generated behavior into trajectories, evaluates that behavior with feedback, and updates the model so better behavior becomes more likely. This tutorial introduces RL from the perspective of LLMs, using prompts, tokens, rewards, verifiers, preference data, and interaction histories as the central objects.

The paper is organized as a gradual bridge rather than a catalog of methods. It first reviews the LLM and RL notation needed to see an LLM as a policy. It then derives PPO-style RLHF from a concrete training example, explains why rewards, advantages, importance sampling, and reference models are needed, and only then moves to stronger reward modeling, efficient optimization, DPO, reasoning models, agentic systems, and multimodal generation.

Chapters

1. Introduction PDF

   Explains why LLM training increasingly needs feedback over complete behavior, not only token-level demonstrations.

2. Preliminary: LLMs and RL PDF

   Builds the shared notation: autoregressive generation on the LLM side, MDPs and trajectories on the RL side, and the mapping between them.

3. Understanding RL in LLM Training PDF

   Starts from a homework-assistant example and derives the PPO-style RLHF workflow step by step: policy gradient, credit assignment, advantages, importance sampling, PPO, and reward models.

4. Improved RL for LLMs PDF

   Revisits the basic workflow and asks what must improve in practice: reward quality, reward sparsity, reward generalization, advantage estimation, training efficiency, and direct preference optimization.

5. RL for Reasoning PDF

   Shows how verifiable reasoning turns sampling and search into training signals through Best-of-N, process verification, MCTS, iterative RL, large-scale RL, and on-policy distillation.

6. Agentic RL PDF

   Extends the same feedback loop to interactive agents, where planning, tool use, environments, credit assignment, memory, skills, and refinement all shape long trajectories.

7. Multimodal RL PDF

   Moves beyond text-only policies by discussing visual reward models for understanding and trajectory-style RL for diffusion and flow matching generation.

8. Conclusions and Future Directions PDF

   Returns to the big picture and highlights open directions: RL for pre-training, cheaper RL, scaling-limit prediction, and continual self-evolving models.

9. Appendix: Systems and Datasets PDF

   Lists RL systems, preference datasets, reasoning data, and agent environments discussed in the paper.

Companion Materials

Citation

@misc{wang2026rlwithouttears,
  title  = {RL without Tears: An Introduction in the Era of LLMs},
  author = {Chenglong Wang and Hang Zhou and Tongran Liu and Jingbo Zhu and Tong Xiao},
  year   = {2026},
  note   = {Companion site}
}