Reinforcement Learning

Keywords

ver. 1.0.0

Comprehensive RL foundations and practical methods from classical value/policy/model algorithms to deep-RL stabilization and modern applications in LLMs, RLHF, and self-play.

This node summarizes core reinforcement learning theory and practice: the MDP and agent-environment loop, central challenges (credit assignment, instability, reward design, exploration), algorithm classes (policy, value, model), value-based and policy-gradient methods (TD, Q‑learning, SARSA, TD(λ), policy gradients), deep‑RL stabilizers (replay, targets, GAE, trust‑region/PPO), and practical applications including SFT, RLHF/RLVR, GRPO, distributed trainer/inference systems, and self-play/MCTS for game-style learning.

Full summary

This unit synthesizes the foundations, algorithms, stabilization techniques, and practical deployments of modern reinforcement learning.

Core concepts and framing - The agent–environment interaction loop, trajectories, rewards vs. supervised labels, and the Markov decision process (MDP) and policy formalisms. - Training loop and data distribution: collect-then-update cycles, nonstationary/stale data, and the resulting difficulties in optimization. - Four central practical challenges: credit assignment, optimisation instability, reward design (specification), and exploration vs. exploitation.

Algorithm taxonomy and principles - Classify methods by the learned object: policies (direct policy optimisation), value functions (V, Q, advantage), and dynamics/models (model-based planning). - Value-function foundations: Bellman equations/backups, dynamic programming, Monte‑Carlo, temporal‑difference (TD) learning, multi-step returns and TD(λ). - Control algorithms: on‑policy SARSA, off‑policy Q‑learning, and how multi-step and λ-return trade bias vs. variance. - Policy-gradient foundations: score-function gradients, variance reduction via baselines, reward‑to‑go, and generalized advantage estimation (GAE).

Scaling and stabilisation for deep RL - Challenges introduced by function approximation and neural networks: catastrophic drift and instability. - Practical stabilizers: experience replay buffers, target networks, multi-step returns, importance sampling corrections for off‑policy data. - Policy-stabilisation methods: trust-region concepts, proximal objectives, and PPO’s clipped surrogate objective as a practical, robust optimizer. - How to derive and implement core update rules, and choose trade-offs (bias vs. variance, stability vs. sample efficiency).

Modern applications: LLMs, preference learning, and games - Post-training regimes for large models: supervised fine-tuning (SFT), preference-based RL (RLHF), and verifier-based RL (RLVR). - Mapping token generation to an MDP and practical algorithmic adjustments: entropy targeting, length control, dynamic sampling, and group‑sampled baselines (e.g., GRPO) that avoid learned critics. - Distributed training and inference engineering: trainer/inference separation, stale-policy corrections, and importance-sampling to handle moving distributions. - Self-play and search: how policy iteration combined with Monte‑Carlo Tree Search and self-play yields AlphaZero-style learning; why zero-sum self-play can converge and how search reuses experience.

What you’ll be able to do - Formulate control and generation problems as MDPs and select suitable algorithm classes. - Derive, implement, and reason about core updates (TD, Q, policy gradients) and stabilization techniques for deep RL. - Analyze credit assignment and bias–variance trade-offs across value-based, policy-gradient, and hybrid methods. - Understand and evaluate practical systems for LLM post‑training (SFT ↔︎ RL hybrids), distributed inference/training concerns, and self-play/search-based learning pipelines.

Overall, the node presents a complete arc from RL theory and algorithmic building blocks to the pragmatic techniques and system-level considerations required to train stable, scalable agents in control, language, and game domains.

Units

I Foundations

Introduces core reinforcement learning concepts, formalism, common algorithm families, practical challenges, and how to formulate and train an RL agent for a control task.

II Diving deeper

From Bellman backups to PPO: value methods, temporal-difference learning, function approximation, policy gradients and modern stabilisation techniques.

III RL at scale

How modern RL is applied to large language models and games, trading critics for sampled baselines, engineering inference-trainer systems, and reusing search via self-play.

--- title: "Reinforcement Learning" keywords: - ver. 1.0.0 --- **Comprehensive RL foundations and practical methods from classical value/policy/model algorithms to deep-RL stabilization and modern applications in LLMs, RLHF, and self-play.** > This node summarizes core reinforcement learning theory and practice: the MDP and agent-environment loop, central challenges (credit assignment, instability, reward design, exploration), algorithm classes (policy, value, model), value-based and policy-gradient methods (TD, Q‑learning, SARSA, TD(λ), policy gradients), deep‑RL stabilizers (replay, targets, GAE, trust‑region/PPO), and practical applications including SFT, RLHF/RLVR, GRPO, distributed trainer/inference systems, and self-play/MCTS for game-style learning. ::: {.callout-note collapse="true" title="Full summary"} This unit synthesizes the foundations, algorithms, stabilization techniques, and practical deployments of modern reinforcement learning. Core concepts and framing - The agent–environment interaction loop, trajectories, rewards vs. supervised labels, and the Markov decision process (MDP) and policy formalisms. - Training loop and data distribution: collect-then-update cycles, nonstationary/stale data, and the resulting difficulties in optimization. - Four central practical challenges: credit assignment, optimisation instability, reward design (specification), and exploration vs. exploitation. Algorithm taxonomy and principles - Classify methods by the learned object: policies (direct policy optimisation), value functions (V, Q, advantage), and dynamics/models (model-based planning). - Value-function foundations: Bellman equations/backups, dynamic programming, Monte‑Carlo, temporal‑difference (TD) learning, multi-step returns and TD(λ). - Control algorithms: on‑policy SARSA, off‑policy Q‑learning, and how multi-step and λ-return trade bias vs. variance. - Policy-gradient foundations: score-function gradients, variance reduction via baselines, reward‑to‑go, and generalized advantage estimation (GAE). Scaling and stabilisation for deep RL - Challenges introduced by function approximation and neural networks: catastrophic drift and instability. - Practical stabilizers: experience replay buffers, target networks, multi-step returns, importance sampling corrections for off‑policy data. - Policy-stabilisation methods: trust-region concepts, proximal objectives, and PPO's clipped surrogate objective as a practical, robust optimizer. - How to derive and implement core update rules, and choose trade-offs (bias vs. variance, stability vs. sample efficiency). Modern applications: LLMs, preference learning, and games - Post-training regimes for large models: supervised fine-tuning (SFT), preference-based RL (RLHF), and verifier-based RL (RLVR). - Mapping token generation to an MDP and practical algorithmic adjustments: entropy targeting, length control, dynamic sampling, and group‑sampled baselines (e.g., GRPO) that avoid learned critics. - Distributed training and inference engineering: trainer/inference separation, stale-policy corrections, and importance-sampling to handle moving distributions. - Self-play and search: how policy iteration combined with Monte‑Carlo Tree Search and self-play yields AlphaZero-style learning; why zero-sum self-play can converge and how search reuses experience. What you’ll be able to do - Formulate control and generation problems as MDPs and select suitable algorithm classes. - Derive, implement, and reason about core updates (TD, Q, policy gradients) and stabilization techniques for deep RL. - Analyze credit assignment and bias–variance trade-offs across value-based, policy-gradient, and hybrid methods. - Understand and evaluate practical systems for LLM post‑training (SFT ↔ RL hybrids), distributed inference/training concerns, and self-play/search-based learning pipelines. Overall, the node presents a complete arc from RL theory and algorithmic building blocks to the pragmatic techniques and system-level considerations required to train stable, scalable agents in control, language, and game domains. ::: ## Units ### [I Foundations](i_foundations/index.qmd) **Introduces core reinforcement learning concepts, formalism, common algorithm families, practical challenges, and how to formulate and train an RL agent for a control task.** ### [II Diving deeper](ii_diving_deeper/index.qmd) **From Bellman backups to PPO: value methods, temporal-difference learning, function approximation, policy gradients and modern stabilisation techniques.** ### [III RL at scale](iii_rl_at_scale/index.qmd) **How modern RL is applied to large language models and games, trading critics for sampled baselines, engineering inference-trainer systems, and reusing search via self-play.**