Topic
All digests tagged Policy Optimization
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
The lecture details advanced RL methods for LLMs, covering PPO, GRPO, and On-Policy Distillation (OPD). These techniques maximize expected rewards by optimizing policies against human preferences or verifiable outcomes, while managing gradient variance and policy deviation.
Read article
This session details implementing Reinforcement Learning with Verifiable Rewards (RLVR) using the GRPO algorithm to train a reasoning model. The process involves calculating rewards, advantages, and policy gradient loss to enhance reasoning capability.
Read article
Long-horizon agents require structured experiments, not just prompts. We introduce the Auto Research loop: a meta-system that runs controlled scenarios, scores behavior via a balanced scorecard, and proposes constrained policy changes to improve social consistency.
Read article
The lecture details model-based RL, emphasizing that when using learned dynamics for planning, quantifying uncertainty (epistemic) is crucial. Techniques like Bayesian statistics and ensembles allow computing expected rewards by averaging over a posterior distribution of possible models.
Read article
GRPO trains agents by sampling multiple completions per prompt and using the group's relative scores as the training signal. This method moves beyond imitation learning, allowing models to learn from their own varied trajectories.
Read article