# Optimizing AI Agents: Prompts vs. Weights (GEPA, GRPO) | Stanford CS329Z Lecture 9 with Michael Ryan

## Executive summary

This lecture provides a comprehensive guide to optimizing AI agents, contrasting prompt-based (context) optimization with weight-based fine-tuning. The speaker, Michael Ryan, details various techniques, including FShot optimization, Reflective Optimization (OPRO), and genetic algorithms (GEPA), alongside weight optimization methods like Supervised Fine-Tuning (SFT) and Generalized Policy Optimization (GRPO). The discussion emphasizes that optimization is crucial for 'last mile adaptation' when general models are insufficient for specific domain tasks, and that the choice between prompt and weight optimization depends on resource constraints and system requirements.

## Key takeaways

- Context Optimization vs. Weight Optimization: Context optimization (prompting) is useful for API-based models where fine-tuning is unavailable, and is often faster for rapid iteration. Weight optimization (fine-tuning) is necessary when context length is a constraint or when a cheaper, smaller model is required.
- Bootstrap Reasoning and FShot Optimization: Methods like DSPI (Bootstrap Random Search) and Fantastically Ordered Prompts use rejection sampling to generate high-quality few-shot examples and reasoning traces from unlabeled data, which are then used in the prompt context.
- Reflective Optimization (OPRO): OPRO uses a meta-prompt to allow the LLM to reflect on past attempts (including their scores) and generate new, improved instructions, effectively using the LLM as an optimizer.
- Weight Optimization Methods (SFT & RL): SFT uses a 'teacher' model's successful trajectories to train a 'student' model. RL methods like GRPO (Generalized Policy Optimization) reward the student model for task completion, allowing the model to learn from diverse and complex environments.
- Agents as Optimizers: Advanced techniques like the Darwin Gödel Machine and Auto-Research demonstrate that agents can self-modify their own code or harness based on performance metrics, leading to open-ended self-improvement.

## Technical details

- FShot Optimization: This involves providing examples in the prompt. Fantastically Ordered Prompts specifically addresses the model's sensitivity to the ordering of these examples, using metrics like global and local entropy to find optimal sequences.
- Bootstrap Random Search (DSPI): This method generates high-quality few-shot examples by running the program multiple times (rejection sampling) and only keeping traces that score well on a metric, which are then combined and tested.
- Reflective Optimization (OPRO): OPRO uses a meta-prompt to feed the LLM a history of past instructions and their scores, asking the LLM to generate a new, improved instruction.
- Genetic Prompt Evolution (GEPA): GEPA starts with a candidate pool of prompts. It uses a Pareto frontier approach to filter out non-optimal candidates and then generates new candidates via 'reflective prompt mutation' or 'system aware merge.'
- Reinforcement Learning (GRPO): GRPO is an efficient RL method that uses a group advantage calculation (based on mean reward and standard deviation) instead of a separate value model, reducing the number of models required compared to PO.

## Practical implications

- Context optimization is essential when using API-based models that do not allow fine-tuning.
- Weight optimization (e.g., distillation) is a viable strategy to reduce cost and latency by training a smaller model to mimic a larger 'teacher' model.
- The choice of optimization strategy (prompt vs. weight) depends on the budget, the availability of a capable teacher model, and the required speed of iteration.
- RL methods like GRPO are highly efficient for training frontier models, especially when a teacher model is not available for SFT.

## Topics

AI Agents, Prompt Engineering, Weight Fine-Tuning, Reinforcement Learning, System Design, Stanford CS329Z: Engineering AI Agents, DSPI, OPRO (Large Language Models as Optimizers)

Source: https://www.youtube.com/watch?v=e_m99aR7_3Q
