# Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean

## Executive summary

This presentation details an optimization agent designed to automate the process of finding optimal vLLM serving configurations for Large Language Models (LLMs). The agent utilizes Bayesian optimization (via Optuna) to systematically search the parameter space, significantly reducing the manual effort required to tune LLM serving recipes. The process involves running multiple trials on specific hardware, generating an HTML report of all runs, and outputting a final optimized YAML configuration.

## Key takeaways

- Optimization Strategy: The agent employs Bayesian optimization, a technique that minimizes the search space to efficiently find optimal parameters within a given vLLM serving recipe, saving substantial time compared to manual tuning.
- Performance Gains Demonstrated: In a run using the GPT-OSS 120B model, the agent achieved a 17.5% improvement in tokens generated compared to the initial configuration. For the DeepSeek R1 Distill model, the agent doubled token throughput and reduced ITL by nearly half.
- Automation Workflow: The process is automated by running a script (`tuner.py`) which requires inputs including the study name, the starting YAML recipe, the target hardware, and the number of trials (e.g., 24).

## Technical details

- Optimization Methodology: The core strategy is Bayesian optimization, implemented using the Optuna tool. This method iteratively assesses a recipe, generates a score, and uses that score to suggest improvements for the next trial until the specified number of trials is completed.
- vLLM Serving Recipe Structure: A vLLM recipe is defined by multiple regimes (e.g., chat short, chat long, rag, batch classify), each having its own set of input tokens. The starting configuration is provided in a YAML file.
- Agent Execution: To run the agent, one must activate a virtual environment and execute the command: `python tuner.py`, providing the study name, starting YAML file, hardware, and number of trials.

## Practical implications

- Reduces the time and manual effort required to find optimal LLM serving parameters.
- Provides a robust, automated method for tuning vLLM configurations across various workloads (e.g., RAG, chat, batch classification).
- The output provides both a detailed HTML report of all trials and a final optimized YAML file for deployment.

## Topics

LLM Serving, vLLM, Bayesian Optimization, MLOps, Optimization Agents, Kubernetes, Rohan Pandey, DigitalOcean, Balaji Varadarajan, DigitalOcean

Source: https://www.youtube.com/watch?v=cWrYEBlWpcY
