AI Engineer

Stop Shipping LLM Settings That Only Win on Paper - Rohan Pandey & Balaji Varadarajan, DigitalOcean

Published 2026-11-12 · Duration 5:33

Summary

This presentation details an optimization agent designed to automate the process of finding optimal vLLM serving configurations for Large Language Models (LLMs). The agent utilizes Bayesian optimization (via Optuna) to systematically search the parameter space, significantly reducing the manual effort required to tune LLM serving recipes. The process involves running multiple trials on specific hardware, generating an HTML report of all runs, and outputting a final optimized YAML configuration.

Download summary

Key takeaways

  1. Optimization Strategy

    The agent employs Bayesian optimization, a technique that minimizes the search space to efficiently find optimal parameters within a given vLLM serving recipe, saving substantial time compared to manual tuning.

  2. Performance Gains Demonstrated 4:00

    In a run using the GPT-OSS 120B model, the agent achieved a 17.5% improvement in tokens generated compared to the initial configuration. For the DeepSeek R1 Distill model, the agent doubled token throughput and reduced ITL by nearly half.

  3. Automation Workflow 2:00

    The process is automated by running a script (`tuner.py`) which requires inputs including the study name, the starting YAML recipe, the target hardware, and the number of trials (e.g., 24).

More digests on LLM Serving

Technical details

  • Optimization Methodology 30s

    The core strategy is Bayesian optimization, implemented using the Optuna tool. This method iteratively assesses a recipe, generates a score, and uses that score to suggest improvements for the next trial until the specified number of trials is completed.

  • vLLM Serving Recipe Structure 80s

    A vLLM recipe is defined by multiple regimes (e.g., chat short, chat long, rag, batch classify), each having its own set of input tokens. The starting configuration is provided in a YAML file.

  • Agent Execution 120s

    To run the agent, one must activate a virtual environment and execute the command: `python tuner.py`, providing the study name, starting YAML file, hardware, and number of trials.

Mentioned resources

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.