Topic
All digests tagged vLLM
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Stop guessing your LLM configs! Learn how to use data-driven benchmarking with BenchPod, a vLLM dashboard, to optimize parameters like max batch tokens and ensure reliable, production-ready AI deployments.
Read article
LLM inference is the process of running a trained model against new input. It involves tokenization, continuous batching, and two phases: compute-bound prefill and memory bandwidth-bound decode.
Read article
The talk details the architecture of inference engines, covering prefill/decode phases, the role of the scheduler, and advanced optimizations like KV caching and speculative decoding to maximize GPU throughput.
Read article
Deep dives into scaling AI: Learn how to customize generative models (Flux), optimize LLM inference using vLLM, and build real-time voice diarization systems for production use.
Read article
Two critical bugs in stateful inference (vLLM/Mamba) were found by starving the GPU and comparing logprobs. The failures were silent, requiring deep kernel debugging and type fixes.
Read article