AI Engineer

Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

Published 2026-11-09 · Duration 52:08

Summary

The talk addresses the critical need for Admission Control layers when deploying self-hosted Large Language Models (LLMs), especially in high-volume, bursty environments generated by coding agents. Since LLM inference consumes GPU memory (via the KV cache) and resources are finite, simply accepting all incoming requests leads to instability, service degradation, and potential crashes (as seen in major provider incidents). The solution is to implement an admission layer at the ingress point that proactively decides whether to admit, defer, or reject a request based on real-time system signals, thereby protecting the GPU cluster's Service Level Objectives (SLOs).

Download summary

Key takeaways

  1. The Necessity of Admission Control 3:40

    Coding agents generate overwhelming, bursty traffic that can quickly saturate self-hosted LLM infrastructure. An admission layer must sit between the client/router and the inference engine to prevent resource exhaustion and ensure stability.

  2. Admission is a Commitment 9:40

    Admitting a request is not a moment; it is a commitment that reserves GPU memory (KV cache) for the entire life of the request, from prefill to decode. This memory commitment must be managed to prevent resource overruns.

  3. Policy Decisions (Admit, Defer, Reject) 16:40

    The admission layer must decide whether to admit (commit resources), defer (wait and retry later, signaled by 503/Retry-After), or reject (fail fast, signaled by 429 or 529). The decision must happen before the request receives a 200 OK status.

  4. Advanced Policy Design 26:40

    Effective policies move beyond simple rate counting. They must consider the cost of a request (e.g., KV memory usage, tokens) and monitor complex signals like KV cache pressure, queue depth, and deadline slack to make informed decisions.

More digests on LLM Inference

Technical details

  • LLM Inference Pipeline 280s

    The request flows through an API Gateway (rate limiting) $\rightarrow$ Router $\rightarrow$ Inference Engine. The Inference Engine manages the KV cache, which stores the memory built up during token generation. The engine groups requests into batches to maximize GPU utilization.

  • Inference Phases and Memory 480s

    Inference consists of two phases: Prefill (reading the prompt and building the initial KV cache) and Decode (generating the answer one token at a time). The memory commitment lasts until the request finishes or the client disconnects.

  • Admission Signals and Rules 1200s

    A robust admission policy requires two views: 1) Request attributes (deadline, client/tenant ID) and 2) Engine state (KV cache size, queue depth, prompt cache hit rate, replica load). Policies can use various rules, from simple thresholds to complex optimizers or formal bounds.

  • Benchmarking Metrics 2000s

    The recommended scoring metric is 'Goodput under Admission' (requests meeting their SLO / total requests arrived). This ensures that the policy's impact is measured against the total incoming load, not just the admitted requests.

Mentioned resources

  • LLM inference at scale handbook (Open-Source Handbook)
  • agentship (Open-Source Framework)
  • AdmitPerf (Tool/Library)

Channel & topics

Watch on YouTube · Back to latest

This independent, AI-assisted summary is provided for commentary and informational purposes. It may contain errors or omit important context. Please watch the original video for the creator's complete presentation. Video, thumbnail, and related copyrights belong to their respective owners.