# Your Self-Hosted LLM Needs to Say No - Harshul Jain & Parv Khatri, Audible & SAP

## Executive summary

The talk addresses the critical need for Admission Control layers when deploying self-hosted Large Language Models (LLMs), especially in high-volume, bursty environments generated by coding agents. Since LLM inference consumes GPU memory (via the KV cache) and resources are finite, simply accepting all incoming requests leads to instability, service degradation, and potential crashes (as seen in major provider incidents). The solution is to implement an admission layer at the ingress point that proactively decides whether to admit, defer, or reject a request based on real-time system signals, thereby protecting the GPU cluster's Service Level Objectives (SLOs).

## Key takeaways

- The Necessity of Admission Control: Coding agents generate overwhelming, bursty traffic that can quickly saturate self-hosted LLM infrastructure. An admission layer must sit between the client/router and the inference engine to prevent resource exhaustion and ensure stability.
- Admission is a Commitment: Admitting a request is not a moment; it is a commitment that reserves GPU memory (KV cache) for the entire life of the request, from prefill to decode. This memory commitment must be managed to prevent resource overruns.
- Policy Decisions (Admit, Defer, Reject): The admission layer must decide whether to admit (commit resources), defer (wait and retry later, signaled by 503/Retry-After), or reject (fail fast, signaled by 429 or 529). The decision must happen before the request receives a 200 OK status.
- Advanced Policy Design: Effective policies move beyond simple rate counting. They must consider the cost of a request (e.g., KV memory usage, tokens) and monitor complex signals like KV cache pressure, queue depth, and deadline slack to make informed decisions.

## Technical details

- LLM Inference Pipeline: The request flows through an API Gateway (rate limiting) $\rightarrow$ Router $\rightarrow$ Inference Engine. The Inference Engine manages the KV cache, which stores the memory built up during token generation. The engine groups requests into batches to maximize GPU utilization.
- Inference Phases and Memory: Inference consists of two phases: Prefill (reading the prompt and building the initial KV cache) and Decode (generating the answer one token at a time). The memory commitment lasts until the request finishes or the client disconnects.
- Admission Signals and Rules: A robust admission policy requires two views: 1) Request attributes (deadline, client/tenant ID) and 2) Engine state (KV cache size, queue depth, prompt cache hit rate, replica load). Policies can use various rules, from simple thresholds to complex optimizers or formal bounds.
- Benchmarking Metrics: The recommended scoring metric is 'Goodput under Admission' (requests meeting their SLO / total requests arrived). This ensures that the policy's impact is measured against the total incoming load, not just the admitted requests.

## Practical implications

- Implement an admission layer at the ingress point of your LLM serving stack to manage bursty traffic from agents and clients.
- Design your policy to monitor multiple signals (e.g., KV cache pressure, queue depth) rather than relying solely on simple rate limiting.
- When benchmarking, use 'Goodput under Admission' and ensure your test workload mimics real, bursty agent traffic.
- Handle the three admission outcomes (200 OK, 503 Retry-After, 429/529) correctly on the client side, especially regarding connection closure and exponential backoff with jitter.

## Topics

LLM Inference, Admission Control, GPU Resource Management, Coding Agents, Distributed Systems, Service Level Objectives (SLOs), LLM inference at scale handbook, agentship, AdmitPerf

Source: https://www.youtube.com/watch?v=ntVpbyDTYaQ
