Topic
All digests tagged LLM Inference
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
LLM inference is the process of running a trained model against new input. It involves tokenization, continuous batching, and two phases: compute-bound prefill and memory bandwidth-bound decode.
Read article
The Memory Wall explains why faster chips don't guarantee faster AI. Learn how Arithmetic Intensity and the Roofline Model help diagnose if your LLM workload is compute-bound or memory-bound.
Read article
An H100 timing side channel allows an attacker to infer an LLM's model type, depth, and decoding speed by monitoring microsecond timing variations on shared GPU hardware, even without reading memory or exploiting bugs.
Read article
Discover how local AI is redefining compute sovereignty. Learn about running Nemotron 3.5 Lightning on DGX Spark and DGX Station, and how unified stacks enable high-performance, private LLM inference.
Read article
Agentic inference requires optimizing end-to-end task latency, not single-request latency. FriendliAI's cloud architecture uses Prefix Caching and Cache Aware Routing to make open-weight models economically viable for production agents.
Read article
Discover why LLM performance benchmarks are often unreliable. Learn how to use Inference Perf to achieve high-fidelity, reproducible, and production-scale benchmarking for your inference stack.
Read article
This deep dive explores inference-time scaling techniques for LLMs, covering Chain-of-Thought prompting, Temperature Scaling, Top-p filtering, and Self-Consistency. Learn how to boost model accuracy by managing compute trade-offs.
Read article
This deep dive covers LLM inference optimization at scale, detailing how memory and compute costs are managed through techniques like Paged Attention, Continuous Batching, and model quantization.
Read article
Learn how KV Cache-Aware Routing and P/D Disaggregation optimize LLM inference for agentic workloads. Discover strategies to reduce P99 latency from 900ms to 100ms using advanced Kubernetes patterns.
Read article