What Is an Inference Engine, Anyway? — Charles Frye, Modal
The talk details the architecture of inference engines, covering prefill/decode phases, the role of the scheduler, and advanced optimizations like KV caching and speculative decoding to maximize GPU throughput.
Read article