Topic
All digests tagged Inference Optimization
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Learn how speculative decoding techniques like MTP and DFlash achieve massive LLM speed-ups by training small 'draft models' to predict multiple tokens in parallel, validating them with the main model in a single forward pass.
Read article
The talk details the architecture of inference engines, covering prefill/decode phases, the role of the scheduler, and advanced optimizations like KV caching and speculative decoding to maximize GPU throughput.
Read article
Evaluating open models requires moving beyond simple scores. The focus is now on agentic use cases, balancing intelligence, cost, and speed across multi-turn tasks. Learn about quantization and rigorous benchmarking.
Read article
Learn how H-Company optimizes VLM serving for computer-use agents using NVIDIA Dynamo. Discover techniques like embedding caching, Model Express, and advanced routing to scale complex, multi-modal AI workloads.
Read article
Learn how NVFP4 compares to BF16 for Nemotron 3.5 Lightning. Discover how this low-precision format shrinks memory footprint and boosts inference speed without a major quality hit.
Read article
This deep dive covers loading pre-trained LLMs (Qwen3), implementing autoregressive text generation in PyTorch, and optimizing performance using KV caching and torch.compile for maximum inference speed.
Read article
This lecture examines test-time compute scaling, showing how repeated sampling and advanced architectures like Arkon can boost model performance without retraining. It covers power laws and optimizing inference pipelines.
Read article
Inkling is a massive, open-weight multimodal model with 1T parameters and 1M context window. It supports BF16/NVFP4 formats and integrates seamlessly with vLLM, SGLang, and llama.cpp for robust deployment.
Read article