Topic
All digests tagged Quantization
Suggested route
Move from earlier context through the available material to the newest signals. This route is a reading aid, not a difficulty ranking.
Score Centering stabilizes Off-policy RL for LLMs by correcting the policy gradient loss when the training and inference policies mismatch. The method uses a Top-K approximation for efficient computation.
Read article
GSQ and RCO are complementary methods for accurate low-bit LLM quantization. GSQ uses Gumbel-Softmax sampling for stochastic optimization, while RCO optimizes bit allocation under fixed size budgets, enabling state-of-the-art model compression.
Read article
Learn how speculative decoding techniques like MTP and DFlash achieve massive LLM speed-ups by training small 'draft models' to predict multiple tokens in parallel, validating them with the main model in a single forward pass.
Read article
Running open LLMs in production requires managing the full loop: inference, data, post-training, and deployment. Nebius Token Factory provides a full-stack solution, optimizing performance using techniques like speculative decoding and KV cache offloading.
Read article
The talk details the 'big three' of inference engineering: Quantization, KV compaction, and Speculation. Learn about DFlash, TurboQuant, and how training is merging with inference optimization.
Read article
The talk presents formal verification for AI-generated GPU kernels, detailing how tools like Tatwa use Z3 solvers to prove semantic equivalence between reference and optimized kernels, moving beyond the limitations of traditional testing.
Read article
Learn how NVFP4 compares to BF16 for Nemotron 3.5 Lightning. Discover how this low-precision format shrinks memory footprint and boosts inference speed without a major quality hit.
Read article
Learn how to run local LLMs like Qwen3 8B on a Raspberry Pi using llama.cpp. This guide covers installation, selecting optimal GGUF quantization, and running models entirely offline.
Read article
Model compression, particularly through advanced quantization like NVFP4 (a 4-bit float format), is critical for running massive LLMs on edge devices. Techniques allow shrinking models dramatically (e.g., GLM 5.2 from 1.5 TB to 250 GB) while maintaining performance.
Read article
Llama.cpp and vLLM are two key engines for local LLMs. Llama.cpp optimizes for consumer hardware using quantization and GGUF files, while vLLM focuses on production scale with continuous batching and paged attention.
Read article
To deploy AI on edge devices, the focus must shift from compute to DRAM cost. Tiny LLMs and aggressive quantization allow running capable models like a 2B parameter Gemma on hardware like Raspberry Pi or Qualcomm NPUs.
Read article
This advanced session dives into local AI inference engines, demonstrating how memory bandwidth and engine selection—not just hardware capacity—determine performance for LLMs. Learn to build robust, private, self-hosted AI stacks.
Read article
Inkling is a massive, open-weight multimodal model with 1T parameters and 1M context window. It supports BF16/NVFP4 formats and integrates seamlessly with vLLM, SGLang, and llama.cpp for robust deployment.
Read article