The Definitive Quantum Leap of 2026! Intel’s “AutoRound” Achieves FP8 Support & Astonishing Accuracy with Ultra-Low Bit Rates

📰 News Overview

• Support for FP8 Block Quantization: Thanks to the March 2026 update, the latest quantization scheme using the --scheme FP8_BLOCK option is now at your fingertips.
• High Precision at Ultra-Low Bit Rates: With the adoption of the SignRoundV2 algorithm, model performance is maximized even at a drastic 2-4 bit width.
• Broad Ecosystem and Hardware Support: Integrated with vLLM, Transformers, and SGLang, supporting a variety of execution environments from Intel CPU/GPU to NVIDIA GPU and Habana Gaudi.

💡 Key Points

• Support for MTP Layers: The latest PR includes support for quantization of MTP (Multi-Token Prediction) layers, enabling optimization for more complex architectures.
• Remarkable Compression Efficiency: In DeepSeek-R1 (INT2 mixed), the size has been successfully reduced to about 200GB while maintaining an impressive accuracy of 97.9%.
• Rapid Quantization Process: For 7B class models, the quantization process can be completed in about 10 minutes using a single GPU, showcasing extraordinary speed.

🦈 Shark’s Eye (Curator’s Perspective)

The implementation of the “SignRoundV2” algorithm is nothing short of revolutionary! It’s not just about “trimming weights”; it cleverly utilizes sign-gradient descent to achieve high precision with minimal tuning, providing a very concrete and practical design. The addition of “FP8 block quantization” since 2026 is a game-changer for those aiming to boost inference speed while saving computational resources—it’s precisely what the industry has been craving! Bringing massive models like DeepSeek-R1 down to a practical 200GB level represents a paradigm shift for local LLM operations!

🚀 What’s Next?

Support for more advanced data types (dtypes) like MXFP4 and NVFP4 is underway, and comprehensive optimizations such as W8A8 (8-bit weights and 8-bit activations) will soon become the standard, going beyond just “weight-only” quantization. Expect Intel hardware to solidify its place as one of the top choices, not just in CUDA environments but across the board.

💬 A Word from HaruShark

Running on just 2 bits is like being on a diet so extreme that you’re down to bones, yet still moving like a superhero! Intel’s serious game is something to bite into! 🦈🔥

📚 Terminology Explained

• FP8 Block Quantization: A method that applies the FP8 format, treating numbers as 8-bit floating-point values in fixed “block” units, dramatically reducing memory consumption while maintaining accuracy.

• SignRoundV2: The core algorithm of AutoRound. It efficiently optimizes post-quantization weights by leveraging the “sign” of the gradients, preventing accuracy degradation.

• MTP (Multi-Token Prediction): A technique that predicts multiple tokens simultaneously, not just the next one. With support for quantization of this layer, the latest high-speed models can now be made lighter.

• Source: Advanced Quantization Algorithm for LLMs