Half-Quadratic Quantization of large machine learning models
Half-Quadratic Quantization (HQQ) is a calibration-free method for compressing large machine learning models while retaining quality comparable to calibration-based techniques such as GPTQ and AWQ. It minimizes weight reconstruction error rather than activation error and uses a sparsity-promoting \(l_p\) loss to better handle outliers. Because HQQ relies on closed-form alternating updates instead of gradient-based optimization, it can quantize models dramatically faster—reportedly processing Llama-2-70B in under five minutes. ## Why Quantization Matters - Large language models require substantial memory for training and inference. - Methods such as bitsandbytes, GPTQ, and AWQ make models like Llama-2 usable on consumer GPUs. - Weight-only quantization approaches fall into two groups: - **Calibration-free methods**, such as bitsandbytes, use only model weights. - **Calibration-based methods**, such as GPTQ and AWQ, use external datasets. - Calibration-based approaches can provide better quality but: - Their results may depend on calibration-data bias. - Calibration can be computationally expensive for very large models. ## HQQ’s Quantization Objective - Standard quantization can significantly distort weights, particularly outliers with unusually large values. - GPTQ and AWQ reduce the effect of these distortions by minimizing layer-output or activation error using calibration data. - HQQ instead minimizes reconstruction error directly in the weights. - It uses a sparsity-promoting \(l_p\) loss, especially with \(p<1\), to model heavy-tailed outlier errors more effectively than squared error. - Quantization is defined using: - A scale \(s\) - A zero-point \(z\) - A quantization operator \(Q_{z,s}(W)=\text{round}(W/s+z)\) - A dequantization operator \(Q^{-1}_{z,s}(W_q)=s(W_q-z)\) - HQQ fixes the scale and optimizes the zero-point, simplifying the optimization problem. ## Half-Quadratic Optimization - Since the \(l_p\) objective with \(p<1\) is non-convex, HQQ introduces an auxiliary error variable \(W_e\). - The resulting problem is solved through alternating optimization: - Update \(W_e\) while holding \(z\) fixed. - Update \(z\) while holding \(W_e\) fixed. - Increase a positive penalty parameter \(\beta\) by a factor \(\kappa\) each iteration. - This decomposition turns the original difficult problem into simpler sub-problems with closed-form solutions. ## Solving the Sub-Problems - The \(W_e\) update is a proximal operation. - For \(l_1\) regularization, it corresponds to soft thresholding. - HQQ uses a generalized soft-thresholding operator for \(0\leq p\leq1\): \[ \text{shrink}_{l_p}(x,\beta) =\text{sign}(x)\,\text{relu}\left(|x|-\frac{|x|^{p-1}}{\beta}\right) \] - The zero-point update: - Recomputes quantized weights using the current zero-point. - Calculates the difference between quantized weights and corrected original weights. - Sets the new zero-point to the average over the quantization grouping axis. - The implementation optimizes the inverse scale \(1/s\), which is more numerically stable in half-precision arithmetic. ## Speed and Practical Advantages - HQQ uses closed-form updates rather than gradients or automatic differentiation. - Quantization can run in inference mode with half-precision arithmetic. - The solver typically converges in only a few iterations. - In contrast, AdamW with PyTorch autograd may require thousands of iterations and fails when using \(p<1\). - The article reports HQQ as: - More than 100 times faster than autograd for quantizing Llama-2-7B. - More than 50 times faster than GPTQ for Llama-2-70B. - Capable of quantizing the largest models in only a few minutes. - A 2-bit HQQ version of Llama-2-70B reportedly outperforms full-precision Llama-2-13B at a comparable memory footprint. HQQ is therefore presented as a practical alternative to calibration-based quantization: it combines calibration-free operation and very high speed with competitive compression quality, making rapid experimentation and deployment of large models more feasible.
Read original(opens in new tab)
