Half-Quadratic Quantization of large machine learning models

Half-Quadratic Quantization (HQQ) is a calibration-free method for compressing large machine learning models while retaining quality comparable to calibration-based techniques such as GPTQ and AWQ. It minimizes weight reconstruction error rather than activation error and uses a sparsity-promoting (l_p) loss to better handle outliers. Because HQQ relies on closed-form alternating updates instead of gradient-based optimization, it can quantize models dramatically faster—reportedly processing Llama-2-70B in under five minutes.

Why Quantization Matters

• Large language models require substantial memory for training and inference.
• Methods such as bitsandbytes, GPTQ, and AWQ make models like Llama-2 usable on consumer GPUs.
• Weight-only quantization approaches fall into two groups:
  • Calibration-free methods, such as bitsandbytes, use only model weights.
  • Calibration-based methods, such as GPTQ and AWQ, use external datasets.
• Calibration-based approaches can provide better quality but:
  • Their results may depend on calibration-data bias.
  • Calibration can be computationally expensive for very large models.

HQQ’s Quantization Objective

• Standard quantization can significantly distort weights, particularly outliers with unusually large values.
• GPTQ and AWQ reduce the effect of these distortions by minimizing layer-output or activation error using calibration data.
• HQQ instead minimizes reconstruction error directly in the weights.
• It uses a sparsity-promoting (l_p) loss, especially with (p<1), to model heavy-tailed outlier errors more effectively than squared error.
• Quantization is defined using:
  • A scale (s)
  • A zero-point (z)
  • A quantization operator (Q_{z,s}(W)=\text{round}(W/s+z))
  • A dequantization operator (Q^{-1}_{z,s}(W_q)=s(W_q-z))
• HQQ fixes the scale and optimizes the zero-point, simplifying the optimization problem.

Half-Quadratic Optimization

• Since the (l_p) objective with (p<1) is non-convex, HQQ introduces an auxiliary error variable (W_e).
• The resulting problem is solved through alternating optimization:
  • Update (W_e) while holding (z) fixed.
  • Update (z) while holding (W_e) fixed.
  • Increase a positive penalty parameter (\beta) by a factor (\kappa) each iteration.
• This decomposition turns the original difficult problem into simpler sub-problems with closed-form solutions.

Solving the Sub-Problems

• The (W_e) update is a proximal operation.

• For (l_1) regularization, it corresponds to soft thresholding.

• HQQ uses a generalized soft-thresholding operator for (0\leq p\leq1):

  [ \text{shrink}_{l_p}(x,\beta) =\text{sign}(x),\text{relu}\left(|x|-\frac{|x|^{p-1}}{\beta}\right) ]

• The zero-point update:

  • Recomputes quantized weights using the current zero-point.
  • Calculates the difference between quantized weights and corrected original weights.
  • Sets the new zero-point to the average over the quantization grouping axis.

• The implementation optimizes the inverse scale (1/s), which is more numerically stable in half-precision arithmetic.

Speed and Practical Advantages

• HQQ uses closed-form updates rather than gradients or automatic differentiation.
• Quantization can run in inference mode with half-precision arithmetic.
• The solver typically converges in only a few iterations.
• In contrast, AdamW with PyTorch autograd may require thousands of iterations and fails when using (p<1).
• The article reports HQQ as:
  • More than 100 times faster than autograd for quantizing Llama-2-7B.
  • More than 50 times faster than GPTQ for Llama-2-70B.
  • Capable of quantizing the largest models in only a few minutes.
• A 2-bit HQQ version of Llama-2-70B reportedly outperforms full-precision Llama-2-13B at a comparable memory footprint.

HQQ is therefore presented as a practical alternative to calibration-based quantization: it combines calibration-free operation and very high speed with competitive compression quality, making rapid experimentation and deployment of large models more feasible.