TurboQuant: Redefining AI efficiency with extreme compression
TurboQuant is a quantization framework designed to dramatically reduce memory use in large language models and vector search without sacrificing accuracy. It combines PolarQuant’s efficient vector compression with QJL’s one-bit residual correction to eliminate the overhead found in traditional quantization. Experiments show that it can compress KV caches to 3 bits, reduce memory by at least 6×, and accelerate attention-logit computation by up to 8×. ## The Memory Challenge in AI - High-dimensional vectors power language understanding, image features, vector search, and model attention. - These vectors consume substantial memory, particularly in the key-value (KV) cache used to store frequently accessed attention information. - Traditional vector quantization reduces vector size but often requires full-precision scaling or normalization constants for each block. - This metadata can add one or two bits per value, undermining the benefits of compression. ## TurboQuant’s Two-Stage Approach - TurboQuant first applies a random rotation to simplify the geometry of the data. - PolarQuant then compresses the transformed vectors using a standard quantizer, dedicating most bits to the vector’s primary information. - A remaining single bit is used by QJL to encode residual error. - QJL removes bias from the initial compression, improving the accuracy of attention-score calculations. - The approach requires no model training or fine-tuning. ## QJL: One-Bit Error Correction - QJL builds on the Johnson-Lindenstrauss Transform, which preserves important distances and relationships in high-dimensional data. - It represents each transformed value using only its sign: +1 or −1. - A specialized estimator combines low-precision stored data with a high-precision query. - This preserves accurate attention scores while introducing effectively zero memory overhead. ## PolarQuant: Compression Without Metadata Overhead - PolarQuant converts vectors from Cartesian coordinates into polar coordinates. - Instead of separately storing coordinate values, it represents vectors through: - A radius, capturing magnitude or signal strength - Angles, capturing direction and semantic structure - Because angular values follow a predictable, concentrated distribution, PolarQuant avoids expensive per-block normalization constants. - It recursively groups coordinate pairs and transforms their radii until the vector becomes one final radius plus a collection of angles. - This produces a compact representation with fixed, predictable boundaries. ## Experimental Results - The methods were tested on LongBench, Needle In A Haystack, ZeroSCROLLS, RULER, and L-Eval using Gemma and Mistral models. - TurboQuant achieved strong dot-product distortion and recall results while minimizing KV-cache memory. - On needle-in-a-haystack tasks, TurboQuant maintained perfect downstream performance while reducing KV memory by at least 6×. - PolarQuant was also nearly lossless on these tasks. - TurboQuant compressed KV caches to 3 bits without accuracy degradation. - Quantized models ran faster than the original uncompressed models. - On H100 GPUs, 4-bit TurboQuant delivered up to an 8× speedup for attention-logit computation compared with 32-bit keys. - The method has negligible runtime overhead and is relatively simple to implement. TurboQuant is presented as a practical way to make long-context LLMs and large-scale vector search more memory-efficient. Its combination of metadata-free PolarQuant compression and one-bit QJL correction is especially promising for deployments constrained by KV-cache capacity, latency, or GPU memory.
Read original(opens in new tab)