Unweight: how we compressed an LLM 22% without sacrificing quality (opens in new tab)

Unweight is Cloudflare’s lossless compression system for LLM weights, reducing model size by 15–22% while preserving bit-exact outputs. It targets the memory-bandwidth bottleneck in GPU inference by compressing weights in HBM and decompressing them directly into fast on-chip memory before tensor-core computation. On Llama-3.1-8B, the approach saves roughly 3 GB of VRAM and enables more models to run per GPU.

The GPU Memory Bottleneck

• LLM inference is often limited by memory bandwidth rather than computation.
• Each generated token requires reading the model’s weights from GPU high-bandwidth memory (HBM).
• NVIDIA H100 tensor cores can process data far faster than HBM can supply it.
• Smaller weights reduce the amount of data transferred across the memory bus.
• Decompression must be carefully integrated: if it adds latency that cannot overlap with matrix multiplication, token generation becomes slower.

Why Lossless Compression Matters

• Quantization commonly converts 16-bit values into 8- or 4-bit integers.
• Because quantization is lossy, it can change model behavior and response quality unpredictably.
• Unweight instead preserves exact outputs and does not require specialized hardware.
• Existing systems were unsuitable because they focused on CPU decompression, custom FPGA hardware, or consumer GPUs rather than Hopper-generation GPUs and production inference.

Compressing BF16 Weights

• BF16 values contain:
  • A sign bit
  • An 8-bit exponent
  • A 7-bit mantissa
• Sign and mantissa values appear largely random and are difficult to compress.
• Exponents are highly predictable: the 16 most common exponent values account for more than 99% of weights in a typical layer.
• Unweight applies Huffman coding to exponent bytes while leaving sign and mantissa bits unchanged.
• Rare exponents are handled by storing an entire row of 64 weights verbatim, avoiding per-element branching during decoding.

Selective Compression of Model Layers

• Unweight compresses the MLP gate, up, and down projection matrices.
• These matrices represent roughly two-thirds of model parameters and generate substantial memory traffic during decoding.
• Attention weights, embeddings, and layer norms remain uncompressed.
• The exponent compression produces about 30% savings in the targeted streams and approximately 20% reduction in total MLP weight size.
• Overall model-size reductions reach 15–22%.

Direct GPU Decompression

• Model weights normally reside in large but slower HBM and are staged into small, fast shared memory before computation.
• Conventional approaches decompress full matrices back into HBM and then run standard matrix multiplication, creating additional memory traffic.
• Unweight decompresses weights in shared memory and feeds them directly to tensor cores.
• Different execution strategies are used depending on the weight matrix and batch size.
• An autotuner selects the fastest strategy for each workload.

Results and Availability

• Tests on Llama-3.1-8B achieved:
  • Around 30% compression for MLP weights
  • 15–22% reduction in total model size
  • Approximately 3 GB of VRAM savings
• The savings allow more models to fit on each GPU, potentially reducing inference cost and improving global deployment coverage.
• Cloudflare is publishing a technical paper and open-sourcing the GPU kernels.

Unweight demonstrates that lossless, inference-time compression can improve GPU utilization without changing model behavior. The practical recommendation is to compress the portions of a model that dominate memory traffic while integrating decoding directly into the GPU execution path.