Quantization | Techlist.io

cloudflareJun 15, 20262 min readCurated summary

Growing the Cloudflare AI team with talent from Ensemble AI

Cloudflare is bringing key members of Ensemble AI onto its team to improve AI infrastructure and inference efficiency. Ensemble’s work on model compression, structured neural architectures, and parameter-efficient fine-tuning complements Cloudflare’s Workers AI platform. The combined effort aims to make powerful AI models faster, cheaper, and easier to deploy globally. ## Incorporating Ensemble AI’s Expertise - Ensemble AI has focused on reducing the memory, compute, and deployment costs of large language and multimodal models. - Its NdLinear technology replaces standard transformer linear layers while preserving multidimensional structure such as attention heads, channels, and spatial dimensions. - NdLinear-LoRA reduces the number of trainable parameters needed to fine-tune large models. - These techniques complement quantization and vector quantization to improve model efficiency without significantly sacrificing quality. ## Improving AI Inference Economics - Cloudflare Workers AI provides serverless GPU-powered inference across Cloudflare’s global network. - Lower model size, memory usage, and compute requirements can improve throughput, GPU utilization, and overall inference costs. - These improvements are increasingly important for agents, multimodal applications, personalization, fine-tuning, retrieval, and reinforcement learning. - The Ensemble team will contribute to Cloudflare’s existing work, including the Infire inference engine, Unweight tensor compression, and systems for running very large language models. ## Supporting Next-Generation Workloads - Developers increasingly need AI infrastructure that is reliable, affordable, globally distributed, and close to end users—not merely access to models. - Cloudflare’s network, serverless platform, and Workers AI provide a foundation for deploying AI with less operational complexity. - Combining Cloudflare’s infrastructure with Ensemble’s efficient model architectures should enable lower-cost, higher-performance AI deployments at scale. Cloudflare’s stated goal is to make advanced AI workloads more accessible by improving the economics and efficiency of inference across its platform.

Read original

Quantization Large Language Models Machine Learning Cloudflare Workers Ai+4

cloudflareApr 17, 20263 min readCurated summary

Unweight: how we compressed an LLM 22% without sacrificing quality

Unweight is Cloudflare’s lossless compression system for LLM weights, reducing model size by 15–22% while preserving bit-exact outputs. It targets the memory-bandwidth bottleneck in GPU inference by compressing weights in HBM and decompressing them directly into fast on-chip memory before tensor-core computation. On Llama-3.1-8B, the approach saves roughly 3 GB of VRAM and enables more models to run per GPU. ## The GPU Memory Bottleneck - LLM inference is often limited by memory bandwidth rather than computation. - Each generated token requires reading the model’s weights from GPU high-bandwidth memory (HBM). - NVIDIA H100 tensor cores can process data far faster than HBM can supply it. - Smaller weights reduce the amount of data transferred across the memory bus. - Decompression must be carefully integrated: if it adds latency that cannot overlap with matrix multiplication, token generation becomes slower. ## Why Lossless Compression Matters - Quantization commonly converts 16-bit values into 8- or 4-bit integers. - Because quantization is lossy, it can change model behavior and response quality unpredictably. - Unweight instead preserves exact outputs and does not require specialized hardware. - Existing systems were unsuitable because they focused on CPU decompression, custom FPGA hardware, or consumer GPUs rather than Hopper-generation GPUs and production inference. ## Compressing BF16 Weights - BF16 values contain: - A sign bit - An 8-bit exponent - A 7-bit mantissa - Sign and mantissa values appear largely random and are difficult to compress. - Exponents are highly predictable: the 16 most common exponent values account for more than 99% of weights in a typical layer. - Unweight applies Huffman coding to exponent bytes while leaving sign and mantissa bits unchanged. - Rare exponents are handled by storing an entire row of 64 weights verbatim, avoiding per-element branching during decoding. ## Selective Compression of Model Layers - Unweight compresses the MLP gate, up, and down projection matrices. - These matrices represent roughly two-thirds of model parameters and generate substantial memory traffic during decoding. - Attention weights, embeddings, and layer norms remain uncompressed. - The exponent compression produces about 30% savings in the targeted streams and approximately 20% reduction in total MLP weight size. - Overall model-size reductions reach 15–22%. ## Direct GPU Decompression - Model weights normally reside in large but slower HBM and are staged into small, fast shared memory before computation. - Conventional approaches decompress full matrices back into HBM and then run standard matrix multiplication, creating additional memory traffic. - Unweight decompresses weights in shared memory and feeds them directly to tensor cores. - Different execution strategies are used depending on the weight matrix and batch size. - An autotuner selects the fastest strategy for each workload. ## Results and Availability - Tests on Llama-3.1-8B achieved: - Around 30% compression for MLP weights - 15–22% reduction in total model size - Approximately 3 GB of VRAM savings - The savings allow more models to fit on each GPU, potentially reducing inference cost and improving global deployment coverage. - Cloudflare is publishing a technical paper and open-sourcing the GPU kernels. Unweight demonstrates that lossless, inference-time compression can improve GPU utilization without changing model behavior. The practical recommendation is to compress the portions of a model that dominate memory traffic while integrating decoding directly into the GPU execution path.

Read original

Quantization Database Design Large Language Models Rust+4

dropboxFeb 12, 20263 min readCurated summary

How low-bit inference enables efficient AI

Low-bit inference reduces the memory, compute, and energy required to serve modern AI models by representing values with fewer bits. Quantization can substantially increase GPU throughput, but its benefits depend on model accuracy, hardware support, and whether workloads prioritize latency or throughput. The article presents low-bit inference as a production trade-off rather than a universally optimal technique. ## The Rising Cost of Modern Models - Models are growing rapidly, increasing demand for: - Memory capacity - Compute power - Energy - Low-latency serving infrastructure - Dropbox uses attention-based models for Dash and other capabilities involving: - Text, image, video, and audio understanding - Search and summarization - Reasoning over large collections of content - Production deployment requires balancing model capability with hardware utilization, cost, and responsiveness. ## Where Inference Compute Is Spent - Most computation comes from repeated matrix multiplications in two areas: - **Linear layers**, including attention projections, MLP layers, and final output layers. - **Attention mechanisms**, which calculate relationships between input tokens and become increasingly expensive with longer contexts. - GPUs accelerate these operations using specialized hardware: - NVIDIA Tensor Cores - AMD Matrix Cores - These cores execute matrix multiply-accumulate operations much faster than general-purpose CUDA cores. ## How Lower Precision Improves Efficiency - Quantization reduces the number of bits used to represent model values. - Converting values from 16-bit to 8-bit or 4-bit formats: - Reduces memory usage - Lowers memory-transfer costs - Can increase matrix-operation throughput - Reduces energy consumption - GPU throughput generally improves as precision decreases; halving precision can approximately double the number of operations performed per second in suitable workloads. - Eight-bit quantization maps values into 256 discrete levels. Formats below 8 bits typically require **bitpacking**, combining multiple values into types such as `uint8` or `int32` because 4-bit values are not normally stored as native hardware types. - Newer hardware, such as Blackwell GPUs with FP4 support, can provide major energy savings compared with higher-precision systems like the H100. ## Limits of Extremely Low-Bit Formats - Binary and ternary quantization restricts weights to two or three possible values, offering greater theoretical savings. - These formats are not well matched to today’s GPUs because they cannot fully use Tensor or Matrix Cores. - Specialized accelerators could make them more practical, but adoption remains limited by: - Weak ecosystem support - Hardware availability - Concerns about model quality - Practical gains therefore depend not only on bit width, but also on how well the format is supported by existing hardware and software. ## Quantization Formats and Deployment Trade-offs - Quantization is a family of techniques with different choices for: - Numerical representation - Scaling - Execution strategy - These choices affect: - Model accuracy - Inference speed - Memory consumption - Hardware utilization - Different workloads have different priorities: - Latency-sensitive applications need fast individual requests. - Throughput-oriented workloads prioritize processing large volumes efficiently. - Depending on the workload, inference may be limited by software overhead, memory bandwidth, or specialized GPU compute units. ## Pre-MXFP and MXFP Approaches - The article divides modern low-bit formats into two broad groups following the introduction of **MXFP microscaling**: - **Pre-MXFP formats** rely on software-managed scaling and explicit dequantization. - **MXFP formats** move scaling and related operations into Tensor Core hardware. - MXFP aims to standardize low-bit data types while making them more directly usable by modern GPUs. - The choice between these approaches depends on the hardware generation and the specific performance requirements of each production workload. Low-bit inference is most effective when quantization formats, model quality, and hardware capabilities are considered together. Teams should select formats based on the actual bottleneck—memory, bandwidth, latency, or compute—rather than assuming that the fewest possible bits will always deliver the best result.

Read original

Quantization Large Language Models Gpu Acceleration Model Optimization+3