cloudflareAug 3, 2026

Smaller, faster, safer: running Kimi and GLM at scale (opens in new tab)

Cloudflare improves the efficiency of serving large, long-context models by optimizing both GPU memory usage and request concurrency. Its approach combines FP8 KV-cache quantization, INT4 weight compression, and integrity checks for shared caches, while using separate prefill and decode pools to apply each optimization where it helps most. These techniques increase throughput and reduce costs without materially affecting model accuracy. ## Quantizing the KV Cache - Long-context models store attention keys and values in a KV cache, which often exhausts GPU memory before model weights do. - Cloudflare stores the cache in FP8 instead of BF16, cutting its size in half. - On Kimi K2.6, this increases available context from roughly 686,000 to 1.37 million tokens. - FP8 is slightly slower at the same concurrency because values must be converted during attention. - However, FP8 supports more concurrent requests: - BF16 runs out of memory at 32 requests. - FP8 reaches 2,192 tokens per second at 64 requests. - Peak throughput improves by about 41%, with roughly 30% lower cost per token. - Cloudflare keeps BF16 for prefill, where workloads are compute-bound. - Evaluation results show FP8 and BF16 produce effectively indistinguishable quality across reasoning, knowledge, tool-calling, and internal benchmarks. ## Compressing Model Weights - GLM 5.2 weights are compressed from FP8 to INT4 for the decode phase. - The checkpoint shrinks from 705 GB to 421 GB, while per-GPU memory in an eight-way deployment falls from about 88 GB to 52 GB. - The freed memory supports approximately 1.18 million tokens of KV cache. - INT4 improves decode performance because generation is memory-bandwidth-bound: - Single-request throughput rises from 60 to 92 tokens per second, a 55% gain. - Gains range from 16% to 27% at higher concurrency. - Prefill becomes slower with INT4 because compressed weights must be expanded before computation: - FP8 prefill: about 10,160 tokens per second. - INT4 prefill: about 8,660 tokens per second. - Cloudflare therefore uses FP8 for prefill and INT4 for decode. - Accuracy remains within 0.8 percentage points of the FP8 model across tested benchmarks. ## Protecting a Shared KV Cache - Greater memory efficiency allows hundreds of requests to share physical KV-cache pages, increasing the risk of page-allocation or bookkeeping errors. - Cloudflare assigns each cache page a changing tag whenever it is reallocated. - Requests record the pages and tags they expect, and the server validates these mappings before supported decode operations. - If a mismatch occurs, the request is aborted instead of reading incorrect data. - In production-style tests, integrity checking caused: - Less than 1% throughput reduction. - Less than 1% increase in p95 latency. - Validation runs as a separate batch check rather than inside the attention kernel, avoiding GPU synchronization races. - The feature is enabled per deployment, while deployments that do not use it incur no measurable overhead. ## Future Work - Cloudflare is expanding FP8 KV caches across its fleet. - It is testing NVFP4 weight compression on NVIDIA Blackwell GPUs. - The company is also working toward making cache integrity checks universally enabled at negligible cost. Together, these optimizations let Cloudflare serve larger models with more concurrent users, lower inference costs, and essentially unchanged model quality. Separating prefill and decode workloads is central to applying each precision choice where it delivers the best trade-off.

gpu-inferencemixture-of-expertskv-cachecontinuous-batching+4

netflixJul 17, 2026

In-House LLM Serving at Netflix (opens in new tab)

Netflix built an in-house LLM serving platform within its existing production ML infrastructure rather than creating a separate ML stack. The platform combines a JVM-based serving layer, NVIDIA Triton, GPU-backed Model Scoring Service, and an OpenAI-compatible HTTP frontend. Its main design choices—vLLM, model packaging, API compatibility, and deployment strategy—prioritize operational flexibility and seamless movement from hosted models to self-hosted ones, while production exposed versioning and compatibility risks. ## Architecture and Serving Model - Netflix’s unified JVM serving system handles routing, A/B testing, feature retrieval, inference, post-processing, and logging. - Callers access models through: - A gRPC path integrated with the existing serving system. - A direct HTTP path for newer LLM applications. - Small CPU models run in-process to avoid remote-call overhead. - Larger GPU models run through Model Scoring Service (MSS), which supports XGBoost, TensorFlow, PyTorch, and LLMs. - NVIDIA Triton manages model loading, batching, and GPU scheduling. - A Java control plane provides deployment, versioning, health checks, autoscaling, and multi-region rollout. ## Choosing vLLM as the Standard Engine - Netflix originally used TensorRT-LLM, but re-evaluated its choice as open-source engines improved and workloads diversified. - vLLM was selected based on operational fit rather than benchmark performance alone: - Supports custom model architectures without lengthy compilation. - Provides hooks for custom decoding and constraint logic. - Is easier to debug than earlier compiled-engine workflows. - Is familiar to many researchers, reducing the research-to-production transition cost. - The workload includes embeddings, prefill-only inference, autoregressive decoding, and custom per-step decoding constraints. ## Triton Integration and Model Packaging - Triton offers both a Python backend and a dedicated vLLM backend. - The Python backend requires explicit input and output tensor definitions, coupling packaged artifacts to frontend changes. - The vLLM backend uses a JSON configuration pointing to model weights and tokenizers, generating tensor specifications dynamically. - Netflix considers the vLLM backend the preferred default because models and frontends can evolve independently. - Production revealed two limitations: - Triton and vLLM must be version-pinned because incompatible APIs can prevent the backend from loading entirely. - Models requiring custom preprocessing, postprocessing, tokenization, or ensemble execution still need Triton’s Python backend. ## OpenAI-Compatible HTTP Frontend - Netflix keeps LLMs compatible with the same internal gRPC model-serving interface used by other model types. - It also exposes an OpenAI-compatible API because that interface is widely supported by inference engines, orchestration tools, evaluation systems, and client libraries. - This makes replacing a hosted model with a fine-tuned self-hosted model largely transparent to callers. - The implementation uses Triton’s OpenAI-compatible frontend, FastAPI, and a `TritonLLMEngine` that translates requests into Triton inference calls. - KServe HTTP and gRPC frontends remain available for the Java control plane. - Netflix found that Triton’s frontend silently discarded the `response_format` parameter, meaning JSON requests could reach vLLM without guided decoding and produce malformed output. - The team patched the frontend to translate `response_format` into vLLM guided-decoding parameters. ## Deployment and Rollout Strategies - GPU services require longer startup times than CPU services, and model versions may change input/output schemas. - Netflix supports Red-Black deployment: - Runs the new version alongside the old one. - Performs health checks before shifting traffic. - Gradually scales up the new version while scaling down the old one. - Supports atomic rollback if deployment fails. - Red-Black deployment works well when the model interface remains stable. - Production exposed a schema-coordination problem: if a new model changes tensor dimensions or other I/O requirements, upstream callers may send old requests to the new model during the migration window, causing failures. - The post introduces a Versioned strategy as a solution, but the provided text ends before explaining its implementation. Netflix’s experience suggests that successful in-house LLM serving depends as much on compatibility and deployment mechanics as on raw inference speed. A practical platform should standardize on an extensible engine such as vLLM, preserve ecosystem-compatible APIs, tightly control engine versions, retain escape hatches for custom models, and explicitly coordinate model-schema changes during rollout.

gpu-inferencegrpcvllmfastapi+4

gitlabMay 21, 2026

More AI models for GitLab Duo Agent Platform Self-Hosted (opens in new tab)

GitLab 19.0 expands open source model support for Duo Agent Platform Self-Hosted, giving regulated and air-gapped teams more capable AI options without sending source code to external APIs. The update supports selecting different models for different workflows and enables both fully on-premises and hybrid deployments. GitLab’s goal is to reduce the capability gap between isolated environments and cloud-based AI services. ## Challenges for Regulated and Air-Gapped Teams - Data residency, compliance rules, and network isolation often prohibit third-party AI APIs. - Air-gapped environments must run inference locally because they have no internet or external connectivity. - Teams have traditionally faced a trade-off between using an underpowered model and deploying an unnecessarily large model for routine tasks. - These constraints have limited AI productivity gains in highly regulated environments. ## Expanded Open Source Model Support GitLab evaluated models for: - Multi-step tool use - Instruction adherence - Code generation - Reasoning across large diffs and multi-file codebases Newly supported models include: - Mistral Devstral 2 123B - GLM-5.1 - Kimi-K2.6 - MiniMax-M2.7 ## Deployment Options - The recommended setup uses on-premises hardware with vLLM for model serving. - Organizations can also deploy models on GPU-enabled virtual machines in private clouds. - Both approaches keep data within the organization’s controlled environment. - Fully air-gapped teams should use locally hosted models and consult hardware requirements for each model. - Hybrid deployments can combine self-hosted and GitLab-managed models on a per-feature basis. ## Availability and Licensing - Offline-license customers need the GitLab Duo Agent Platform Self-Hosted add-on. - Online-license customers can use usage-based models and combine self-hosted and GitLab-managed models. GitLab recommends choosing models and infrastructure based on network isolation, compliance requirements, hardware availability, and workflow needs. The expanded support makes self-hosted AI a more practical option for organizations that require strict control over their code and data.

gpu-inferencegitlab-duo-agent-platformvllmself-hosted-ai+3

pinterestFeb 2, 2026

Beyond Two Towers: Re-architecting the Serving Stack for Next-Gen Ads Lightweight Ranking Models… (opens in new tab)

Two-Tower models make retrieval and lightweight ranking highly efficient by scoring user and item embeddings with a dot product, but they cannot represent rich user-item interactions or deep feature crossings. This post describes an ads-serving redesign that introduces general-purpose GPU models while preserving end-to-end latency. The main strategy is to reduce data movement, move filtering logic onto the GPU, and optimize inference from an initial 4-second p90 latency to about 20 milliseconds. ## Why Move Beyond Two-Tower Models - Two-Tower architectures independently encode users and items, enabling fast scoring across millions of candidates. - Their decoupled structure limits: - User-item interaction features - Target attention - Early feature crossing - Deep architectures requiring simultaneous access to user and candidate data - More expressive models require GPU-based general-purpose inference rather than specialized dot-product or ANN retrieval. - The existing retrieval stack was not designed to transfer large candidate and feature sets to a GPU, creating a major latency challenge. ## Restructuring the Serving Funnel The traditional funnel consisted of: - Feature expansion for thousands of candidates - Retrieval and Two-Tower lightweight ranking - Heavy ranking and auction processing for the top documents Adding GPU inference directly to this flow would require fetching, serializing, transferring, and returning features for tens of thousands of documents. The authors therefore redesigned the entire early-stage serving pipeline instead of optimizing the model alone. ## Segmenting the Inventory for Feature Fetching Feature retrieval was a major latency source, often taking longer than model inference for workloads ranging from 10,000 to 100,000 documents. - **High-value inventory:** Roughly 1 million documents responsible for a substantial share of revenue have their features embedded in the PyTorch model as registered buffers. - Features become part of the model state, similar to weights. - They remain in GPU high-bandwidth memory. - Requests avoid remote feature-service calls and host-to-device transfers. - The model file must be periodically updated to refresh features. - Future work may include GPU-based caching. - **Long-tail inventory:** The remaining roughly 1 billion documents use a high-performance key-value store with in-host caching. - The post focuses on the first strategy, which is already running in production. ## Moving Business Logic onto the GPU Previously, the model returned scores for approximately 100,000 candidates, while CPU-side code handled utility calculation, filtering, diversity, deduplication, and top-k selection. - The new PyTorch model performs these operations directly: - Combines pCTR, pCVR, bid, and other signals into utility scores. - Applies diversity and filtering rules. - Performs top-k selection. - The GPU returns only the final winners—typically around 1,000 documents—instead of all candidate scores. - This reduces device-to-host data transfer and takes advantage of GPU parallelism. - The approach works because lightweight-ranking business rules are sufficiently simple to express with tensor operations. ## Reducing GPU Inference Latency Initial GPU inference measured roughly 4,000 ms at p90, far too slow for real-time serving. Several systems optimizations reduced this to approximately 20 ms: - **Multiple CUDA streams:** Separate streams for workers allow host-to-device transfers, computation, and device-to-host transfers to overlap. - **Worker alignment:** Worker threads are matched and pinned to physical CPU cores to reduce context switching and lock contention. - **Kernel fusion:** Triton kernels combine operations such as linear layers and activations, reducing memory traffic. - **BF16 computation:** Brain Floating Point 16 lowers memory usage and accelerates arithmetic compared with FP32. - **Profiling tools:** PyTorch Profiler and NVIDIA Nsight Systems were used to identify bottlenecks. ## Practical Recommendation Deploying more expressive ranking models requires rethinking the serving architecture around data movement and execution placement. Embedding frequently used features, executing business logic on the GPU, and applying low-level CUDA and kernel optimizations can make complex neural ranking feasible without increasing end-to-end latency.

gpu-inferencemachine-learningrecommendation-systemspytorch+4