In-House LLM Serving at Netflix (opens in new tab)
Netflix built an in-house LLM serving platform within its existing production ML infrastructure rather than creating a separate ML stack. The platform combines a JVM-based serving layer, NVIDIA Triton, GPU-backed Model Scoring Service, and an OpenAI-compatible HTTP frontend. Its main design choices—vLLM, model packaging, API compatibility, and deployment strategy—prioritize operational flexibility and seamless movement from hosted models to self-hosted ones, while production exposed versioning and compatibility risks. ## Architecture and Serving Model - Netflix’s unified JVM serving system handles routing, A/B testing, feature retrieval, inference, post-processing, and logging. - Callers access models through: - A gRPC path integrated with the existing serving system. - A direct HTTP path for newer LLM applications. - Small CPU models run in-process to avoid remote-call overhead. - Larger GPU models run through Model Scoring Service (MSS), which supports XGBoost, TensorFlow, PyTorch, and LLMs. - NVIDIA Triton manages model loading, batching, and GPU scheduling. - A Java control plane provides deployment, versioning, health checks, autoscaling, and multi-region rollout. ## Choosing vLLM as the Standard Engine - Netflix originally used TensorRT-LLM, but re-evaluated its choice as open-source engines improved and workloads diversified. - vLLM was selected based on operational fit rather than benchmark performance alone: - Supports custom model architectures without lengthy compilation. - Provides hooks for custom decoding and constraint logic. - Is easier to debug than earlier compiled-engine workflows. - Is familiar to many researchers, reducing the research-to-production transition cost. - The workload includes embeddings, prefill-only inference, autoregressive decoding, and custom per-step decoding constraints. ## Triton Integration and Model Packaging - Triton offers both a Python backend and a dedicated vLLM backend. - The Python backend requires explicit input and output tensor definitions, coupling packaged artifacts to frontend changes. - The vLLM backend uses a JSON configuration pointing to model weights and tokenizers, generating tensor specifications dynamically. - Netflix considers the vLLM backend the preferred default because models and frontends can evolve independently. - Production revealed two limitations: - Triton and vLLM must be version-pinned because incompatible APIs can prevent the backend from loading entirely. - Models requiring custom preprocessing, postprocessing, tokenization, or ensemble execution still need Triton’s Python backend. ## OpenAI-Compatible HTTP Frontend - Netflix keeps LLMs compatible with the same internal gRPC model-serving interface used by other model types. - It also exposes an OpenAI-compatible API because that interface is widely supported by inference engines, orchestration tools, evaluation systems, and client libraries. - This makes replacing a hosted model with a fine-tuned self-hosted model largely transparent to callers. - The implementation uses Triton’s OpenAI-compatible frontend, FastAPI, and a `TritonLLMEngine` that translates requests into Triton inference calls. - KServe HTTP and gRPC frontends remain available for the Java control plane. - Netflix found that Triton’s frontend silently discarded the `response_format` parameter, meaning JSON requests could reach vLLM without guided decoding and produce malformed output. - The team patched the frontend to translate `response_format` into vLLM guided-decoding parameters. ## Deployment and Rollout Strategies - GPU services require longer startup times than CPU services, and model versions may change input/output schemas. - Netflix supports Red-Black deployment: - Runs the new version alongside the old one. - Performs health checks before shifting traffic. - Gradually scales up the new version while scaling down the old one. - Supports atomic rollback if deployment fails. - Red-Black deployment works well when the model interface remains stable. - Production exposed a schema-coordination problem: if a new model changes tensor dimensions or other I/O requirements, upstream callers may send old requests to the new model during the migration window, causing failures. - The post introduces a Versioned strategy as a solution, but the provided text ends before explaining its implementation. Netflix’s experience suggests that successful in-house LLM serving depends as much on compatibility and deployment mechanics as on raw inference speed. A practical platform should standardize on an extensible engine such as vLLM, preserve ecosystem-compatible APIs, tightly control engine versions, retain escape hatches for custom models, and explicitly coordinate model-schema changes during rollout.