In-House LLM Serving at Netflix (opens in new tab)

Netflix built an in-house LLM serving platform within its existing production ML infrastructure rather than creating a separate ML stack. The platform combines a JVM-based serving layer, NVIDIA Triton, GPU-backed Model Scoring Service, and an OpenAI-compatible HTTP frontend. Its main design choices—vLLM, model packaging, API compatibility, and deployment strategy—prioritize operational flexibility and seamless movement from hosted models to self-hosted ones, while production exposed versioning and compatibility risks.

Architecture and Serving Model

• Netflix’s unified JVM serving system handles routing, A/B testing, feature retrieval, inference, post-processing, and logging.
• Callers access models through:
  • A gRPC path integrated with the existing serving system.
  • A direct HTTP path for newer LLM applications.
• Small CPU models run in-process to avoid remote-call overhead.
• Larger GPU models run through Model Scoring Service (MSS), which supports XGBoost, TensorFlow, PyTorch, and LLMs.
• NVIDIA Triton manages model loading, batching, and GPU scheduling.
• A Java control plane provides deployment, versioning, health checks, autoscaling, and multi-region rollout.

Choosing vLLM as the Standard Engine

• Netflix originally used TensorRT-LLM, but re-evaluated its choice as open-source engines improved and workloads diversified.
• vLLM was selected based on operational fit rather than benchmark performance alone:
  • Supports custom model architectures without lengthy compilation.
  • Provides hooks for custom decoding and constraint logic.
  • Is easier to debug than earlier compiled-engine workflows.
  • Is familiar to many researchers, reducing the research-to-production transition cost.
• The workload includes embeddings, prefill-only inference, autoregressive decoding, and custom per-step decoding constraints.

Triton Integration and Model Packaging

• Triton offers both a Python backend and a dedicated vLLM backend.
• The Python backend requires explicit input and output tensor definitions, coupling packaged artifacts to frontend changes.
• The vLLM backend uses a JSON configuration pointing to model weights and tokenizers, generating tensor specifications dynamically.
• Netflix considers the vLLM backend the preferred default because models and frontends can evolve independently.
• Production revealed two limitations:
  • Triton and vLLM must be version-pinned because incompatible APIs can prevent the backend from loading entirely.
  • Models requiring custom preprocessing, postprocessing, tokenization, or ensemble execution still need Triton’s Python backend.

OpenAI-Compatible HTTP Frontend

• Netflix keeps LLMs compatible with the same internal gRPC model-serving interface used by other model types.
• It also exposes an OpenAI-compatible API because that interface is widely supported by inference engines, orchestration tools, evaluation systems, and client libraries.
• This makes replacing a hosted model with a fine-tuned self-hosted model largely transparent to callers.
• The implementation uses Triton’s OpenAI-compatible frontend, FastAPI, and a TritonLLMEngine that translates requests into Triton inference calls.
• KServe HTTP and gRPC frontends remain available for the Java control plane.
• Netflix found that Triton’s frontend silently discarded the response_format parameter, meaning JSON requests could reach vLLM without guided decoding and produce malformed output.
• The team patched the frontend to translate response_format into vLLM guided-decoding parameters.

Deployment and Rollout Strategies

• GPU services require longer startup times than CPU services, and model versions may change input/output schemas.
• Netflix supports Red-Black deployment:
  • Runs the new version alongside the old one.
  • Performs health checks before shifting traffic.
  • Gradually scales up the new version while scaling down the old one.
  • Supports atomic rollback if deployment fails.
• Red-Black deployment works well when the model interface remains stable.
• Production exposed a schema-coordination problem: if a new model changes tensor dimensions or other I/O requirements, upstream callers may send old requests to the new model during the migration window, causing failures.
• The post introduces a Versioned strategy as a solution, but the provided text ends before explaining its implementation.

Netflix’s experience suggests that successful in-house LLM serving depends as much on compatibility and deployment mechanics as on raw inference speed. A practical platform should standardize on an extensible engine such as vLLM, preserve ecosystem-compatible APIs, tightly control engine versions, retain escape hatches for custom models, and explicitly coordinate model-schema changes during rollout.