LY Corporation's Cloud Infrastructure Reorganization: Introducing the Architecture of Flava, a Next-Generation Platform Integrating Two Massive Clouds
LY Corporation is consolidating the former LINE “Verda” and Yahoo Japan “YNW” private clouds into Flava, a next-generation platform designed for large-scale, uninterrupted operations. Its approach assumes failures will occur, prioritizing stateless services, application-led availability, rapid IaC-based recovery, and extensive automation. Flava also restructures the architecture around shared resources, upstream OpenStack, default VPC networking, and user-driven cost optimization. ## Failure-Aware Design and Operations - VM root disks are treated as temporary; persistent data is placed in external storage so instance failures have limited service impact. - Availability is achieved through cooperation between infrastructure and applications rather than excessive infrastructure-side guarantees. - Recovery focuses on maintaining service continuity, rebuilding environments quickly with infrastructure as code, and avoiding lengthy root-cause investigations during incidents. - The company promotes KaaS and PaaS to help developers build resilient services without managing low-level infrastructure. - OS configuration, package installation, networking, and other changes are managed as code through CI/CD. - Deployments are performed by availability zone to limit the blast radius of failures. ## Observability from Fleet-Wide Trends to Root Causes - Prometheus, Grafana, and custom dashboards monitor overall cloud health and long-term trends. - When anomalies appear, engineers investigate at a deeper level using kernel traces, packet captures, and other low-level diagnostics. - This combination of broad monitoring and detailed investigation allows teams to move between “forest” and “tree” perspectives. - The operational model depends not only on tools but also on engineers capable of tracing problems down to their fundamental causes. ## OSS, Software-Defined Infrastructure, and Custom Development - The platform relies heavily on OpenStack, Envoy, Linux kernel technologies such as eBPF/XDP, FRR, and Ceph. - LY contributes patches and new capabilities upstream instead of maintaining long-lived private forks. - It has developed SRv6 BGP functionality required for Flava’s VPCs and contributed related work to FRRouting and the Linux kernel. - Compute, VPC, DNS, and load-balancing services run primarily on commodity x86 servers rather than specialized appliances. - XDP-based data planes, hardware offload, and system tuning are used to achieve near-wire-speed throughput and low latency. - Where OSS cannot meet internal requirements, LY builds systems from scratch, including the Dragon object store, SDN control-plane components, load-balancer health agents, and service discovery tools written in Rust, Go, and Python. ## Autonomous Hardware Operations - With tens of thousands of hypervisors and petabyte-scale storage, hardware failures occur continuously. - Failure detection, requests to data-center technicians, hardware replacement, and cluster reintegration are largely automated. - Some exceptional cases still require engineers, but LY plans to use LLMs to automate more of these operational tasks. ## Flava’s Architectural Improvements ### Shared Resource Pools - Older clouds used many dedicated clusters and resource pools, making capacity planning complex and reducing utilization. - Flava consolidates most products and services into one large shared resource pool. - This reduces planning variables, improves resource efficiency, and accelerates provisioning. ### Upstream-Compatible OpenStack - Excessive customization in the legacy environment made upgrades difficult. - Flava minimizes private patches, follows upstream OpenStack, and contributes necessary improvements back to the project. - This enables regular upgrade cycles and keeps security fixes and features current. ### VPC by Default - VPC networking is the standard security model for multi-tenant workloads. - Logical isolation replaces many cases where dedicated VLANs or firewalls previously required months of preparation. - Equivalent security environments can now be provisioned in minutes. - The VPC data plane is being redesigned with XDP to support the reliability and performance required at company-wide scale. ### Built-In Cost Optimization - Development environments require resource lifetimes, allowing unused “zombie” resources to be deleted automatically. - Object storage offers bucket classes such as “High Performance” and “Scalable.” - Users can change storage classes without changing endpoints, adapting cost and performance as access patterns evolve. ## Remaining Challenges - Flava currently offers only a limited set of products and must expand its capabilities while addressing post-launch bugs and overlooked requirements. - The largest challenge is migrating users from the legacy platforms. - LY is working to provide transparent migration tools and reduce manual effort while shortening the period of duplicate investment in old and new infrastructure. ## Team and Engineering Culture - The team includes specialists ranging from kernel developers to web-front-end engineers. - Engineers are expected to understand and control infrastructure rather than treat it as a black box. - Deep source-level expertise enables upstream OSS contributions and informed negotiations with commercial vendors. - This culture of ownership and technical control is presented as a core reason the platform can evolve at LY’s scale. LY’s experience demonstrates that large private clouds can combine OSS, custom software, commodity hardware, and rigorous automation effectively. The practical recommendation is to design for failure, keep infrastructure reproducible through IaC, contribute changes upstream where possible, and use custom development selectively for requirements that general-purpose platforms cannot satisfy.
Read original(opens in new tab)