Scaling to Infinity: LY Corporation’s

LY Corporation’s observability team evolved its time-series database to handle rapidly growing infrastructure and Kubernetes workloads. After outgrowing MySQL and OpenTSDB, the team built an engine optimized for high-cardinality metrics, low-latency queries, and seamless API compatibility. Its architecture now combines in-memory, Cassandra, and S3-compatible storage, enabling cost-efficient scaling while supporting trillions of daily metrics.

Why Time-Series Storage Matters

• Metrics record system state as timestamped numerical values.
• They support dashboards, threshold-based alerts, and predictive analysis using tools such as ARIMA and Prophet.
• Even a small metric record can consume about 280 bytes when timestamps, values, and tags are included.
• One CPU metric collected every 15 seconds requires roughly 562 MiB per server annually; across 1,000 servers, this grows to about 548 GiB before adding memory, disk, and network metrics.
• High-cardinality cloud environments make both storage cost and query latency critical operational concerns.

Moving Beyond MySQL and OpenTSDB

• MySQL initially became inadequate as the organization moved from SOA to MSA:
  • Write load increased sharply.
  • Storage costs and capacity requirements grew.
  • Query latency worsened for large datasets.
  • Rigid schemas could not easily represent changing cloud resources.
• MySQL sharding provided temporary relief but could not support high-resolution metrics collected at intervals under one minute.
• OpenTSDB, introduced in 2016 on Apache HBase, improved write performance but had important limitations:
  • Tag growth harmed UID-table lookup performance.
  • Metadata was restricted to a narrow character set.
  • Large queries required cache warm-up procedures.
• These constraints led to the development of an internal database beginning in 2018.

Building the Internal Time-Series Database

• The 2019 engine was designed around:
  • Flexible protocol support independent of a particular agent.
  • Linear scalability without downtime.
  • Low-latency processing of high-resolution metrics.
  • Strong availability during failures.
• Inspired by Meta’s Gorilla research, the team used access patterns in which most queries target recent data.
• Frequently accessed metrics were kept in an in-memory database, while colder data was stored in Apache Cassandra.
• The new engine enabled metric volumes to grow by more than 200 billion records annually while preserving existing APIs.
• Users benefited from the new backend without migration work or code changes.

Scaling for Kubernetes Workloads

• Kubernetes introduced rapidly changing pods, dynamically allocated volumes, and much higher metric churn.
• Both major storage layers encountered scaling problems:
  • IMDB initially required adding identical hardware, limiting expansion options.
  • Cassandra rebalancing could take tens of hours because of its data volume.
• The team improved IMDB with weighted load balancing so nodes with different capacities could be used effectively.
• Storage was divided into tiers:
  • Recent 14-day data remained in Cassandra for high-performance access.
  • Older data was moved to S3-compatible storage.
• This reduced Cassandra dependency, lowered costs, simplified operations, and enabled more flexible hardware and Kubernetes-based deployment.

Writing and Reading Through S3

• The write path separates data processing from long-term storage:
  • A Dumper reads metric slots from IMDB.
  • It converts them into internally defined sub-blocks.
  • A Block Dumper combines sub-blocks into blocks and writes them to S3.
• A Storage Gateway reads the blocks for queries and caches them on local disks.
• Disk caching initially caused excessive page-cache use and rapid memory exhaustion.
• Direct I/O was considered but withdrawn after the cloud storage team warned that it consumed too much shared bandwidth.
• Through cross-team collaboration, the team adopted a B+ tree-based cache that made better use of the kernel page cache without overloading infrastructure.

Future Direction: From Storage to Intelligence

• The team aims to move beyond recording metrics toward prediction and AI-assisted operations.
• Achieving this requires consolidating time-series data currently scattered across internal systems.
• A key requirement is to perform this integration without imposing migration work or breaking changes on users.
• The broader goal is an observability platform that turns unified metrics into predictive and intelligent operational capabilities.

The main recommendation is to design time-series platforms around real access patterns, tier storage according to data age, and preserve compatibility while evolving the backend. At extreme scale, careful storage architecture and collaboration across infrastructure teams are as important as raw database performance.