Dynamic Repartitioning for Time Series Workloads
Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level. ## Cassandra and the Wide-Partition Problem - Cassandra provides: - High-throughput, low-latency reads and writes - Cost-effective operation at scale - Strong operational familiarity within Netflix - TimeSeries datasets accumulate events over time, creating potentially very wide partitions. - Wide partitions can lead to: - Read latencies increasing from milliseconds to seconds - Request timeouts - Garbage-collection pauses - High CPU utilization and thread queueing - Scaling Cassandra clusters can help, but Netflix sought more targeted solutions. ## Initial Time-Based Partitioning - TimeSeries divides data into discrete time slices to keep partitions manageable. - This structure also makes it efficient to: - Query data by time - Drop old data without creating large tombstone problems - At dataset creation, users provide expected workload characteristics. - Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings. ## Why Static Provisioning Falls Short - Workloads may be unknown or inaccurately estimated during initial provisioning. - Traffic patterns, client behavior, and product needs can change over time. - A small number of TimeSeries IDs may generate far more events than the rest. - Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical. ## Repartitioning Entire Time Slices - Cassandra introspection tools, such as `nodetool tablehistograms`, expose partition-size distributions. - Netflix added a background worker that: - Monitors partition histograms for time slices - Publishes observations through a Cassandra virtual table - Detects partitions that are too large or too small - Calculates a new partitioning adjustment factor - Target partition density is typically between 2 MiB and 10 MiB, depending on workload. - The worker updates the strategy for future time slices. For example, it may expand a `time_bucket` interval from 60 seconds to 604,800 seconds when partitions are too small. - This approach reduced read latency and timeouts caused by thread queueing. - Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions. ## Handling Isolated Problem IDs Netflix considers several responses when only some IDs are problematic: - **Do nothing:** Appropriate when wide partitions do not affect application-level metrics. - **Partial returns:** Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness. - **Block IDs:** Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system. - These options are inadequate when valid, important IDs must return all their data despite generating large partitions. ## Dynamic Partitioning per ID Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table. The asynchronous pipeline has three stages: - **Detection:** The read path identifies partitions that exceed a configured size threshold. - **Planning and splitting:** The system asynchronously plans and executes splits into appropriately sized partitions. - **Serving reads:** Once splits are available, read requests are transparently rerouted to them. During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as: - The Cassandra time-slice table - The affected TimeSeries ID - The existing time and event bucket - Whether the partition is immutable - A version identifier ## Practical Recommendation Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.
Read original(opens in new tab)