Apache Cassandra | Techlist.io

netflixJun 19, 20263 min readCurated summary

The Evolution of Cassandra Data Movement at Netflix

Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency. ## Casspactor’s Role and Limitations - Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups. - It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions. - Its metadata view depended on several independent systems, each with different failure modes and update schedules. - Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed. - Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second. ## Constraints for Higher-Level Data Abstractions - Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations. - Large or skewed partitions caused executor memory failures and out-of-memory crashes. - Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing. - Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity. - Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes. - The monolithic connector could not serve as a reusable foundation for specialized connectors. ## Direct S3 Metadata as the Source of Truth - The new design reads backup metadata directly from the S3 storage layer. - This removes the chain of external metadata dependencies. - Backup existence and completeness are determined from the files that actually contain the data. - Direct backup access also enables restoration of historical backup states. ## A Layered Connector Architecture - The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format. - It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames. - A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector. - Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics. - Improvements to the shared reading engine automatically benefit every connector. ## Performance and Operational Improvements - Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions. - Reduced data shuffling helps prevent memory failures on large datasets. - Direct DataFrame output eliminates costly intermediary Iceberg tables. - Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning. - Fewer dependencies improve reliability and make the system easier to maintain. Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.

Read original

Apache Cassandra Amazon S3 Apache Spark Apache Iceberg+3

netflixJun 3, 20264 min readCurated summary

Dynamic Repartitioning for Time Series Workloads

Netflix’s TimeSeries Abstraction uses Cassandra to ingest and query petabytes of temporal data with millisecond-scale latency, but growing partitions can cause seconds-long reads, timeouts, and resource exhaustion. Its initial time-based partitioning works well when workload estimates are accurate, yet traffic changes and outlier IDs can make partitions too large or too small. Netflix therefore developed automated time-slice repartitioning and, for isolated hot IDs, asynchronous dynamic partitioning at the individual-ID level. ## Cassandra and the Wide-Partition Problem - Cassandra provides: - High-throughput, low-latency reads and writes - Cost-effective operation at scale - Strong operational familiarity within Netflix - TimeSeries datasets accumulate events over time, creating potentially very wide partitions. - Wide partitions can lead to: - Read latencies increasing from milliseconds to seconds - Request timeouts - Garbage-collection pauses - High CPU utilization and thread queueing - Scaling Cassandra clusters can help, but Netflix sought more targeted solutions. ## Initial Time-Based Partitioning - TimeSeries divides data into discrete time slices to keep partitions manageable. - This structure also makes it efficient to: - Query data by time - Drop old data without creating large tombstone problems - At dataset creation, users provide expected workload characteristics. - Netflix’s provisioning pipeline uses those inputs, along with Monte Carlo simulations, to select infrastructure and partition settings. ## Why Static Provisioning Falls Short - Workloads may be unknown or inaccurately estimated during initial provisioning. - Traffic patterns, client behavior, and product needs can change over time. - A small number of TimeSeries IDs may generate far more events than the rest. - Time slices provide a way to change partitioning for future data, but manually updating thousands of datasets is impractical. ## Repartitioning Entire Time Slices - Cassandra introspection tools, such as `nodetool tablehistograms`, expose partition-size distributions. - Netflix added a background worker that: - Monitors partition histograms for time slices - Publishes observations through a Cassandra virtual table - Detects partitions that are too large or too small - Calculates a new partitioning adjustment factor - Target partition density is typically between 2 MiB and 10 MiB, depending on workload. - The worker updates the strategy for future time slices. For example, it may expand a `time_bucket` interval from 60 seconds to 604,800 seconds when partitions are too small. - This approach reduced read latency and timeouts caused by thread queueing. - Its limitation is that it changes partitioning broadly and is ineffective when only a minority of IDs produce oversized partitions. ## Handling Isolated Problem IDs Netflix considers several responses when only some IDs are problematic: - **Do nothing:** Appropriate when wide partitions do not affect application-level metrics. - **Partial returns:** Abort a request after it exceeds a latency SLO while returning data already collected; useful when latency matters more than completeness. - **Block IDs:** Prevent exceptionally bad test, spam, or otherwise harmful IDs from destabilizing the system. - These options are inadequate when valid, important IDs must return all their data despite generating large partitions. ## Dynamic Partitioning per ID Dynamic partitioning addresses outliers by splitting partitions for individual TimeSeries IDs rather than modifying an entire table. The asynchronous pipeline has three stages: - **Detection:** The read path identifies partitions that exceed a configured size threshold. - **Planning and splitting:** The system asynchronously plans and executes splits into appropriately sized partitions. - **Serving reads:** Once splits are available, read requests are transparently rerouted to them. During each read, the server tracks the bytes retrieved for a partition. If usage exceeds the threshold, it emits a detection event to Kafka containing information such as: - The Cassandra time-slice table - The affected TimeSeries ID - The existing time and event bucket - Whether the partition is immutable - A version identifier ## Practical Recommendation Use whole-time-slice repartitioning when an entire dataset is systematically over- or under-partitioned. For isolated but important high-volume IDs, dynamic per-ID partitioning provides a more precise way to control latency without disrupting the rest of the dataset.

Read original

Apache Cassandra Database Design Apache Kafka Time Series Database+1

netflixApr 4, 20263 min readCurated summary

Powering Multimodal Intelligence for Video Search

Video search is difficult because it must combine many kinds of information—characters, scenes, dialogue, labels, and embeddings—across enormous volumes of footage. The post argues that solving this problem requires a distributed pipeline that separates reliable ingestion, computationally intensive data fusion, and low-latency search indexing. Temporal bucketing, hybrid ranking, and deduplication turn billions of model outputs into searchable moments for editors. ## Why Video Search Is Complex - Video contains multiple overlapping modalities, each analyzed by specialized models. - Models produce different outputs, including: - Text labels such as characters or objects - Scene classifications - High-dimensional embedding vectors - Time ranges with varying boundaries - Overlapping model timelines must be synchronized into a chronological representation. - A 2,000-hour archive may contain more than 216 million frames, expanding to billions of records after multimodal processing. - Search must avoid returning thousands of redundant clips from continuous shots. - Ranking therefore combines: - Symbolic text matching for precision and interpretability - Semantic vector similarity for contextual relevance - Clustering and deduplication to identify the best moments - Sub-second response times are essential because delays interrupt editors’ creative workflows. ## Three-Stage Ingestion and Fusion Pipeline ### Transactional Persistence - Raw model annotations are ingested through highly available pipelines. - Apache Cassandra stores the annotations with an emphasis on: - Data integrity - Distributed availability - High write throughput - An annotation can include a type, nanosecond time range, embedding vector, label, and confidence score. ### Offline Data Fusion - After persistence, Apache Kafka publishes an event that starts asynchronous processing. - The offline pipeline performs expensive temporal intersections without slowing ingestion or search. - Model outputs are normalized into fixed one-second time buckets. - The fusion process: - Maps continuous detections into discrete intervals - Intersects annotations sharing a bucket - Combines them into unified records - Writes the enriched records back to Cassandra - For example, a “Joey” character detection from seconds 2–8 can be combined with a “kitchen” scene detection from seconds 4–9 to create a fused record for the 4–5 second interval. - Each fused record retains links to the original annotations and source asset. ### Real-Time Search Indexing - Enriched buckets are later sent from Cassandra to Elasticsearch. - Upserts use a composite key consisting of the asset ID and time bucket. - If a bucket already exists, it is updated rather than duplicated. - This creates one consistent record for each second of footage while allowing new model results to be incorporated. The overall recommendation is to treat multimodal video search as a distributed data-fusion problem rather than a single-model retrieval task. Decoupling ingestion, offline processing, and indexing allows the system to handle massive archives while preserving reliable data capture and fast, context-rich search.

Read original

Apache Cassandra Database Design Apache Kafka Elasticsearch+4