The Evolution of Cassandra Data Movement at Netflix

Netflix replaced its monolithic Cassandra-to-Iceberg connector, Casspactor, with a layered data movement engine built around direct reads from Cassandra backups in Amazon S3. Casspactor handled about 1,200 jobs and 3 PB daily but suffered from fragile metadata dependencies, skewed-partition failures, excessive intermediate tables, and limited support for higher-level data models. The new architecture uses Spark DataFrames and reusable, data-model-aware connectors to improve reliability, scalability, and cost efficiency.

Casspactor’s Role and Limitations

• Casspactor moved Cassandra data into Apache Iceberg using SSTables and metadata stored in S3 backups.
• It supported critical Netflix workloads, including Member, Billing, Recommendations, and Subscriptions.
• Its metadata view depended on several independent systems, each with different failure modes and update schedules.
• Metadata could become inconsistent with actual backups, causing stale or incorrect data to be processed.
• Cassandra maintenance or node replacement could break an entire region’s movement jobs because all nodes had to snapshot at the same clock second.

Constraints for Higher-Level Data Abstractions

• Cassandra-backed abstractions such as Key Value and Time Series inherited Casspactor’s limitations.
• Large or skewed partitions caused executor memory failures and out-of-memory crashes.
• Casspactor had no awareness of application-level data models, forcing downstream connectors to reconstruct them through costly post-processing.
• Multiple intermediate Iceberg and snapshot tables increased storage costs and operational complexity.
• Its backup composition model prevented reliable time travel to earlier backups after topology or keyspace schema changes.
• The monolithic connector could not serve as a reusable foundation for specialized connectors.

Direct S3 Metadata as the Source of Truth

• The new design reads backup metadata directly from the S3 storage layer.
• This removes the chain of external metadata dependencies.
• Backup existence and completeness are determined from the files that actually contain the data.
• Direct backup access also enables restoration of historical backup states.

A Layered Connector Architecture

• The Cassandra Analytics Wrapper builds on open-source Cassandra Analytics and Netflix’s internal backup format.
• It uses an S3 client to read Cassandra backup files and convert them into standard Spark DataFrames.
• A Connector Factory, implemented through Java UDFs and transforms, lets each abstraction define its own optimized connector.
• Key Value, Time Series, and other models can transform generic DataFrames according to their own semantics.
• Improvements to the shared reading engine automatically benefit every connector.

Performance and Operational Improvements

• Mutation compaction and processing run at Spark executor level, allowing better handling of wide and highly skewed partitions.
• Reduced data shuffling helps prevent memory failures on large datasets.
• Direct DataFrame output eliminates costly intermediary Iceberg tables.
• Automatic job sizing adjusts resource usage based on source-table characteristics, reducing manual tuning.
• Fewer dependencies improve reliability and make the system easier to maintain.

Netflix’s new engine provides a shared, backup-native foundation while keeping data-model-specific logic in separate connectors. This approach is better suited to expanding Cassandra abstractions and large-scale data movement than maintaining another monolithic connector.