From Hive to Iceberg: The Secret to 12x Faster Data Reflection
LINE Plus replaced a full-dump ETL pipeline for product data with incremental processing using Apache Iceberg and Apache Flink. The previous HBase/Hive workflow rewrote hundreds of millions of rows for every update, causing high compute costs and delays that left data up to an hour out of date. With the new architecture, update intervals were reduced from 60 minutes to 5 minutes—roughly a 12× improvement—while preserving consistency and fault tolerance. ## Limitations of Full-Data ETL - The existing HBase and Hive pipeline continuously collected CDC data in HDFS but had to merge it with existing data and rewrite the entire table before changes became queryable. - This caused: - High compute and storage costs - Dependence on limited shared Hadoop resources - Delayed updates and stale data - Snapshot-based extraction provides consistency, but large snapshots can take hours and retain old versions through MVCC, increasing system overhead. - Processing only the changed rows would reduce the workload from hundreds of millions of records to tens of thousands, separating update cost from total dataset size. ## Introducing Apache Iceberg - Iceberg manages data through metadata and table snapshots rather than relying solely on directory structures like traditional Hive tables. - It supports row-level `upsert` and `delete` operations. - This allows incremental changes to be written without rewriting the entire table, making much shorter ETL intervals possible. ## Requirements for the Streaming Pipeline The team evaluated Spark and Flink against three essential requirements: - **Data freshness:** Late-arriving compensation or replay data must not overwrite newer records. - **End-to-end exactly-once processing:** Iceberg updates and Kafka status messages must not partially succeed. - **Fault tolerance and state management:** Processing state must survive failures and restarts. A Kafka message indicating that all CDC data through a specific timestamp—such as 13:03—has been applied serves as the signal that a bulk extraction can safely begin. This requires complete confidence that the message accurately represents the Iceberg table’s committed state. ## Why Two-Phase Commit Was Necessary - Iceberg and Kafka are independent systems, so writing to one while failing to write to the other could create inconsistent state. - Two-phase commit (2PC) prevents partial success: - Both systems prepare their writes. - They commit only when all required operations succeed. - Any failure causes the operation to roll back. - Exactly-once processing also prevents duplicate or missing records during retries, network failures, or node restarts. - Together, these guarantees make Kafka status messages a reliable representation of the Iceberg table’s state. ## Choosing Flink over Spark - Spark Structured Streaming uses a micro-batch model, which makes fine-grained event-time and state control more difficult. - Flink provides native event-by-event streaming and better support for the required consistency model. - The team used Flink state to track each record’s `updatedate`: - Older late-arriving events are ignored. - Replayed historical data cannot overwrite newer values. - Flink checkpoints: - Persist streaming state externally. - Enable recovery from the latest consistent point. - Integrate with the Kafka sink’s 2PC mechanism. - Kafka messages remain in a pre-commit state until the Iceberg write and checkpoint both succeed. ## Kubernetes Deployment Options - The team compared: - **Native Kubernetes:** Requires manually configuring roles, service accounts, services, routing, deployments, slots, and jobs. - **Flink Kubernetes Operator:** Represents Flink infrastructure and jobs as custom resources, automating configuration such as routing and the web UI through Helm values. - Although Flink has greater operational complexity and a steeper learning curve than Spark, it was selected because it was the only option that satisfied all three core requirements at the engine level. The recommended architecture is an incremental Iceberg pipeline powered by Flink, with stateful processing, checkpoints, and two-phase commit between Iceberg and Kafka. This approach keeps data current, avoids expensive full-table rewrites, and provides reliable recovery and consistency at a five-minute update interval.
Read original(opens in new tab)
