naver·Nov 23, 2025Original article

Naver TV (opens in new tab)

Naver Webtoon developed "Flow.er," an on-demand data lineage pipeline service designed to overcome the operational inefficiencies and high maintenance costs of legacy data workflows. By integrating dbt for modular modeling and Airflow for scalable orchestration, the platform automates complex backfill and recovery processes while maintaining high data integrity. This shift to a lineage-centric architecture allows the engineering team to manage data as a high-quality product rather than a series of disconnected tasks. ### Challenges in Traditional Data Pipelines * High operational burdens were caused by manual backfilling and recovery tasks, which became increasingly difficult as data volume and complexity grew. * Legacy systems lacked transparency in data dependencies, making it hard to predict the downstream impact of code changes or upstream data failures. * Fragmented development environments led to inconsistencies between local testing and production outputs, slowing down the deployment of new data products. ### Core Architecture and the Role of dbt and Airflow * dbt serves as the central modeling layer, defining transformations and establishing clear data lineage that maps how information flows between tables. * Airflow functions as the orchestration engine, utilizing the lineage defined in dbt to trigger tasks in the correct order and manage execution schedules. * Individual development instances provide engineers with isolated environments to test dbt models, ensuring that logic is validated before being merged into the main pipeline. * The system includes a dedicated model management page and a robust CI/CD pipeline to streamline the transition from development to production. ### Expanding the Platform with Tower and Playground * "Tower" and "Playground" were introduced as supplementary components to support a broader range of data organizations and facilitate easier experimentation. * A specialized Partition Checker was developed to enhance data integrity by automatically verifying that all required data partitions are present before downstream processing begins. * Improvements to the Manager DAG system allow the platform to handle large-scale pipeline deployments across different teams while maintaining a unified view of the data lineage. ### Future Evolution with AI and MCP * The team is exploring the integration of Model Context Protocol (MCP) servers to bridge the gap between data pipelines and AI applications. * Future developments focus on utilizing AI agents to further automate pipeline monitoring and troubleshooting, reducing the need for human intervention in routine maintenance. To build a sustainable and scalable data infrastructure, organizations should transition from simple task scheduling to a lineage-aware architecture. Adopting a framework like Flow.er, which combines the modeling strengths of dbt with the orchestration power of Airflow, enables teams to automate the most labor-intensive parts of data engineering—such as backfills and dependency management—while ensuring the reliability of the final data product.

SparkCI/CDData PipelineData Engineering+4

datadog·Apr 2, 2019·3 min readCurated summary

Building highly reliable data pipelines at Datadog

Datadog’s approach to reliable data pipelines focuses on delivering correct data on time, even when individual jobs fail. Reliability therefore requires fault tolerance, monitoring, and fast recovery rather than eliminating every failure. The company achieves this through isolated, short-lived clusters and pipelines designed to limit the impact of failures. ## Reliability Means Timely, Correct Results - A reliable pipeline is one that consistently produces correct outputs within the required time window. - Occasional crashes do not necessarily make a pipeline unreliable if automatic recovery still delivers the data on schedule. - Pipelines should be designed with the expectation that failures will eventually occur. - Monitoring must detect unexpected failures early, while operational processes should support rapid recovery. ## Architecture for Batch Pipelines - Datadog streams and analyzes live data in real time but uses batch pipelines for features such as optimized long-term storage. - Historical data is stored in object storage. - Cloud Hadoop/Spark services launch and configure processing clusters. - Luigi workers manage tasks and workflows, while Spark workers compile code and submit jobs. - Jobs can be launched through a web interface, command line, or scheduler. ## One Cluster per Pipeline Instead of placing all workloads on one large Hadoop cluster, Datadog gives each pipeline its own cluster. - **Isolation:** Jobs do not compete for resources or interfere with one another, simplifying monitoring and diagnosis. - **Workload-specific hardware:** Clusters can use CPU-optimized or memory-optimized instances depending on the job. - **Elastic scaling:** Clusters can be expanded to catch up with delays or handle growing data volumes without waiting for a shared cluster. - **Safer upgrades:** Hadoop and Spark versions can be upgraded gradually across separate clusters. - Clusters are typically short-lived, averaging about three hours, although dozens may run simultaneously. ## Using Spot Instances to Encourage Fault Tolerance - AWS spot instances can reduce infrastructure costs by as much as 80%, but their nodes may be terminated whenever capacity or demand changes. - Rather than avoiding this failure mode, Datadog designs pipelines to tolerate disappearing clusters. - Long-running jobs are risky because failures discard more work and make recovery slower. - Pipelines are split into smaller jobs: - **Vertically:** Separate transformations into multiple stages, persisting intermediate results in S3. - **Horizontally:** Partition input data so multiple jobs process different portions concurrently. ## Breaking Up the Rollup Pipeline - Datadog’s rollup pipeline generates aggregated time-series data for historical metrics queries. - A single job would take more than 14 hours, making failures costly and difficult to recover from. - The pipeline is divided into two stages: - Aggregate high-resolution data and checkpoint it to S3 as Parquet files. - Convert the intermediate data into a custom format optimized for queries. - As these jobs grew, they were partitioned further using Kafka’s partitioning scheme. - Kafka partitions are grouped into shards, allowing Datadog to: - Adjust how much data each job processes. - Run more or fewer jobs as needed. - Isolate unusually large or sensitive shards. - This decomposition adds overhead because launching jobs and checkpointing to S3 take extra time, but it substantially limits the work lost during failures. ## Practical Recommendation Design pipelines around failure rather than assuming uninterrupted execution. Use isolated, scalable clusters, short jobs, intermediate checkpoints, and partitioned processing so that failures affect only a small portion of the workload and recovery remains fast.

Read original(opens in new tab)

SparkDatabase DesignPostgreSQLAWS+4