Data Ingestion | Techlist.io

metaMay 12, 20263 min readCurated summary

Migrating Data Ingestion Systems at Meta Scale

Meta rebuilt its hyperscale MySQL data ingestion system to improve reliability, efficiency, and data-langing latency. The migration moved workloads from customer-owned pipelines to a simpler, self-managed warehouse service and ultimately transitioned 100% of jobs. Success depended on staged validation, continuous data comparison, and fast rollback mechanisms. ## Why Meta Migrated - The system incrementally moved several petabytes of social graph data from MySQL into Meta’s data warehouse each day. - This data supports analytics, reporting, machine learning, and product development. - The legacy architecture became increasingly unstable as data-landing requirements grew stricter. - Customer-owned pipelines worked at smaller scales but became difficult to manage reliably at hyperscale. ## Migration Success Criteria Each job had to meet defined requirements before advancing: - **Data correctness:** Old and new systems had matching row counts and checksums. - **Landing latency:** The new system performed at least as well as the legacy system. - **Resource usage:** Compute and storage consumption did not regress. - **Critical-table requirements:** Additional criteria were agreed upon with dependent teams. ## Three-Phase Migration Lifecycle ### Shadow Phase - New-system shadow jobs ran against the same production sources as existing jobs. - Their output was written to separate shadow tables. - Row counts and checksums were continuously compared with production data. - Compute and storage requirements were measured before production rollout. - Once validated in pre-production, shadow jobs were tested in production. ### Reverse Shadow Phase - The new system began writing to the production table. - The legacy system continued running, but wrote to a shadow table. - This preserved continuous comparison between both systems. - If discrepancies appeared, Meta could quickly restore the old system without rebuilding its configuration. ### Migration Cleanup - Both systems continued to be monitored for mismatches. - After validation, the legacy shadow job was removed. - The new system became the sole production pipeline. ## Data Quality and Debugging Tooling - Meta built tooling to compare corresponding table partitions from the two systems. - Comparisons included row counts, checksums, and example rows responsible for mismatches. - Mismatch records and debugging details were logged to Scuba for real-time analysis. - Hourly queries helped engineers identify root causes and determine whether issues were already known. - The same tooling remains part of post-migration release validation. ## Rollout and Rollback Controls - Both systems used change data capture (CDC), with internal full-dump and delta tables feeding customer-facing target tables. - Because CDC builds new data from previously landed data, an existing defect could propagate after migration. - Meta therefore emphasized: - Detecting problems before they reached data consumers. - Stopping further propagation quickly during rollback. - The reverse-shadow design provided early quality signals and preserved a ready-to-use legacy pipeline for rapid recovery. Meta’s migration demonstrates that large-scale infrastructure changes are safest when treated as controlled, observable lifecycle transitions rather than one-time cutovers. Parallel execution, automated data validation, explicit resource checks, and reversible rollouts enabled the company to migrate the entire workload while protecting downstream consumers.

Read original

Data Ingestion Machine Learning Mysql Change Data Capture+3

netflixApr 6, 20264 min readCurated summary

Stop Answering the Same Question Twice: Interval-Aware Caching for Druid at Netflix Scale

Netflix’s Druid deployment now exceeds 10 trillion rows and can ingest 15 million events per second, but repetitive dashboard queries became a scaling problem. Its new experimental caching layer handles rolling time windows by reusing settled historical results and querying Druid only for recent, changing data. Netflix accepts up to five seconds of additional staleness in exchange for substantially lower query load. ## The Scaling Problem - A dashboard with 26 charts can issue 64 queries per load. - Viewed by 30 people and refreshed every 10 seconds, that becomes roughly 192 queries per second. - Druid’s full-result cache misses whenever a rolling time interval changes. - Druid avoids caching realtime segments to preserve result correctness and determinism. - Per-segment caching reduces historical scans but still requires brokers to gather and merge results for every request. - Adding hardware to handle this redundant workload would be prohibitively expensive. ## Caching Only the Unsettled Data - In a three-hour query, most data is already stable; only the newest minutes are likely to change. - The cache stores previously returned historical portions and sends Druid only the uncached interval. - This approach is designed for time-grouped queries such as timeseries and groupBy queries. ## Deliberate Staleness - The cache can make the newest data up to five seconds stale. - This is acceptable because dashboards typically refresh every 10–30 seconds. - Netflix’s pipeline already has up to roughly five seconds of latency at P90. - Many queries also intentionally end at `now-1m` or `now-5s` to avoid unstable, newly arriving data. ## Exponential TTLs - Cache lifetimes increase with the age of each data point because older data is less likely to change. - Data under two minutes old has a minimum TTL of five seconds. - After that, TTL doubles for each additional minute: - 10 seconds at two minutes old - 20 seconds at three minutes - 40 seconds at four minutes - TTLs are capped at one hour. - Fresh data is refreshed frequently to account for late-arriving events, while older data remains cached longer. ## Time-Based Bucketing - A single cache entry per query and interval would still miss whenever a rolling window shifted. - Netflix instead uses a map-of-maps: - The outer key is a hash of the query excluding its time interval. - Inner keys represent timestamps bucketed by query granularity or at least one minute. - Big-endian timestamp encoding preserves chronological order for efficient range scans. - A three-hour query at one-minute granularity becomes 180 independently cached buckets. - When the window moves, most buckets can be reused and only the newly exposed range must be fetched. ## Router-Integrated Cache Service - The cache currently operates as an external service behind the Druid Router. - Cacheable requests are intercepted transparently: - Fully cached requests are answered directly. - Partially cached requests are narrowed to the missing interval and sent to Druid. - Metadata queries and queries without time-based grouping bypass the cache. - The proxy can be enabled or disabled without changing clients. - Netflix views this as an interim design while exploring deeper integration with Druid. ## Query Identification and Lookup - Incoming queries are parsed to extract their interval, granularity, and structure. - A SHA-256 hash is generated from the query’s logical contents, including datasource, filters, aggregations, and relevant context properties, while excluding the time interval. - The cache looks for buckets within the requested range. - Lookup requires cached buckets to be contiguous from the beginning of the requested interval; the provided article text ends while explaining the handling of expired or missing buckets. Netflix’s approach is best suited to frequently repeated rolling-window dashboards where a small, slightly stale tail is acceptable. Segmenting results by time and assigning age-based TTLs allows the system to preserve freshness where it matters while eliminating most redundant Druid work.

Read original

Data Ingestion Database Design Caching Query Optimization+3

pinterestFeb 5, 20263 min readCurated summary

Next Generation DB Ingestion at Pinterest

Pinterest replaced fragmented, batch-oriented database ingestion with a unified Change Data Capture (CDC) framework. The new architecture uses Debezium/TiCDC, Kafka, Flink, Spark, and Iceberg to process only changed records, reducing latency from over 24 hours to minutes while lowering infrastructure costs. It also provides native row-level deletion, scalable operations, and improved compliance. ## Problems with the Legacy System - Batch workflows often delayed updates by more than 24 hours. - Full-table processing was inefficient because many tables changed by less than 5% each day. - Lack of row-level deletion support complicated data compliance. - Multiple independently maintained pipelines created operational complexity and inconsistent data quality. ## Unified CDC-Based Architecture - Supports MySQL, TiDB, and KVStore. - Captures database changes through a generic CDC service and publishes them to Kafka, typically in under one second. - Flink processes events in near real time and stores them in append-only CDC Iceberg tables on S3. - Spark jobs run periodically—often every 15 minutes—to merge recent changes into base Iceberg tables. - A bootstrap pipeline initializes base tables from historical database dumps. - Maintenance jobs handle compaction and snapshot expiration. - The framework is designed for at-least-once processing, petabyte-scale data, thousands of pipelines, and YAML-based configuration. ## CDC Tables and Base Tables - CDC tables act as time-series ledgers containing every change event. - CDC data typically becomes available within five minutes. - Base tables mirror the current state of the source database while retaining historical records. - Base-table latency is generally between 15 minutes and one hour. ## Upserting Changes into Base Tables - Spark first identifies the newest event for each primary key. - Events are ranked by timestamp and GTID, then deduplicated. - Iceberg’s `MERGE INTO` applies the resulting changes: - Deletes matching records when the event represents a deletion. - Updates existing records. - Inserts new records unless the event is a deletion. - The process uses a recent CDC window and a processing watermark to avoid reprocessing unnecessary data. ## Choosing Merge-on-Read - Pinterest standardized on Iceberg’s Merge-on-Read (MOR) strategy. - Copy-on-Write (COW) was rejected for most workloads because: - It requires more computation during writes. - It produces substantially larger replacement files, increasing storage costs. - MOR better balances update performance and storage efficiency for frequent incremental changes. ## Partitioning for Faster Upserts - Large base tables can be partitioned using a hash bucket of the primary key. - For example, `bucket(100, id)` distributes records across 100 partitions. - This allows Spark to process partitions in parallel and reduces the data scanned or rewritten during merges. - Iceberg tables are configured with format version 2, identifier fields, merge-on-read update and delete modes, and target file sizes. ## Small-File Challenge - Bucketing improved parallelism but caused each upsert to generate many small files within partitions. - The article indicates that Pinterest investigated this bottleneck and introduced further optimizations, though the supplied excerpt ends before describing them. Pinterest’s CDC-based design provides a substantially faster and more efficient alternative to full-table batch ingestion. Teams adopting a similar system should combine incremental CDC processing with partitioning, merge-on-read storage, bootstrapping, and ongoing file-maintenance strategies.

Read original

Data Ingestion Apache Kafka Apache Spark Mysql+4

netflixNov 14, 2025Original article

How and Why Netflix Built a Real-Time Distributed Graph: Part 1 — Ingesting and Processing Data Streams at Internet Scale | by Netflix Technology Blog | Netflix TechBlog (opens in new tab)

Netflix has developed a Real-Time Distributed Graph (RDG) to unify member interaction data across its expanding business verticals, including streaming, live events, and mobile gaming. By transitioning from siloed microservice data to a graph-based model, the company can perform low-latency, relationship-centric queries that were previously hindered by expensive manual joins and data fragmentation. The resulting system enables Netflix to track user journeys across various devices and platforms in real-time, providing a foundation for deeper personalization and pattern detection. ### Challenges of Data Isolation in Microservices * While Netflix’s microservices architecture facilitates independent scaling and service decomposition, it inherently leads to data isolation where each service manages its own storage. * Data scientists and engineers previously had to "stitch" together disparate data from various databases and the central data warehouse, which was a slow and manual process. * The RDG moves away from table-based models to a relationship-centric model, allowing for efficient "hops" across nodes without the need for complex denormalization. * This flexibility allows the system to adapt to new business entities (like live sports or games) without requiring massive schema re-architectures. ### Real-Time Ingestion and Normalization * The ingestion layer is designed to capture events from diverse upstream sources, including Change Data Capture (CDC) from databases and request/response logs. * Netflix utilizes its internal data pipeline, Keystone, to funnel these high-volume event streams into the processing framework. * The system must handle "Internet scale" data, ensuring that events from millions of members are captured as they happen to maintain an up-to-date view of the graph. ### Stream Processing with Apache Flink * Netflix uses Apache Flink as the core stream processing engine to handle the transformation of raw events into graph entities. * Incoming data undergoes normalization to ensure a standardized format, regardless of which microservice or business vertical the data originated from. * The pipeline performs data enrichment, joining incoming streams with auxiliary metadata to provide a comprehensive context for each interaction. * The final step of the processing layer involves mapping these enriched events into a graph structure of nodes (entities) and edges (relationships), which are then emitted to the system's storage layer. ### Practical Conclusion Organizations operating with a highly decoupled microservices architecture should consider a graph-based ingestion strategy to overcome the limitations of data silos. By leveraging stream processing tools like Apache Flink to build a real-time graph, engineering teams can provide stakeholders with the ability to discover hidden relationships and cross-domain insights that are often lost in traditional data warehouses.

Data Ingestion Distributed Systems Microservices Real Time Data+3

datadogFeb 22, 20233 min readCurated summary

Husky: Exactly-once ingestion and multi-tenancy at scale

Husky, Datadog’s distributed, time-series-oriented event store, is optimized for large scans and aggregations rather than high-volume, low-latency point lookups. This makes exactly-once ingestion challenging, especially at Datadog’s multi-tenant scale. Datadog addresses the problem with deterministic, locality-aware routing that limits deduplication scope, improves storage efficiency, and supports autoscaling ingestion pipelines. ## Husky’s Ingestion Challenge - Husky separates storage and compute, allowing each to scale independently. - Its storage engine is designed primarily for large analytical scans and aggregations. - It is not optimized for massive numbers of low-latency point lookups, complicating duplicate detection during ingestion. - The ingestion system must guarantee that every event is stored exactly once while maintaining: - Multi-tenant scalability - Reasonable ingestion latency - Controlled infrastructure and storage costs ## Routing Events to Storage Shards - Datadog uses an upstream **Shard Router** to introduce locality into Kafka pipelines. - Events are deterministically assigned to shards based on their tenant, timestamp, and event ID. - Each tenant receives a list of shards rather than being permanently assigned to one shard. - A deterministic choice from that list distributes the tenant’s events while keeping the number of active shards as small as practical. - Downstream workers consume one or more shards and perform exactly-once ingestion into Husky. ## Benefits of Data Locality - **Simpler deduplication** - An event with the same timestamp and ID always reaches the same shard. - Deduplication only needs to occur within that shard. - Workers handle smaller sets of event IDs, making in-memory deduplication more efficient. - **Lower storage costs and better performance** - Each shard processes a relatively small set of tenants. - Husky stores each tenant in a separate table and does not mix tenants within files. - More tenants per writer produce more output files, increasing blob-storage costs and compaction work. - Restricting tenant cardinality reduces file creation and improves writer and compactor efficiency. ## Challenges in Deterministic Routing - **Changing shard assignments** - Tenant traffic can increase by one or two orders of magnitude. - Assignments may change when scaling a tenant across more shards or rebalancing traffic among existing shards. - **Distributed router consensus** - Every Shard Router node must make the same routing decision for a given event. - Inconsistent decisions could send duplicates to different shards and undermine exactly-once ingestion. - **Load balancing** - Shards must receive roughly equal traffic so downstream ingestion workers remain balanced. ## Time-Bounded Shard Placements - A simple deterministic mapping can select a shard using a hash of the event ID: ```text shard = shards[hash(event_id) % num_shards] ``` - This approach is cheap and stateless when all routers know the same shard list. - However, changing the shard list can cause the same event ID to map to a different shard, so assignment changes require additional coordination. - The article introduces **time-bounded Shard Placements** to preserve consistent routing while allowing tenant assignments to evolve, though the supplied excerpt ends before explaining the mechanism in detail. Datadog’s core recommendation is to combine deterministic, tenant-aware routing with carefully coordinated assignment changes. This narrows the scope of deduplication while reducing storage overhead and enabling balanced, scalable exactly-once ingestion.

Read original

Data Ingestion Database Design Apache Kafka Distributed Systems+4