Query Engine | Techlist.io

datadogOct 1, 20253 min readCurated summary

Inside Husky’s query engine: Real-time access to 100 trillion events

Datadog’s Husky event store is designed not merely to store more than 100 trillion events, but to make them queryable interactively at massive scale. Its query engine handles schemaless data, highly variable tenant workloads, petabytes of object-store data, and both highly selective searches and broad analytics queries. The architecture combines distributed planning, metadata-driven pruning, coordinated execution, and reader-level optimizations to minimize the data and fragments that must be scanned. ## Husky’s Data and Query Workloads - Events contain a timestamp and flexible attributes. - Data shapes vary widely: - Logs, network events, and traces have different schemas. - Tenants may produce either a few large events or enormous numbers of small events. - Most queries fall into two categories: - **Needle-in-a-haystack searches**, such as finding a specific IP connection, error message, or trace. - **Analytics-style searches**, such as time series, grouped breakdowns, and distributions. - The engine must also support raw event retrieval and more complex queries such as joins. ## Distributed Query Path Husky divides query execution among four multi-tenant services deployed across regions and data centers. ### Query Planner - Serves as the main entry point for event-store queries. - Resolves query context, validates and throttles requests, and integrates with other Datadog data stores. - Applies optimizations and statistics to split queries into time-based steps. - Schedules those steps on orchestrators and merges their results into the final response. ### Query Orchestrator - Acts as the gateway to Husky’s stored data. - Fetches fragment metadata, including: - File paths and versions - Row counts - Timestamp boundaries - Zone maps for query matching - Dispatches only relevant fragments to reader nodes. - Uses zone-map pruning to reduce downstream work by up to 60% for structured events and about 30% on average. - Aggregates results after fragment processing, which can require more computation than query planning. ### Metadata Service - Provides an abstraction over FoundationDB clusters. - Preserves atomicity during operations such as compaction, preventing duplicate data from appearing in query results. - Separates FoundationDB implementation details from the rest of the query system. - Must work within FoundationDB’s five-second transaction limit. ### Reader Service - Receives a query and selected fragments, then returns results quickly. - Performs the direct scan and execution work over fragment data. - Contains multiple optimizations intended to keep queries interactive despite scanning data stored in blob storage at extreme scale. ## Minimizing Data Scans The reader service follows the principle that the fastest query is the one that avoids unnecessary work. - Scanning less data reduces both latency and storage costs. - Touching fewer fragments limits expensive object-store operations. - This is especially important because Husky stores millions of fragments daily and cannot afford multiple blob-storage GET requests for every fragment in every query. ## Row Groups and Reader Execution - Fragments can contain millions of rows, making fully in-memory processing risky and potentially causing memory pressure or failures. - To limit data retrieval, fragments are physically organized into **row groups**. - Row groups allow the reader to fetch only portions of a fragment needed for a query rather than loading the entire file. - The reader uses an iterator-based execution model inspired by the Volcano query-processing architecture. - The provided article ends as it begins explaining how this row-group layout supports efficient iterator-based query execution.

Read original

Query Engine Database Design Data Architecture Foundationdb+3

datadogMay 17, 20223 min readCurated summary

Introducing Husky, Datadog's third-generation event store

Datadog built Husky, a new event-storage system, after its original log architecture struggled with multi-tenant reliability, rapid platform growth, and evolving product requirements. The post explains how Datadog moved from metrics-oriented storage to event storage, introduced custom sharding and routing, and eventually recognized the need for a more flexible system. Husky emerged from these lessons about isolation, scalability, and retaining high-cardinality event data. ## From Metrics to Logs - Metrics systems store pre-aggregated tuples such as `<timeseries_id, timestamp, float64>`. - Aggregation makes metrics extremely efficient: millions of events in a second can become one compact datapoint, often requiring less than two bytes with delta-of-delta encoding. - This model is poorly suited to logs because logs must preserve individual events and their full context. - Metrics typically favor long-lived, low-cardinality dimensions such as: - Datacenter - Service - Pod name - Short-lived, high-cardinality fields such as transaction IDs and packet IDs are usually pre-aggregated or omitted. - Logs instead need to support: - Multi-kilobyte events - High-cardinality values such as UUIDs and stack traces - Arbitrary aggregations performed at query time ## Limitations of the Initial Logs System - Datadog’s first Logs architecture initially worked well but became vulnerable in a multi-tenant environment. - A single unhealthy or overloaded node could degrade service for every tenant in the cluster. - Scaling overloaded clusters could worsen the situation because nodes began streaming data to one another while already handling excessive read and write workloads. - Diagnosing and mitigating these cascading failures was difficult. ## Separating Storage from Clustering Datadog’s second architecture retained the same single-node storage engine but moved clustering responsibilities into dedicated services. - Storage nodes no longer knew about one another and behaved like independent one-node clusters. - Failures were isolated to the tenants assigned to a particular shard instead of spreading across the entire cluster. - A Shard Router: - Read events from Kafka - Reorganized them into shard-based Kafka partitions - Dynamically assigned tenants to an appropriate number of shards based on their recent five-minute data volume - Each shard was consumed by two storage-node replicas for redundancy. - A custom query engine tracked tenant-to-shard assignments, queried the relevant replicas, merged partial aggregates, and produced final results. ## Growth of the Event Platform - The new architecture substantially improved reliability and reduced operational burden. - Datadog expanded the platform beyond Logs to support products including: - Network Performance Monitoring - Real User Monitoring - Continuous Profiler - These products generated structured, multi-kilobyte events with storage and indexing requirements similar to logs. - As usage grew, new problems appeared: - A tenant producing a sudden burst of events could degrade query performance for other tenants sharing its shard. - Product teams requested longer retention for important but infrequently queried data, while still requiring it to remain immediately queryable. - The existing architecture was increasingly difficult to adapt to these isolation, scalability, and retention requirements, motivating the development of Husky. Datadog’s progression shows that event storage cannot simply reuse metrics-oriented designs. Systems must preserve event-level context, isolate tenants from one another, and support changing retention and query requirements as products and workloads evolve.

Read original

Query Engine Database Design Datadog Apache Kafka+4