10 posts
Benjamin Barton We shipped a feature that made perfect sense. It improved a specific type of investigation we had been testing against. Then other investigations started getting worse. Nothing crashed. No tests failed. But the overall quality of the agent had shifted, and we had…
Sanketh Balakrishna Andrew Zhang At Datadog, we operate thousands of services that rely on consistent, low-latency data access. Moving data between diverse systems—quickly and reliably—is essential, but complex. Each application may have its own requirements for data freshness,…
Kai Zong Khor William Yu Many Datadog products offer a live view of their telemetry, allowing you to access your data in near real time from across your infrastructure. Live views improve responsiveness, but they also introduce strict requirements on data ingestion latency and s…
Gabriel Reid Package delivery services like UPS, FedEx, and the postal service have a tough job. They contend with a never-ending stream of packages to be delivered, with expectations of prompt and reliable delivery. In building the Datadog Log Forwarding feature, we had to cont…
Gabriel Reid When you think about engineering challenges faced in handling incoming log data in Datadog, one of the first things that comes to mind is the scale: quickly and reliably parsing and processing millions of logs per second over thousands of containers. By comparison,…
Guillaume Bort At Datadog, we use Apache Kafka as a durable, flexible buffer in our Data Platform. Ingesting hundreds of trillions of observability events each day requires massive Kafka deployments: hundreds of clusters, thousands of topics, and millions of partitions spanning…
Artem Krylysov May Lee Datadog collects billions of events from millions of hosts every minute and that number keeps growing and fast. Our data volumes grew 30x between 2017 and 2022. On top of that, the kind of queries we receive from our users has changed significantly. Why? B…
Daniel Intskirveli Cecilia Watt We introduced Husky in a previous blog post (Introducing Husky, Datadog's Third-Generation Event Store) as Datadog's third-generation event store. To recap, Husky is a distributed, time-series oriented, columnar store optimized for streaming inges…
Richard Artoul Cecilia Watt This is the story of "Husky", a new event storage system we built at Datadog. Building a new storage system is a fun and exciting undertaking—and one that shouldn’t be taken lightly. Most importantly, it does not happen in a vacuum. To understand the…
Jamie Alquiza For a company with data in the name, it’s no surprise that we ingest large amounts of it. Kafka is our messaging persistence layer of choice underlying many of our high-traffic services. Consequently, our Kafka usage is quite high: the intake of trillions of data p…