Datadog | Techlist.io

datadogOct 27, 20232 min readCurated summary

Engineering spotlight: Jeromy Carriere | Datadog

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The provided content does not include the report’s evaluation criteria or detailed rationale, but it presents Datadog as a broad observability platform spanning infrastructure, applications, data, logs, security, digital experience, software delivery, and AI. ## Gartner Recognition - Datadog highlights its position as a Leader in Gartner’s 2026 Magic Quadrant for Observability Platforms. - The linked resource appears to provide the full Gartner-related announcement and assessment. ## Broad Observability Coverage - **Infrastructure:** infrastructure, container, network, serverless, GPU, storage, and cloud-cost monitoring. - **Applications:** APM, service monitoring, continuous profiling, dynamic instrumentation, and agent observability. - **Data and logs:** database monitoring, data-stream and job monitoring, log management, sensitive-data scanning, and observability pipelines. - **Security:** code, cloud, workload, vulnerability, compliance, SIEM, and application/API protection. - **Digital experience:** real-user monitoring, session replay, synthetic monitoring, product analytics, mobile testing, and error tracking. - **Software delivery and service management:** CI visibility, test optimization, feature flags, incident response, SLOs, workflow automation, and developer portals. - **AI capabilities:** AI agents, investigation tools, GPU monitoring, integrations, MCP support, and AI-assisted chat and coding. Datadog’s positioning rests on consolidating telemetry and operational workflows across the technology stack. To understand the Gartner recognition in depth, readers would need the linked report, since the supplied excerpt contains the announcement and product navigation but not the supporting analysis.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogOct 27, 20233 min readCurated summary

Engineering spotlight: Jeromy Carriere

Jeromy Carriere, Datadog’s SVP of Product Engineering, describes engineering leadership as balancing strategy, execution, people development, and organizational processes. His career across Google, Facebook, and Datadog shaped his passion for observability and taught him to lead through mistakes, autonomy, and accountability. He argues that sustained innovation requires both intentional direction and space for teams and individuals to grow. ## The Responsibilities of Engineering Leadership - Carriere’s work shifts with organizational cycles: - Quarterly planning focused on connecting initiatives and increasing collaboration. - Execution periods focused on removing resource and decision-making blockers. - Ongoing performance management across the broader engineering organization. - He works on how Engineering operates, including: - Process definition and improvement. - Hiring and performance management. - Reviewing design documents and code. - Observing incidents and postmortems. - His central challenge is balancing attention across strategy, execution, people, and technical quality. ## From Cloud Monitoring to Datadog - At Google in 2014, Carriere helped create a cloud monitoring offering because Google Cloud lacked capabilities comparable to Datadog. - He later worked on observability at Facebook and developed a strong interest in improving developer and engineer productivity. - He returned to Datadog after seeing its ability to innovate and deliver products with sustained velocity. - He emphasizes that velocity means more than moving quickly: it requires direction, strategy, and consistency over time. ## Learning Through Mistakes - Carriere believes the most valuable lessons come from making and owning mistakes. - Earlier in his career, he was sometimes too directive, limiting team creativity and ownership. - At other times, he was too distant and failed to provide enough support. - His current leadership approach aims to: - Give teams substantial autonomy. - Provide support when needed. - Hold teams accountable for agreed-upon outcomes. - He also learned that people may not have a clear five-year career plan. Leaders should help them identify the work that provides satisfaction and enables them to perform at their best. ## Creating Space for Career Decisions - People often become focused on the immediate task and overlook other possibilities. - Carriere recommends deliberately stepping back to observe: - What activities feel satisfying. - What opportunities exist nearby. - What kinds of work could better match an individual’s strengths and interests. - This reflection requires intentional time and freedom rather than waiting for clarity to emerge automatically. ## The Value of Co-op and Internship Programs - Carriere credits the University of Waterloo’s co-op program with giving him early experience as a professional software developer. - The combination of strong academic training and repeated, high-quality industry placements helped connect theory with real work. - He sees a similar benefit in Datadog’s internship program, where interns are trusted with meaningful projects and often produce some of the company’s strongest work. Datadog’s engineering approach, as described by Carriere, combines strategic product velocity with thoughtful organizational support. For both leaders and individual contributors, the practical recommendation is to learn from mistakes, create room for reflection, and build environments where people have autonomy, meaningful work, and accountability.

Read original

Database Design Datadog Observability Software Development+4

datadogJun 30, 20233 min readCurated summary

How we migrated our acceptance tests to use Synthetic Monitoring

Datadog’s Frontend Developer Experience team migrated 565 flaky, maintenance-heavy Puppeteer acceptance tests to Synthetic Monitoring. The change replaced manually scripted browser interactions with recorded tests that could run reliably from CI through a dedicated CLI. The year-long migration improved maintainability and built engineer confidence through documentation, gradual adoption, and non-blocking rollout tooling. ## Why the Existing Acceptance Tests Were Failing - Tests ran in Node.js on a custom Puppeteer-based runner. - End-to-end tests were flaky because they depended on browsers, virtual graphics, dedicated machines, navigation, and application timing. - Even simple actions required extensive scripting: - Confirming an element existed - Checking that it was enabled - Performing the interaction - Handling compatibility with Puppeteer - Custom UI elements, such as dropdowns, made reliable automation substantially harder. - Product changes frequently required updates to both tests and the testing infrastructure. - The six CI jobs took up to 14 minutes, with total machine time reaching 35 minutes per commit. ## Synthetic Monitoring as the Replacement - The team adopted Datadog’s own Synthetic Monitoring product to record page interactions rather than manually script them. - They created `synthetics-ci`, a CLI that: - Finds files named `*.synthetics.json` - Accepts configuration overrides - Triggers Synthetic tests - Polls for result statuses - Prints human-readable output - Because the tool represented a broader pattern for using Datadog from CI/CD, it was generalized into `datadog-ci`. ## Scope of the Migration At the start of the migration in June 2021, the frontend repository had: - 300 engineers working in one repository - Approximately 90 new pull requests and 1,120 commits every day - Six acceptance-test CI jobs - 35 minutes of machine time per commit - 84 relevant files - 565 tests - About 100,000 lines of test and infrastructure code The scale of the repository and its rapid development activity meant the migration needed to be gradual and carefully coordinated. ## Building Trust and Adoption - The team wrote documentation covering: - How to write effective Synthetic tests - Which behaviors were worth testing - Which testing patterns to avoid - They demonstrated the system in company-wide and frontend-focused meetings. - Teams learned how to: - Record tests through the UI - Use scheduled tests in CI - Replace existing acceptance tests - Reduce ongoing maintenance - The team worked directly with groups that owned the largest test collections. - Jira tickets tracked the migration of every acceptance test and assigned ownership to the appropriate team. ## Gradual CI Integration - A non-blocking CI job allowed teams to introduce Synthetic tests without risking the entire pipeline. - Failures appeared as pull-request comments rather than blocking merges. - This gave engineers time to understand and trust the new system. - Once tests ran reliably, the team made the pipeline blocking. - The old acceptance-testing platform could then be retired progressively instead of being removed all at once. The migration took roughly one year and succeeded through a combination of better tooling, clear communication, incremental rollout, and shared ownership across frontend teams. For large organizations replacing a critical testing system, introducing the new workflow safely before enforcing it can make adoption far less disruptive.

Read original

Datadog CI/CD Synthetic Monitoring Node.Js+3

datadogJun 30, 2023Original article

How we migrated our acceptance tests to use Synthetic Monitoring | Datadog (opens in new tab)

Datadog’s Frontend Developer Experience team migrated their massive codebase from a fragile, custom Puppeteer-based acceptance testing framework to Datadog Synthetic Monitoring to address persistent flakiness and high maintenance overhead. By leveraging a record-and-play approach and integrating it into their CI/CD pipelines via the `datadog-ci` tool, they successfully reduced developer friction and improved testing reliability for over 300 engineers. This transition demonstrates how replacing manual browser scripting with specialized monitoring tools can significantly streamline high-scale frontend workflows. ### Limitations of Puppeteer-Based Testing * Custom runners built on Puppeteer suffered from inherent flakiness because they relied on a complex chain of virtual graphic engines, browser manipulation, and network stability that frequently failed unexpectedly. * Writing tests was unintuitive, requiring engineers to manually script interaction details—such as verifying if a button is present and enabled before clicking—which became exponentially more complex for custom elements like dropdowns. * The testing infrastructure was slow and expensive, with CI jobs taking up to 35 minutes of machine time per commit to cover the application's 565 tests and 100,000 lines of test code. * Maintenance was a constant burden; every product update required a corresponding manual update to the scripts, making the process as labor-intensive as writing new features. ### Adopting Synthetic Monitoring and Tooling * The team moved to Synthetic Monitoring, which allows engineers to record browser interactions directly rather than writing code, significantly lowering the barrier to entry for creating tests. * To integrate these tests into the development lifecycle, the team developed `datadog-ci`, a CLI tool designed to trigger tests and poll result statuses directly from the CI environment. * The new system uses a specific file format (`.synthetics.json`) to identify tests within the codebase, allowing for configuration overrides and human-readable output in the build logs. * This transition turned an internal need into a product improvement, as the `datadog-ci` tool was generalized to help all Datadog users execute commands from within their CI/CD scripts. ### Strategies for High-Scale Migration and Adoption * The team utilized comprehensive documentation and internal "frontend gatherings" to educate 300 engineers on how to record tests and why the new system required less maintenance. * To build developer trust, the team initially implemented the new tests as non-blocking CI jobs, surfacing failures as PR comments rather than breaking builds. * Migration was treated as a distributed effort, with 565 individual tests tracked via Jira and assigned to their respective product teams to ensure ownership and a steady pace. * By progressively sunsetting the old platform as tests were migrated, the team managed a year-long transition without disrupting the daily output of 160 authors pushing 90 new PRs every day. To successfully migrate large-scale testing infrastructures, organizations should prioritize developer trust by introducing new tools through non-blocking pipelines and providing comprehensive documentation. Transitioning from manual browser scripting to automated recording tools not only reduces technical debt but also empowers engineers to maintain high-quality codebases without the burden of managing complex testing infrastructure.

Datadog Web Development Synthetic Monitoring Node.Js+2

datadogJun 16, 2023Original article

2023-03-08 incident: A deep dive into the platform-level recovery | Datadog (opens in new tab)

Following a massive system-wide outage in March 2023, Datadog successfully restored its EU1 region by identifying that a simple node reboot could resolve network connectivity issues caused by a faulty system patch. While the team managed to restore 100 percent of compute capacity within hours, the recovery effort was subsequently hindered by cloud provider infrastructure limits and IP address exhaustion. This post-mortem highlights the complexities of scaling hierarchical Kubernetes environments under extreme pressure and the importance of accounting for "black swan" capacity requirements. ## Hierarchical Kubernetes Recovery Datadog utilizes a strict hierarchy of Kubernetes clusters to manage its infrastructure, which necessitated a granular, three-tiered recovery approach. Because the outage affected network connectivity via `systemd-networkd`, the team had to restore components in a specific order to regain control of the environment. * **Parent Control Planes:** Engineers first rebooted the virtual machines hosting the parent clusters, which manage the control planes for all other clusters. * **Child Control Planes:** Once parent clusters were stable, the team restored the control planes for application clusters, which run as pods within the parent infrastructure. * **Application Worker Nodes:** Thousands of worker nodes across dozens of clusters were restarted progressively to avoid overwhelming the control planes, reaching full capacity by 12:05 UTC. ## Scaling Bottlenecks and Cloud Quotas Once the infrastructure was online, the team attempted to scale out rapidly to process a massive backlog of buffered data. This surge in demand triggered previously unencountered limitations within the Google Cloud environment. * **VPC Peering Limits:** At 14:18 UTC, the platform hit a documented but overlooked limit of 15,500 VM instances within a single network peering group, blocking all further scaling. * **Provider Intervention:** Datadog worked directly with Google Cloud support to manually raise the peering group limit, which allowed scaling to resume after a nearly four-hour delay. ## IP Address and Subnet Capacity Even after cloud-level instance quotas were lifted, specific high-traffic clusters processing logs and traces hit a secondary bottleneck related to internal networking. * **Subnet Exhaustion:** These clusters attempted to scale to more than twice their normal size, quickly exhausting all available IP addresses in their assigned subnets. * **Capacity Planning Gaps:** While Datadog typically targets a 66% maximum IP usage to allow for a 50% scale-out, the extreme demands of the recovery backlog exceeded these safety margins. * **Impact on Backlog:** For six hours, the lack of available IPs forced these clusters to process data significantly slower than the rest of the recovered infrastructure. ## Recovery Summary The EU1 recovery demonstrates that even when hardware is functional, software-defined limits can create cascading delays. Organizations should not only monitor their own resource usage but also maintain visibility into cloud provider quotas and ensure that subnet allocations account for extreme recovery scenarios where workloads may need to double or triple in size momentarily.

Database Design Kubernetes Datadog Cloud Computing+4

datadogJun 16, 20233 min readCurated summary

2023-03-08 incident: A deep dive into the platform-level recovery

Datadog’s March 8, 2023 outage removed 60% of its compute capacity, forcing teams to restore infrastructure in stages while accounting for regional and cloud-provider differences. In EU1, recovery depended on rebooting affected nodes, restoring Kubernetes control planes in a strict hierarchy, and gradually bringing application capacity back online. Scaling afterward exposed infrastructure limits that had not been considered during normal operations. ## EU1 Platform Recovery - A system patch disconnected affected EU1 nodes from the network, but the nodes could be recovered through reboots. - Recovery was initially slowed by the lack of observability and unavailable Kubernetes APIs. - Datadog operates: - **Parent clusters**, which host the control-plane pods for other clusters. - **Child clusters**, where Datadog applications run. - This hierarchy allows Datadog to use Kubernetes deployment, replacement, rolling-update, and autoscaling capabilities for child-cluster control planes. - Parent-cluster control planes run on VMs and are managed with `systemd`. ## Restoring Kubernetes Clusters Because both parent and child environments were affected by the Ubuntu 22.04 issue, recovery had to follow a strict sequence: - **Parent control planes:** Nodes running Cilium were rebooted to restore network connectivity. This finished by 08:45 UTC. - **Child control planes:** All parent-cluster nodes hosting child control-plane pods were rebooted. This finished by 09:30 UTC. - **Application nodes:** Thousands of instances across dozens of child clusters were restarted. - Recovery reached 60% by 10:20 UTC. - All application nodes were restored by 12:05 UTC. - Restarts were prioritized by workload importance and paced to avoid overwhelming Kubernetes control planes. ## Scaling Capacity and Recovering Backlogs After restoring the clusters, Datadog needed substantial additional capacity to process data buffered during the outage. - EU1 hit a Google Cloud mesh limit of **15,500 VM instances** at 14:18 UTC. - Instance creation failures became apparent around 15:00 UTC. - Datadog had not checked this documented limit before the incident, but Google Cloud quickly raised it after Datadog submitted a high-priority request. - Autoscaling also exhausted the IP capacity of subnets used by three log- and trace-processing clusters. - These clusters normally used about 35–45% of their IP capacity, but the backlog caused autoscaling to request more than twice their usual replica counts, filling the subnets. ## Practical Lessons The recovery demonstrated that restoring compute capacity is not enough: teams must also understand dependency order, control-plane architecture, cloud-provider quotas, and network-address limits. Capacity planning should account for severe backlog-driven scaling, not just normal operating utilization, and documented infrastructure limits should be validated before emergencies occur.

Read original

Kubernetes Observability Cloud Computing Cloud Infrastructure+4

datadogJun 1, 20233 min readCurated summary

2023-03-08 incident: A deep dive into our incident response

Datadog’s March 8, 2023 global outage tested an incident-response process designed for large-scale failures. The company’s monitoring, on-call structure, training, and blameless culture enabled a coordinated response, but the incident also exposed challenges in diagnosing and managing a rapidly evolving, global outage. Datadog’s central lesson is that effective response depends less on rigid runbooks than on preparation, clear ownership, autonomous decision-making, and continuous learning. ## Datadog’s Incident Response Model - Datadog follows a “you build it, you own it” operating model. - Teams instrument their services extensively and configure monitors to detect problems around the clock. - Independent, out-of-band monitoring checks Datadog’s APIs from outside its infrastructure, ensuring that monitoring still works if Datadog itself becomes unavailable. - Slack channels are automatically created for incidents to provide shared situational awareness and enable additional engineers to contribute. ## Handling High-Severity Incidents - Senior engineers rotate on call for incidents involving substantial customer impact or multiple teams. - The first senior responder becomes the incident commander and retains overall responsibility. - A communications lead may manage internal updates and coordination. - For the most serious incidents, an engineering executive and customer-support manager join to provide leadership, business context, and customer-facing communication. - The incident commander remains accountable for coordinating the overall response. ## Preparation, Training, and Postmortems - Datadog uses a relatively low threshold for declaring incidents, giving engineers frequent practice with its response process. - Engineers complete incident-response training before joining an on-call rotation and repeat refresher training every six months. - Training covers on-call responsibilities, response roles, and blameless investigation practices. - Every high-severity incident receives a detailed postmortem focused on preventing recurrence. - Automation prompts responders to begin postmortems while the incident is still fresh. ## Autonomy and a Blameless Culture - Because large systems change constantly, detailed recovery procedures can quickly become outdated. - Datadog therefore gives engineers authority to choose the best response based on their knowledge of the affected services. - The company treats failures as weaknesses in systems rather than evidence of individual fault. - Blamelessness is intended to encourage creativity, honesty, and effective decision-making under pressure. ## The March 8 Outage - A systemd upgrade began around 06:00 UTC and ultimately triggered the outage. - Monitoring detected the problem within three minutes, and engineering teams were paged shortly afterward. - A high-severity incident was declared at 06:18, with an incident commander joining five minutes later. - The first public status update was posted at 06:31, and the outage was officially diagnosed as global at 06:32. - By 07:20, responders identified a Kubernetes failure and unhealthy intake systems as central problems. - Engineers confirmed by 08:00 that the Kubernetes failure was not spreading to additional or newly provisioned nodes. - A working mitigation for the EU1 region was found by 08:30. - Most US1 compute capacity recovered automatically by 11:00, while teams began organizing a longer recovery effort. - At 11:36, unattended upgrades were identified as the triggering event. - Compute capacity in EU1—the first step toward recovery—was restored by 12:05. ## Practical Lessons Datadog’s experience demonstrates the value of independent monitoring, practiced incident roles, rapid communication, and empowered responders. Organizations operating complex systems should regularly rehearse incident management, invest in resilient observability outside the primary platform, and use blameless postmortems to turn major outages into improvements.

Read original

Kubernetes Datadog Automation Monitoring+4

datadogJun 1, 2023Original article

2023-03-08 incident: A deep dive into our incident response | Datadog (opens in new tab)

Datadog’s first global outage on March 8, 2023, served as a rigorous stress test for their established incident response framework and "you build it, you own it" philosophy. While the outage was triggered by a systemic failure during a routine systemd upgrade, the company's commitment to blameless culture and decentralized engineering autonomy allowed hundreds of responders to coordinate a complex recovery across multiple regions. Ultimately, the event validated their investment in out-of-band monitoring and rigorous, bi-annual incident training as essential components for managing high-scale system disasters. ## Incident Response Structure and Philosophy * Datadog employs a decentralized "you build it, you own it" model where individual engineering teams are responsible for the 24/7 health and monitoring of the services they build. * For high-severity incidents, a specialized rotation is paged, consisting of an Incident Commander to lead the response, a communications lead, and a customer liaison to manage external messaging. * The organization prioritizes "people over process," empowering engineers to use their judgment to find creative solutions rather than following rigid, pre-written playbooks that may not apply to unprecedented failures. * A blameless culture is strictly maintained across all levels of the company, ensuring that post-incident investigations focus on systemic improvements rather than assigning fault to individuals. ## Multi-Layered Monitoring Strategy * Standard telemetry provides internal visibility, but Datadog also maintains "out-of-band" monitoring that operates completely outside its own infrastructure. * This out-of-band system interacts with Datadog APIs exactly like a customer would, ensuring that engineers are alerted even if the internal monitoring platform itself becomes unavailable. * Communication is streamlined through a dedicated Slack incident app that automatically generates coordination channels, providing situational awareness to any engineer who joins the effort. ## Anatomy of the March 8 Outage * The outage began at 06:00 UTC, triggered by a systemd upgrade that caused widespread Kubernetes failures and prevented pods from restarting correctly. * The global nature of the outage was diagnosed within 32 minutes of the initial monitoring alerts, leading to the activation of executive on-calls and the customer support management team. * Responders identified "unattended upgrades" as the incident trigger approximately five and a half hours after the initial failure. * Recovery was executed in stages: compute capacity was restored first in the EU1 region, followed by the US1 region, with full infrastructure restoration completed by 19:00 UTC. Organizations should treat incident response as a perishable skill that requires constant practice through a low threshold for declaring incidents and regular training. By combining out-of-band monitoring with a culture that empowers individual engineers to act autonomously during a crisis, teams can more effectively navigate the "not if, but when" reality of large-scale system failures.

Kubernetes Datadog Automation Slack+4

datadogMay 26, 20232 min readCurated summary

Not just another network latency issue: How we unraveled a series of hidden bottlenecks | Datadog

Datadog announces that Gartner has named it a Leader in the 2026 Magic Quadrant for Observability Platforms. The provided content primarily consists of Datadog’s product navigation rather than the full blog post, so it does not include Gartner’s evaluation criteria, Datadog’s specific strengths, or comparative analysis. ## Gartner Recognition - Datadog is identified as a **Leader** in Gartner’s **Magic Quadrant for Observability Platforms 2026**. - The announcement links to a Gartner-related resource hosted by Datadog. - No details are provided in the supplied text about: - Gartner’s assessment of Datadog’s execution or vision - The other vendors evaluated - Specific capabilities that led to the ranking - Any limitations or cautions noted by Gartner ## Datadog’s Observability Offering The surrounding navigation shows that Datadog positions observability as part of a broad platform covering: - Infrastructure and cloud monitoring - Application performance monitoring and profiling - Logs, metrics, databases, and data pipelines - Real user, synthetic, and mobile monitoring - Security and cloud protection - CI/CD and software delivery - Incident response, service management, and automation - AI-powered investigation and agent observability The supplied content does not explain how these products are integrated or how they support the Leader designation. Overall, the material communicates Datadog’s recognition as a Gartner observability-platform Leader, but the full article or report is needed for a meaningful technical summary.

Read original

Kubernetes Datadog Observability Application Performance Monitoring+3

datadogMay 26, 20233 min readCurated summary

Not just another network latency issue: How we unraveled a series of hidden bottlenecks

Repeated high-startup-latency pages in Datadog’s usage estimation service were caused by several independent bottlenecks rather than application changes. The investigation eventually identified four issues: CPU-throttled Envoy sidecars, a Linux kernel bug affecting ENA transmit queues, insufficient EC2 network bandwidth, and requests routed to terminating cache pods. Fixing each layer progressively reduced remote-cache p99 latency from roughly one second to its normal level of about 100 ms. ## Service Architecture and the Original Symptoms - The service consists of router, counter, and aggregator applications. - At startup, `counter` loads data from a remote cache into a local cache. - While the local cache is populating, request processing is slower and backlog grows. - Normal p99 remote-cache latency was approximately 100 ms, but it exceeded one second during deployments. - Scaling the remote cache did not help, indicating that the cache itself was not underprovisioned. ## CPU-Throttled Envoy Sidecars - Requests to the remote cache passed through an Envoy sidecar that batched queries into packets. - When `counter` restarted, Envoy reached its two-core CPU limit and was throttled. - Delayed request and response processing caused retries, TCP retransmits, and increased remote-cache latency. - Increasing Envoy’s CPU allocation eliminated the issue in staging and reduced production latency, but did not fully resolve rollout spikes. ## Linux Kernel and ENA Transmit-Queue Bug - Investigation of system and network metrics revealed a Linux kernel bug affecting AWS Elastic Network Adapter traffic. - The kernel mapped all outbound traffic to the first transmit queue instead of distributing it across eight queues. - This limited throughput and caused retransmits during high-traffic periods such as deployments. - A hotfix distributed traffic across all eight queues. - The change removed non-rollout latency spikes but rollout latency still fluctuated between 200 and 600 ms. ## EC2 Network Bandwidth Limits - ENA metrics showed that instances exceeded AWS inbound and outbound bandwidth allowances. - AWS dropped packets at the hypervisor when those limits were exceeded, causing retransmissions and slower cache requests. - Migrating to network-optimized EC2 instance types with higher bandwidth allowances largely restored p99 latency to around 100 ms. - Occasional one-second spikes continued despite the improvement. ## Requests Sent to Terminating Cache Pods - Remaining spikes correlated with remote-cache pods that were shutting down. - Clients continued sending requests to terminating pods, leading to one-second timeouts and retries. - The cache’s graceful-shutdown behavior did not adequately wait for Envoy clients’ in-flight requests. - The team added a `preStop` hook that sets an `XXX_MAINTENANCE_MODE` key to notify clients before termination and began coordinating shutdown with outstanding requests. The incident demonstrates the importance of tracing latency across the entire request path, from application startup through proxies, kernel networking, hardware interfaces, cloud bandwidth limits, and pod lifecycle behavior. Layered system metrics and component-level investigation were necessary to eliminate alert fatigue and restore reliable deployment behavior.

Read original

Database Design Linux Kernel Aws Ec2 Envoy+4

datadogMay 24, 20233 min readCurated summary

2023-03-08 incident: A deep dive into the platform-level impact

Datadog’s March 8, 2023 outage was caused by an unexpected interaction between Ubuntu 22.04, systemd-networkd, and an automated security patch. A systemd change introduced behavior that flushed unfamiliar IP routing rules whenever systemd-networkd restarted; a CVE patch triggered that restart across many hosts. Because the patch was installed automatically and outside Datadog’s carefully staged deployment process, infrastructure across regions and cloud providers was affected simultaneously. ## A Systemd Behavior Change - systemd v248 introduced a systemd-networkd startup behavior that removed IP rules it did not recognize. - systemd v249 added the `ManageForeignRoutingPolicyRules` setting, which could disable this behavior, but the default configuration continued managing foreign rules. - These changes were backported to older systemd releases. - Ubuntu 20.04 used systemd v245, which did not flush IP rules during a systemd-networkd restart. - Ubuntu 22.04, adopted progressively by Datadog beginning in November 2022, used systemd v249 with the behavior enabled. Initially, the change caused no visible problems because systemd-networkd generally started only when new hosts were created, before Datadog’s custom routing rules existed. ## The Security Patch That Triggered the Problem - On March 7, 2023, Ubuntu released systemd patch `249.11-0ubuntu3.7` for a CVE. - Installing the patch restarted all systemd components, including systemd-networkd. - That restart caused systemd-networkd to flush routing policy rules on affected Ubuntu 22.04 hosts. - Ubuntu 20.04 hosts received a similar patch but were not affected because systemd v245 did not exhibit the problematic restart behavior. ## Unattended Upgrades Created Broad Exposure Datadog used Ubuntu’s default unattended-upgrade configuration: - Package metadata was downloaded twice daily using `apt-daily.timer`, with randomized delays of up to 12 hours. - Upgrades ran daily using `apt-daily-upgrade.timer`, between 06:00 and 07:00 UTC. - Only security updates and required dependencies were automatically installed. - Regular updates from the `-updates` repository were excluded. This meant many hosts automatically installed the systemd security patch during the same daily upgrade window. Not every host was affected: more than 90% of the fleet used Ubuntu 22.04, and some nodes had not yet downloaded the patch when their upgrade ran. ## Conflict with Datadog’s Deployment Process - Datadog normally updates nodes by replacing them automatically rather than relying on unattended upgrades. - Its standard process validates changes on experimental clusters, then progressively deploys them through staging and production. - Deployments are normally limited to selected clusters, availability zones, and regions before expanding. - The unattended systemd patch bypassed this process because it was installed directly on existing hosts. - As a result, a low-level networking change propagated across otherwise isolated regions and cloud providers at nearly the same time. Datadog’s experience demonstrates that even security-only automated updates can introduce coordinated infrastructure risk. Critical system packages should be tested and rolled out through the same staged process as other production changes, or their automated upgrades should be carefully constrained and monitored.

Read original

Kubernetes Cloud Infrastructure Systemd Ubuntu+4

datadogMay 24, 2023Original article

2023-03-08 incident: A deep dive into the platform-level impact | Datadog (opens in new tab)

The March 2023 Datadog outage was triggered by a simultaneous, global failure across multiple cloud providers and regions, caused by an unexpected interaction between a systemd security patch and Ubuntu 22.04’s default networking behavior. While Datadog typically employs rigorous, staged rollouts for infrastructure changes, the automated nature of OS-level security updates bypassed these controls. The incident highlights the hidden risks in system-level defaults and the potential for "unattended upgrades" to create synchronized failures across supposedly isolated environments. ## The systemd-networkd Routing Change * In December 2020, systemd version 248 introduced a change where `systemd-networkd` flushes all IP routing rules it does not recognize upon startup. * Version 249 introduced the `ManageForeignRoutingPolicyRules` setting, which defaults to "yes," confirming this management behavior for any rules not explicitly defined in systemd configuration files. * These changes were backported to earlier versions (v247 and v248) but were notably absent from v245, the version used in Ubuntu 20.04. ## Dormant Risks in the Ubuntu 22.04 Migration * Datadog began migrating its fleet from Ubuntu 20.04 to 22.04 in late 2022, eventually reaching 90% coverage across its infrastructure. * Ubuntu 22.04 utilizes systemd v249, meaning the majority of the fleet was susceptible to the routing rule flushing behavior. * The risk remained dormant during the initial rollout because `systemd-networkd` typically only starts during the initial boot sequence when no complex routing rules have been established yet. ## The Trigger: Unattended Upgrades and the CVE Patch * On March 7, 2023, a security patch for a systemd CVE was released to the Ubuntu security repositories. * Datadog’s fleet used the Ubuntu default configuration for `unattended-upgrades`, which automatically installs security-labeled patches once a day, typically between 06:00 and 07:00 UTC. * The installation of the patch forced a restart of the `systemd-networkd` service on active, running nodes. * Upon restarting, the service identified existing IP routing rules (crucial for container networking) as "foreign" and deleted them, effectively severing network connectivity for the nodes. ## Failure of Regional Isolation * Because the security patch was released globally and the automated upgrade window was synchronized across regions, the failure occurred nearly simultaneously worldwide. * This automation bypassed Datadog’s standard practice of "baking" changes in staging and experimental clusters for weeks before proceeding to production. * Nodes on the older Ubuntu 20.04 (systemd v245) were unaffected by the patch, as that version of systemd does not flush IP rules upon a service restart. To mitigate similar risks, infrastructure teams should consider explicitly disabling the management of foreign routing rules in systemd-networkd configuration when using third-party networking plugins. Furthermore, while automated security patching is a best practice, organizations must balance the speed of patching with the need for controlled, staged rollouts to prevent global configuration drift or synchronized failures.

Kubernetes Docker & Containers Systemd Ubuntu+4

datadogApr 17, 20233 min readCurated summary

Making fetch happen: Building a general-purpose query and render scheduler

Datadog rebuilt its dashboard scheduler to improve responsiveness while distributing network and rendering work more efficiently. The legacy system helped, but had grown into a complex set of roughly 20 interdependent heuristics that were difficult to maintain and poorly separated query scheduling from rendering. A simpler, general-purpose approach reduced request spikes, improved fetching performance, and created a foundation for browser-aware task scheduling. ## Limitations of the Original Scheduler - A periodic updater determined when widgets should request fresh data based on factors such as time range and browser focus. - Query tasks for visible widgets ran immediately; offscreen queries were delayed using heuristics such as pending-query counts and historical fetch durations. - Render tasks similarly prioritized visible widgets and delayed offscreen work. - The system improved performance over an unscheduled baseline by reducing main-thread work and flattening query traffic. - Over time, it accumulated around 20 parameters and interlinked rules. - Query and render concerns were mixed together: - Queries could be delayed because too many render tasks were pending. - Renders could be delayed based on data size even when browser resources were available. - The dashboard-specific implementation could not easily be reused across Datadog’s increasingly generalized widget framework. ## A General-Purpose Scheduling Strategy - Datadog separated query scheduling from render scheduling so each could be developed, tested, and rolled out independently. - The team evaluated existing heuristics across dashboards of different sizes and under different browser conditions. - Several rules were removed without harming performance: - Unfocused or occluded tabs did not need special delays because the periodic updater and browser already throttle them. - The redesign aimed to preserve two goals: - Keep query execution distributed over time. - Prioritize widgets visible to the user. - The new system was progressively deployed, first to dashboards and then to the shared data-fetching framework used across Datadog. ## Simpler Query Scheduling The new query algorithm uses a small set of straightforward rules: - Fetches for visible widgets run immediately. - Non-visible queries are ranked and executed in fixed time windows, subject to a task limit. - Query execution pauses when the number of pending fetches becomes too high. - The chosen configuration uses: - A 2,000-millisecond time window. - A maximum of 10 tasks per window. - FIFO-style ranking for offscreen queries, favoring earlier requests. - The scheduler uses only about six parameters instead of the legacy system’s roughly 20. - The simplified algorithm produced a better task distribution than the old scheduler. - “429 Too many requests” errors dropped significantly, reducing retries and helping data arrive sooner. ## Browser-Aware Render Scheduling - The old render scheduler did not account for the browser’s available CPU and memory resources. - Datadog adopted the Browser Scheduling API to create prioritized tasks that the browser can schedule natively. - Tasks can receive priorities such as: - `user-blocking` - `user-visible` - `background` - A `TaskController` assigns a priority signal to scheduled work. - Priorities can later be changed for all tasks controlled by the same controller, and tasks can be aborted. - The API was supported in Chromium and Firefox Nightly, with a polyfill for other browsers. Datadog’s experience suggests that performance schedulers benefit from simple, independently testable rules: prioritize visible work, smooth network activity, and let the browser manage expensive rendering when possible.

Read original

Database Design Performance Optimization Query Scheduling Browser Scheduling Api+4

datadogApr 17, 2023Original article

Making fetch happen: Building a general-purpose query and render scheduler | Datadog (opens in new tab)

Datadog replaced its complex, dashboard-specific scheduling system with a generalized, modular query and render scheduler to improve performance across all its web applications. By simplifying query heuristics and leveraging the Browser Scheduling API for renders, the engineering team achieved a more stable backend load and smoother UI interactions. This transition transformed a brittle set of rules into a scalable framework that optimizes resource utilization based on widget visibility and browser availability. ## Limitations of Legacy Scheduling The original scheduling system was a complex web of over 20 interlinked heuristics that became difficult for developers to maintain or reason about. While it performed better than an unscheduled baseline, it suffered from several structural flaws: * **Tight Coupling:** Query and render logic were unnecessarily linked; for example, fetches were sometimes delayed based on pending render tasks, even when throttling fetches wasn’t necessary. * **Lack of Generalization:** The system was hardcoded specifically for dashboards, making it impossible to use the same optimization benefits for other widget-heavy products in the Datadog suite. * **Inefficient Resource Management:** Renders were often delayed based on arbitrary data size rules rather than the actual real-time availability of the browser's CPU and memory resources. ## A Simplified Query Algorithm To create a more predictable and efficient system, the team stripped away redundant rules—such as manual throttling for unfocused tabs, which modern browsers already handle—and moved to a streamlined query model. The new algorithm is governed by only six parameters: * **Visibility Priority:** Fetches for widgets currently visible in the viewport are executed immediately to ensure a responsive user experience. * **Fixed Time Windows:** Non-visible queries are ranked by enqueue time and processed in 2000ms windows with a limit of 10 tasks per window. * **Error Reduction:** The more stable distribution of tasks significantly reduced "429 (Too many requests)" errors, leading to faster overall data loading since fewer retries are required. * **Framework Integration:** This simplified logic was moved into a standard data-fetching framework, allowing any Datadog product using generalized components to benefit from the scheduler. ## Render Scheduling with the Browser Scheduling API While the query scheduler handles data fetching, a separate render scheduler manages the impact on the browser’s main thread. By moving away from legacy heuristics and adopting the Browser Scheduling API, Datadog can now schedule tasks based on native browser priorities: * **Prioritization:** The API allows developers to categorize tasks as `user-blocking`, `user-visible`, or `background`, ensuring the browser prioritizes critical UI updates while deferring heavy computations to idle periods. * **Resource Awareness:** Unlike the old system, this API is natively aware of CPU and memory pressure, allowing the browser to manage execution timing more effectively than a JavaScript-based heuristic. * **Future-Proofing:** Currently supported in Chromium and Firefox Nightly (with polyfills for others), this approach allows for mass updates to task priorities and the ability to abort stale tasks via `TaskController`. Standardizing on a modular scheduling architecture allows engineering teams to optimize both network traffic and main-thread performance without the maintenance overhead of complex, custom rule sets. For high-density data applications, leveraging native browser APIs for task prioritization is recommended to ensure smooth rendering across varying hardware capabilities.

Datadog Query Scheduling Browser Scheduling Api Task Optimization+3

datadogFeb 22, 20233 min readCurated summary

Husky: Exactly-once ingestion and multi-tenancy at scale

Husky, Datadog’s distributed, time-series-oriented event store, is optimized for large scans and aggregations rather than high-volume, low-latency point lookups. This makes exactly-once ingestion challenging, especially at Datadog’s multi-tenant scale. Datadog addresses the problem with deterministic, locality-aware routing that limits deduplication scope, improves storage efficiency, and supports autoscaling ingestion pipelines. ## Husky’s Ingestion Challenge - Husky separates storage and compute, allowing each to scale independently. - Its storage engine is designed primarily for large analytical scans and aggregations. - It is not optimized for massive numbers of low-latency point lookups, complicating duplicate detection during ingestion. - The ingestion system must guarantee that every event is stored exactly once while maintaining: - Multi-tenant scalability - Reasonable ingestion latency - Controlled infrastructure and storage costs ## Routing Events to Storage Shards - Datadog uses an upstream **Shard Router** to introduce locality into Kafka pipelines. - Events are deterministically assigned to shards based on their tenant, timestamp, and event ID. - Each tenant receives a list of shards rather than being permanently assigned to one shard. - A deterministic choice from that list distributes the tenant’s events while keeping the number of active shards as small as practical. - Downstream workers consume one or more shards and perform exactly-once ingestion into Husky. ## Benefits of Data Locality - **Simpler deduplication** - An event with the same timestamp and ID always reaches the same shard. - Deduplication only needs to occur within that shard. - Workers handle smaller sets of event IDs, making in-memory deduplication more efficient. - **Lower storage costs and better performance** - Each shard processes a relatively small set of tenants. - Husky stores each tenant in a separate table and does not mix tenants within files. - More tenants per writer produce more output files, increasing blob-storage costs and compaction work. - Restricting tenant cardinality reduces file creation and improves writer and compactor efficiency. ## Challenges in Deterministic Routing - **Changing shard assignments** - Tenant traffic can increase by one or two orders of magnitude. - Assignments may change when scaling a tenant across more shards or rebalancing traffic among existing shards. - **Distributed router consensus** - Every Shard Router node must make the same routing decision for a given event. - Inconsistent decisions could send duplicates to different shards and undermine exactly-once ingestion. - **Load balancing** - Shards must receive roughly equal traffic so downstream ingestion workers remain balanced. ## Time-Bounded Shard Placements - A simple deterministic mapping can select a shard using a hash of the event ID: ```text shard = shards[hash(event_id) % num_shards] ``` - This approach is cheap and stateless when all routers know the same shard list. - However, changing the shard list can cause the same event ID to map to a different shard, so assignment changes require additional coordination. - The article introduces **time-bounded Shard Placements** to preserve consistent routing while allowing tenant assignments to evolve, though the supplied excerpt ends before explaining the mechanism in detail. Datadog’s core recommendation is to combine deterministic, tenant-aware routing with carefully coordinated assignment changes. This narrows the scope of deduplication while reducing storage overhead and enabling balanced, scalable exactly-once ingestion.

Read original

Database Design Apache Kafka Distributed Systems Load Balancing+4

Filter by tag