Improving cloud security visibility with ChatOps
Datadog built a largely serverless AWS security monitoring pipeline to detect suspicious API activity across more than 15 AWS accounts. Rather than process every CloudTrail event equally, it categorizes actions as log, notify, or alert, reducing false positives while preserving broad visibility. The system centralizes events, batches activity, and uses automated workflows to verify engineers’ actions or escalate potential compromises. ## The Security Monitoring Challenge - Datadog manages extensive AWS infrastructure across multiple accounts and nearly 200 geographically distributed engineers. - Every console or CLI operation generates an AWS API call, creating a high-volume CloudTrail data stream. - The monitoring system must detect malicious activity while also catching accidental exposure caused by configuration mistakes. - Processing every event manually would require an expensive, constantly staffed Security Operations Center. ## Prioritizing Relevant API Calls - Datadog maintains a focused list of security-relevant AWS API calls and assigns each to one of three categories: - **Log:** Lower-risk events retained for investigation, such as `CreateGroup` or `UpdateUser`. - **Notify:** Events that require the initiating engineer to confirm their identity and intent, such as `CreateUser` or `PutUserPolicy`. - **Alert:** Rare, dangerous, or clearly misconfigured actions sent directly to the security team. - A representative alert is `AuthorizeSecurityGroupIngress` with `0.0.0.0/0`, which exposes an EC2 security group to the entire Internet. - User verification reduces false positives and helps identify compromised AWS credentials. ## Cross-Account Event Pipeline - CloudTrail records API activity in each AWS account. - CloudWatch Event Rules filter for the selected API calls and publish matching events to SNS. - SNS forwards events across accounts to an SQS queue in a dedicated security AWS account. - Centralization is necessary because CloudWatch could not directly send events cross-account to SQS. - The queue also supports batching, which is important when Terraform generates many AWS changes in a short period. - A CloudWatch rule triggers a Lambda function every two minutes to drain the SQS queue and forward events to the security orchestration layer. ## Automated Decision-Making with Komand - Datadog uses Komand, a security orchestration and automation platform, to construct workflows from built-in and custom plugins. - A custom decision plugin evaluates: - The calling user - The event’s age - Request parameters and their content - Other contextual details - Based on the analysis, the workflow silently logs the event, notifies the engineer, or pages the security team through PagerDuty. ## Engineer Verification and Escalation - For notification-level events, the engineer receives an interactive Slack message containing API call details. - Confirming the action triggers a Duo push for second-factor identity verification. - If the engineer denies the action or fails to respond promptly, the workflow alerts the security team. - Komand coordinates the Slack, Duo, PagerDuty, and custom integration logic in one centralized workflow. ## Visibility and Continuous Improvement - Every workflow execution is logged and sent to Elasticsearch. - The resulting data helps Datadog visualize security events, measure detection effectiveness, identify behavioral trends, and improve alerting. - The pipeline is designed to provide actionable security intelligence without overwhelming engineers or security personnel. Datadog’s approach combines selective event filtering, cross-account centralization, batching, and automated identity verification. Organizations facing similar AWS-scale monitoring challenges can use the same principles to reduce alert fatigue while maintaining strong detection and response capabilities.
Read original(opens in new tab)