Anomaly Detection | Techlist.io

cloudflareAug 5, 20264 min readCurated summary

Catching rogue AI behavior with identity-aware analytics

AI usage is difficult to govern without knowing both who made each request and what normal usage looks like for that person or agent. Cloudflare’s new Identity-aware AI Gateway and User Insights address this by attaching verified identities to requests and detecting behavior that significantly deviates from historical patterns. Together, they provide centralized visibility, per-user cost controls, and anomaly detection without requiring additional setup for traffic already routed through AI Gateway. ## AI Gateway as a Central Control Plane - AI Gateway routes requests from applications, developer tools, and agent harnesses—including Claude Code, Codex, and GitHub Copilot—through one platform. - It provides centralized observability, security, governance, and spend management across providers such as OpenAI, Anthropic, Google, and Workers AI. - This centralization makes it possible to analyze usage consistently across both human users and automated agents. ## Identity-Aware Requests with Cloudflare Access - The Cloudflare Access integration places a custom domain, such as `ai.example.com`, in front of the gateway. - Organizations can: - Authenticate users through SAML-compatible providers such as Okta or Microsoft Entra. - Apply access policies to specific users. - Avoid distributing Cloudflare API keys. - Each authenticated request includes the Access user ID as `cf.user_id`. - Administrators can filter logs, analytics, and spending by the actual requester rather than by a shared API key. - Per-user spend limits can assign each person a separate budget and either block requests or route them to cheaper models after the limit is reached. - Planned improvements will use identity-provider groups to control model access and spending—for example, granting frontier-model access to machine learning teams while limiting support teams. ## User Insights and Behavioral Baselines - User Insights is available to all AI Gateway customers at no extra cost. - It analyzes existing gateway traffic without requiring additional configuration. - The feature builds behavioral profiles for every account, including both people and agents. - It tracks cost inefficiencies such as poor cache-hit rates and oversized context windows, but focuses primarily on whether usage is normal for that particular account. - Human users and automated agents are evaluated according to their own patterns: - Agents may have regular, predictable sessions. - Humans typically have more irregular prompts, timing, and session lengths. ## Session-Based Anomaly Detection - User Insights evaluates sessions rather than individual requests, reducing noise from isolated events. - Each session is compared with the account’s rolling 95th-percentile session cost over the previous 30 days. - A session becomes a strong anomaly candidate when it exceeds twice that personal p95 baseline. - This relative comparison avoids misleading fixed thresholds: - A $500 session may be normal for a consistently heavy user. - A $50 session may be highly unusual for an agent that normally spends $5. - Baselines adjust over time as an account’s usage changes. ## Combining Personal and Organization-Wide Thresholds - User Insights also applies an organization-wide p99 cost ceiling. - In the example analysis: - Most sessions cost less than $10. - The organizational p95 is $20. - The p99 is $200, meaning only 1% of sessions reach that amount. - Alerts are triggered only when a session is both: - More than twice the account’s personal p95. - Above the organization’s p99 ceiling. - This prevents alerts for: - Small-dollar spikes that are statistically unusual but not worth investigating. - Expensive sessions that are routine for a particular user. - A dollar floor also prevents tiny accounts from triggering alerts because of insignificant percentage increases. ## Filtering for Rogue Behavior - The resulting interface presents a feed of accounts that have broken their established usage patterns. - This focuses administrators on potentially meaningful incidents instead of showing every unusual request. - The approach is designed to detect trusted users or agents that suddenly perform more of an already-authorized activity—behavior that traditional controls may not block because no new tool or forbidden action is involved. Cloudflare’s recommendation is to route AI traffic through AI Gateway, authenticate it with Cloudflare Access, and use identity-based budgets alongside behavioral baselines. This combination helps organizations connect spending and activity to specific people or agents while concentrating investigations on statistically significant, high-impact deviations.

Read original

Anomaly Detection Ai Security Ai Gateway Cloudflare Access+4

kakaoMar 15, 20264 min readCurated summary

Finding Real Threats Among Hundreds of Millions of Security Signals — Transforming the Security Monitoring Paradigm with AI

Kakao argues that monitoring hundreds of millions of daily security events cannot scale through human analysts and increasingly complex rules alone. Its solution is a hybrid AI pipeline that filters noise early, analyzes only high-value events with multiple models, and continuously improves through verified feedback. The goal is not to generate more alerts, but to understand context and identify threats worth investigating. ## The Scale Problem: Finding Threats in a Haystack - Endpoint activity such as process execution, network connections, file changes, and privilege escalation produces hundreds of millions of events. - The volume grows rapidly as services expand, while the proportion of genuine attacks remains very small. - Increasing the number of analysts alongside event volume is economically and operationally unsustainable. - AI is needed to correlate events, interpret behavior statistically and contextually, and dynamically distinguish normal activity from anomalies. ## Limitations of Rule-Based Monitoring - Rules can identify what happened, but not why, who initiated it, or whether it fits the environment. - Legitimate deployment commands can resemble backdoor installation, causing high false-positive rates. - Analysis quality varies by analyst experience, shift, and time of day. - Analysts must manually assemble host information, network sessions, process histories, and related logs into an incident narrative. - Expanding detection categories—behavior sequences, statistical anomalies, multi-source correlations, and rare events—makes manual rule maintenance impractical. - SIEM correlation improves on single-event rules but remains limited to predefined scenarios and struggles with unknown attack patterns. - As rule sets and event volumes grow, both maintenance costs and matching performance become problematic. ## A Funnel-Based Hybrid Architecture - Kakao filters events through multiple stages before using AI: - Rule-based filters remove obvious noise. - Learned normal patterns are automatically excluded. - AI performs detailed analysis only on the small remainder requiring judgment. - Rules handle clear, deterministic patterns quickly, while AI evaluates complex contextual situations. - The framework is designed to accommodate new threat types and detection categories without creating a separate system for each scenario. ## Multi-Model Verification and Operational Resilience - Multiple AI models independently analyze the same event and cross-check one another. - Disagreement is treated as an uncertainty signal that can trigger deeper analyst review. - Model diversity helps reduce bias, false positives, and missed detections. - It also provides resilience against model failures, API outages, and quality changes after model updates. - The design balances cost, processing speed, and accuracy rather than optimizing only for detection precision. ## Teaching AI the Environment’s Context - Generic LLMs initially misclassified legitimate activity because they lacked knowledge of Kakao’s infrastructure. - The system supplies structured context, including: - Host roles - Services running on each host - Accounts used for automation - Normal communication and operational patterns - This context allows the model to act more like an analyst familiar with the organization than a generic security classifier. ## Analyzing Complete Behavior Flows - Individual commands such as `curl`, `chmod`, and script execution can occur in both normal deployments and attacks. - Kakao therefore reconstructs activity at the host level, linking: - Process execution history - Network sessions - File changes - Temporal ordering - The same command can have different meanings depending on when, where, and in what sequence it occurred. - AI evaluates the complete sequence to distinguish routine operations from intrusion behavior. ## Translating Events into AI-Usable Data - Sending raw events directly to an LLM wastes tokens on irrelevant information and reduces accuracy. - Different detection tasks require different signals; statistical anomaly detection and sequence analysis cannot rely on one fixed format. - Kakao introduced: - A standardized event schema - Dynamic feature construction tailored to each detection type - This reduces token usage while improving the relevance and precision of AI analysis. ## WALT: A Self-Learning Detection Loop - Initially, analysts had to manually convert AI conclusions into new detection policies. - Kakao developed WALT, or **Whitelist-Assisted Learning and Tuning**, to automate this feedback process. - Repeatedly verified normal patterns are converted into exception policies. - Those policies filter future matching events before they reach the AI engine. - Thousands of detection policies are reportedly being generated and operated this way, allowing accuracy to improve over time. ## Cost and Performance Constraints - Sending every event to an AI model caused unsustainable costs and processing delays. - The funnel architecture addresses this by reserving expensive AI analysis for events that survive earlier filtering. - The overall system must continuously balance economic cost, response speed, detection accuracy, and reliability. Kakao’s practical recommendation is to treat AI as part of a carefully designed security pipeline—not as a replacement for rules or analysts. Effective large-scale monitoring combines deterministic filtering, contextual multi-model analysis, structured data, and a controlled feedback loop that learns from verified outcomes.

Read original

Anomaly Detection AI Large Language Models Security Monitoring+4

cloudflareFeb 27, 20263 min readCurated summary

Toxic combinations: when small signals add up to a security incident

Small security signals can become dangerous when combined: automated bots probing sensitive paths, unusual request behavior, and weak authentication or configuration. Cloudflare argues that analyzing these signals together—rather than judging each request independently—can reveal likely attack campaigns before compromise. Although toxic combinations are uncommon outside WordPress, the affected hosts may be highly exposed. ## What “Toxic Combinations” Mean - A toxic combination occurs when attackers compound several minor weaknesses into a viable breach. - Relevant signals include: - Bot activity and automated scanning - Sensitive paths such as `/admin`, `/debug`, `/metrics`, search, and payment endpoints - Anomalies such as unexpected HTTP status codes, geographic jumps, identity mismatches, high identifier churn, distributed rate-limit evasion, and traffic spikes - Missing session cookies or authorization headers and predictable identifiers - Traditional WAF, bot, and API defenses often assess the risk of individual requests. - Cloudflare’s approach examines the broader context across multiple requests, hosts, and paths. ## Measuring Exposure - Cloudflare analyzed a 24-hour sample of application-security data. - About 11% of analyzed hosts appeared susceptible to toxic combinations, largely because of vulnerable WordPress sites. - Excluding WordPress, only about 0.25% of hosts showed signs of exploitable combinations. - The analysis separated attacks into three stages: - **Hosts probed:** systems receiving requests for sensitive paths such as `/wp-admin` - **Hosts matching a toxic combination:** systems meeting the full detection criteria - **Reachable hosts:** systems that successfully responded to an exploit attempt - A `200 OK` response alone is not proof of exposure. Cloudflare recommends validating results against authentication requirements, redirects, and origin configurations to eliminate false positives. ## Probing Administrative Endpoints - Automated scanners targeted common administrative interfaces, including: - WordPress `/wp-admin` pages - Database management tools - Server dashboards - Cloudflare’s Log Explorer query groups successful requests by host, filters for likely bot traffic using a low bot score, and searches for configurable path patterns. - The query also excludes hosts represented only by raw IP addresses unless that filter is removed. ## Why Public Admin Panels Are Dangerous - Exposed administrative panels enable brute-force login attempts. - A successful compromise can allow attackers to: - Identify software and versions such as WordPress or Tomcat - Search for relevant CVEs and launch targeted exploits - Add the compromised host to a botnet that scans other websites - A sensitive endpoint returning successfully should therefore be tested for actual reachability and authentication weakness, not treated as conclusive evidence on its own. ## Practical Recommendation Monitor combinations of bot activity, sensitive-path access, anomalous behavior, and missing authentication signals. Investigate confirmed reachable endpoints, restrict or protect administrative interfaces, remove debug exposure, and validate detection queries against real application behavior to distinguish exploitable systems from false positives.

Read original

Anomaly Detection Cloudflare Vulnerability Management Application Security+3

metaDec 19, 2025Original article

DrP: Meta's Root Cause Analysis Platform at Scale (opens in new tab)

DrP is Meta’s programmatic root cause analysis (RCA) platform designed to automate incident investigations and reduce the burden of manual on-call tasks. By codifying investigation playbooks into executable "analyzers," the platform significantly lowers the mean time to resolve (MTTR) by 20% to 80% for over 300 teams. This systematic approach replaces outdated manual scripts with a scalable backend that executes 50,000 automated analyses daily, providing immediate context when alerts fire. ## Architecture and Core Components * **Expressive SDK:** Provides a framework for engineers to codify investigation workflows into "analyzers," utilizing a rich library of helper functions and machine learning algorithms. * **Built-in Analysis Tools:** The platform includes native support for anomaly detection, event isolation, time-series correlation, and dimension analysis to identify specific problem areas. * **Scalable Backend:** A multi-tenant execution environment manages a worker pool that handles thousands of requests securely and asynchronously. * **Workflow Integration:** DrP is integrated directly into Meta’s internal alerting and incident management systems, allowing for automatic triggering without human intervention. ## Authoring and Verification Workflow * **Template Bootstrapping:** Engineers use the SDK to generate boilerplate code that captures required input parameters and context in a type-safe manner. * **Analyzer Chaining:** The system allows for seamless dependency analysis by passing context between different analyzers, enabling investigations to span multiple interconnected services. * **Automated Backtesting:** Before deployment, analyzers undergo automated backtesting integrated into the code review process to ensure accuracy and performance. * **Decision Tree Logic:** Investigation steps are modeled as decision trees within the code, allowing the analyzer to follow different paths based on the data it retrieves. ## Execution and Post-Processing * **Trigger-based Analysis:** When an alert is activated, the backend automatically queues the relevant analyzer, ensuring findings are available as soon as an engineer begins triaging. * **Automated Mitigation:** A post-processing system can take direct action based on investigation results, such as creating tasks or submitting pull requests to resolve identified issues. * **DrP Insights:** This system periodically reviews historical analysis outputs to identify and rank the top causes of alerts, helping teams prioritize long-term reliability fixes. * **Alert Annotation:** Results are presented in both human-readable text and machine-readable formats, directly annotating the incident logs for the on-call responder. ## Practical Conclusion Organizations managing large-scale distributed systems should transition from static markdown playbooks to executable investigation code. By implementing a programmatic RCA framework like DrP, teams can scale their troubleshooting expertise and significantly reduce "on-call fatigue" by automating the repetitive triage steps that typically consume the first hour of an incident.

Anomaly Detection AI Machine Learning Observability+4

woowahanDec 2, 2025Original article

How Woowa Brothers Detects (opens in new tab)

Woowa Brothers addresses the inevitability of system failures by shifting from traditional resource-based monitoring to a specialized Service Anomaly Detection system. By focusing on high-level service metrics such as order volume and login counts rather than just CPU or memory usage, they can identify incidents that directly impact the user experience. This approach ensures near real-time detection and provides a structured response framework to minimize damage during peak service hours. ### The Shift to Service-Level Monitoring * Traditional monitoring focuses on infrastructure metrics like CPU and memory, but it is impossible to monitor every system variable, leading to "blind spots" in failure detection. * Service metrics, such as real-time login counts and payment success rates, are finite and offer a direct reflection of the actual customer experience. * By monitoring these core indicators, the SRE team can detect anomalies that system-level alerts might overlook, ensuring that no failure goes unnoticed. ### Requirements for Effective Anomaly Detection * **Real-time Performance:** Alerts must be triggered in near-real-time to allow for immediate intervention before the impact scales. * **Explainability:** The system favors transparent logic over "black-box" AI models, allowing developers to quickly understand why an alert was triggered and how to improve the detection logic. * **Integrated Response:** Beyond just detection, the system must provide a clear response process so that any engineer, regardless of experience, can follow a standardized path to resolution. ### Technical Implementation and Logic * The system leverages the predictable, pattern-based nature of delivery service traffic, which typically peaks during lunch and dinner. * The team chose a Median-based approach to generate "Prediction" values from historical data, as it is more robust against outliers and easier to analyze than complex methods like IQR or 2-sigma. * Detection is determined by comparing "Actual" values against "Warning" and "Critical" thresholds derived from the predicted median. * To prevent false positives caused by temporary spikes, the system tracks "threshold reach counts," requiring a metric to stay in an abnormal state for a specific number of consecutive cycles before firing a Slack alert. ### Optimization of Alert Accuracy * Each service metric requires a tailored "settling period" to find the optimal balance between detection speed and accuracy. * Setting a high threshold reach count improves accuracy but slows down detection, while a low count accelerates detection at the risk of increased false positives. * Alerts are delivered via Slack with comprehensive context, including current status and urgency, to facilitate rapid decision-making. For organizations running high-traffic services, prioritizing service-level indicators (SLIs) over infrastructure metrics can significantly reduce the time to detect critical failures. Implementing simple, explainable statistical models like the Median approach allows teams to maintain a reliable monitoring system that evolves alongside the service without the complexity of uninterpretable AI models.

Anomaly Detection Monitoring Incident Management Sre+2

naverNov 27, 2025Original article

Naver TV (opens in new tab)

Naver’s Integrated Search team is transitioning from manual fault response to an automated system using LLM Agents to manage the increasing complexity of search infrastructure. By integrating Large Language Models into the DevOps pipeline, the system evolves through accumulated experience, moving beyond simple alert monitoring to intelligent diagnostic analysis and action recommendation. ### Limitations of Traditional Fault Response * **Complex Search Flows:** Naver’s search architecture involves multiple interdependent layers, which makes manual root cause analysis slow and prone to human error. * **Fragmented Context:** Existing monitoring requires developers to manually synthesize logs and metrics from disparate telemetry sources, leading to high cognitive load during outages. * **Delayed Intervention:** Human-led responses often suffer from a "detection-to-action" lag, especially during high-traffic periods or subtle service regressions. ### Architecture of DevOps Agent v1 * **Initial Design:** Focused on automating basic data gathering and providing preliminary textual reports to engineers. * **Infrastructure Integration:** Built using a specialized software stack designed to bridge frontend (FE) and backend (BE) telemetry within the search infrastructure. * **Standardized Logic:** The v1 agent operated on a fixed set of instructions to perform predefined diagnostic tasks when triggered by specific system alarms. ### Evolution to DevOps Agent v2 * **Overcoming V1 Limitations:** The first iteration struggled with maintaining deep context and providing diverse actionable insights, necessitating a more robust agentic structure. * **Enhanced Memory and Learning:** V2 incorporates a more sophisticated architecture that allows the agent to reference historical failure data and learn from past incident resolutions. * **Advanced Tool Interaction:** The system was upgraded to handle more complex tool-calling capabilities, allowing the agent to interact more deeply with internal infrastructure APIs. ### System Operations and Evaluation * **Trigger Queue Management:** Implements a queuing system to efficiently process and prioritize multiple concurrent system alerts without overwhelming the diagnostic pipeline. * **Anomaly Detection:** Utilizes advanced detection methods to distinguish between routine traffic fluctuations and genuine service anomalies that require LLM intervention. * **Rigorous Evaluation:** The agent’s performance is measured through a dedicated evaluation framework that assesses the accuracy of its diagnoses against known ground-truth incidents. ### Scaling and Future Challenges * **Context Expansion:** Efforts are focused on integrating a wider range of metadata and environmental context to provide a holistic view of system health. * **Action Recommendation:** The system is moving toward suggesting specific recovery actions, such as rollbacks or traffic rerouting, rather than just identifying the problem. * **Sustainability:** Ensuring the DevOps Agent remains maintainable and cost-effective as the underlying search infrastructure and LLM models continue to evolve. Organizations managing high-scale search traffic should consider LLM-based agents as integrated infrastructure components rather than standalone tools. Moving from reactive monitoring to a proactive, experience-based agent system is essential for reducing the mean time to recovery (MTTR) in complex distributed environments.

Anomaly Detection AI Large Language Models DevOps+3

dropboxNov 26, 20253 min readCurated summary

Building the future: highlights from Dropbox’s 2025 summer intern class

Dropbox’s 2025 intern program brought together 43 interns from 27 universities for 12 weeks of mentorship, technical work, and community-building. The 28 engineering interns contributed to systems spanning AI, search, storage, data infrastructure, and developer tools, with many projects supporting Dropbox Dash. Their work demonstrates how targeted refactoring, automation, and intelligent infrastructure can improve reliability, reduce costs, and expand product capabilities. ## A Diverse, Mentorship-Focused Internship Program - Interns received more than 6,000 hours of one-on-one mentorship. - The cohort included students from institutions in the United States, Canada, Poland, and Ireland. - Programming included Virtual First events, employee resource group activities, and an in-person Emerging Talent Summit. - Projects were aligned with Dropbox’s production systems and company goals rather than being isolated experiments. ## Infrastructure and Reliability Improvements - **Filesystem Data:** Rhea Rai redesigned Dropbox’s file history tracking system, emphasizing strongly tested code and simpler metadata infrastructure while reducing operational costs. - **Storage Core:** Albert Joon Sung reduced Magic Pocket write latency during disk restarts by adding storage-health caching and routing writes away from degraded volumes. - **Metrics:** Yonatan Ginsburg developed adaptive anomaly detection for Vortex2, accounting for changing patterns and seasonality to reduce alert fatigue and improve incident response. - **Analytics Platform:** Sanjith Udupa built recommendations for optimizing Databricks queries and ETL pipelines, and documented a plan to migrate a 500 TB mobile-events dataset to liquid clustering. ## AI, Search, and Dropbox Dash - **ML Platform:** Ben Juntilla created AI Sentinel, which gives engineers real-time visibility into machine-learning deployment health and improves confidence in model releases. - **Connector Platform:** Eddie Ormseth built tools that provide access to fresher Dash persistence data and additional third-party metadata without requiring connector teams to reprocess data. - **Retrieval Platform:** Rishi Peddakama expanded unified search to more than 20 languages by integrating language detection into indexing and retrieval. - **Find & Discover:** Francesca Venditti created in-context document previews for Dash, including PDF viewing and links to AI-powered follow-up chat. - **Conversational AI:** Alan Zhu developed a modular web-automation agent and connected it to Dropbox APIs for actions such as searching for and uploading files. ## Developer Automation - Ahmed Ibrahim built an AI-assisted code migration tool on Dropbox’s internal migration platform. - Developers can run migrations on selected folders, configure them through a CLI or automated workflow, and receive pull requests automatically when jobs succeed. - The tool enabled two major migrations during the internship and illustrates how automation can reduce repetitive engineering work. ## Broader Impact The interns’ projects improved system performance, operational visibility, multilingual accessibility, data freshness, and developer productivity. Together, they supported Dropbox’s move toward AI-first products while reinforcing the company’s emphasis on trustworthy, maintainable, and efficient engineering. For students interested in production-scale software, AI, and infrastructure, Dropbox presents its internship program and open roles as opportunities to contribute directly to products such as Dropbox Dash.

Read original

Anomaly Detection AI Machine Learning Distributed Systems+4

slackSep 4, 20253 min readCurated summary

Building Slack’s Anomaly Event Response

Slack’s Anomaly Event Response (AER) is designed to close the gap between detecting suspicious activity and stopping it. By combining real-time monitoring, adaptive analytics, and automated session termination, AER can disrupt high-confidence attacks within minutes rather than hours or days. Slack presents it as a built-in security capability for Enterprise Grid customers that works without additional tools or security staff. ## Shared Responsibility for Securing Slack - Slack processes billions of daily interactions from tens of millions of weekly users. - Enterprise customers receive audit logs covering hundreds of platform actions. - Specialized anomaly logs flag activity such as: - Irregular logins - Malware uploads - Unexpected data transfers - Audit logs provide early warning but traditionally require security personnel or third-party systems to interpret and act on them. - AER provides automated response for customers that lack the resources or infrastructure to build those integrations. - Advanced customers can still combine AER with customized security controls. ## Configurable Threat Detection AER focuses on common indicators of account compromise, data exfiltration, and automated abuse: - Access from Tor exit nodes - Excessive downloading - Data scraping through non-native automation tools - Session fingerprint mismatches - Unexpected API-call volumes or patterns - Unusual user agents, including virtual or non-standard clients Organizations can choose which anomaly types should terminate sessions and which should only be logged. Notification settings are also configurable, with alerts available for organization owners and security administrators through email or Slack. ## Detection Engine - The detection engine analyzes billions of Slack events each day. - It combines rule-based heuristics with dynamic thresholds. - Thresholds are calibrated to each enterprise’s historical usage patterns. - This prevents normal high-volume activity in one organization from being treated as anomalous in another. - Adaptive thresholds help reduce false positives while allowing Slack to refine detection sensitivity over time. ## AER Architecture AER consists of three main components: - **Detection engine:** Identifies suspicious activity and creates anomaly audit payloads. - **Decision framework:** Validates detected behavior and determines whether it qualifies for automated response. - **Response orchestrator:** Carries out the configured response, including terminating user sessions. The overall flow is: 1. Suspicious user activity is analyzed. 2. An anomaly is detected. 3. The AER controller determines whether it is a supported anomaly and whether the organization’s settings require action. 4. Associated user sessions may be terminated. 5. The event is always recorded in audit logs. 6. Customer notifications are sent according to configured preferences. AER’s practical value is that it turns anomaly detection into immediate containment, helping organizations interrupt attack chains before attackers can complete data theft or compromise.

Read original

Anomaly Detection Cybersecurity Threat Detection Session Management+3

datadogJan 22, 20193 min readCurated summary

Rethinking UX for AI-driven alerting

Alerting UX is shifting from manually configured static thresholds toward statistical systems that understand trends, seasonality, and group behavior. Forecasting, anomaly detection, and outlier detection reduce maintenance and nuisance alerts, while algorithmic feeds can discover problems users never explicitly configured. The post argues that supervised feeds—trained by user feedback—could become the most significant evolution in monitoring. ## Traditional Alerting UX Most alerts are built from four dimensions: - **Scope:** The hosts, containers, services, or other targets being monitored. - **Metric:** The value tracked, such as free disk space. - **Thresholds:** Static warning or critical values that trigger alerts. - **Time:** A duration or time window during which the condition must occur. Static threshold alerts remain common, with many taking the form: “If free disk space equals zero, alert.” ## Problems with Static Thresholds - Static thresholds do not adapt to system growth, changing workloads, or temporary events such as holiday traffic. - They require regular review and maintenance to remain useful. - Warning thresholds often serve as manual “heads-ups” so engineers can inspect a graph and estimate whether intervention is necessary. - Large numbers of warning alerts create false positives and alert fatigue. - Monitoring systems must be explicitly told which scopes and metrics to watch, leading to duplicated configuration and ongoing maintenance. ## Algorithmic Alerting Statistical alerting introduces three primary methods: - **Forecasting** - Uses historical data to predict when a metric will cross a threshold. - Changes “alert when disk reaches zero” into “alert if disk will reach zero within 24 hours.” - Lets teams specify how much remediation time they need. - Can eliminate separate warning thresholds because the forecast provides advance notice. - **Anomaly detection** - Predicts what should be happening now based on historical behavior. - Considers configurable confidence intervals and seasonality, such as daily or weekly patterns. - Alerts when current behavior deviates significantly from the expected range. - **Outlier detection** - Compares members of a group that should behave similarly. - Flags an individual server or service whose behavior differs from its peers. - Does not depend on historical behavior. These methods make thresholds and time behavior more flexible, but they still require users to define the metrics and scopes in advance. ## Algorithmic Feeds Algorithmic feeds apply similar statistical techniques without requiring detailed alert configuration. - They can monitor systems without predefined individual scopes or metrics. - They are especially useful for unpredictable anomalies and outliers. - Examples include Slack Highlights and Datadog Watchdog. - Feeds shift monitoring from **opt-in alerting**—where users specify what to watch—to discovering noteworthy activity automatically. - The post presents this shift as potentially the largest change in alerting UX, while noting that algorithmic feeds are still immature. ## Supervised Algorithmic Feeds Once a monitoring system generates a stream of events, user feedback can help train it to surface more relevant information. - The model is compared to social media feeds, where actions such as “likes” guide future recommendations. - This suggests a future in which engineers can teach monitoring systems which anomalies and events matter to them. - The provided excerpt ends while introducing this concept, so it does not describe the specific feedback mechanisms or implementation details. Monitoring is likely moving toward adaptive systems that combine statistical detection with user-guided prioritization. Teams should use forecasting and anomaly-based alerts where appropriate, while treating algorithmic feeds as a complementary way to discover issues outside manually configured monitoring.

Read original

Anomaly Detection Database Design Datadog Cloud Infrastructure+4

datadogSep 6, 20173 min readCurated summary

Robust statistical distances for machine learning

Statistical distances provide quantitative ways to measure how similar or different data distributions are, complementing visual tools such as histograms and Q-Q plots. The post compares the Kolmogorov-Smirnov, Earth Mover’s, and Cramér-von Mises distances, showing that each responds differently to local changes, long tails, and shifts in distribution. No single metric is universally best; the appropriate choice depends on which differences matter most. ## Visual Inspection and Q-Q Plots - Histograms offer a quick comparison of: - Minimum and maximum values - Center or average - Spread and overall shape - Q-Q plots sort both datasets and plot corresponding values against one another. - Points close to the first bisector indicate that the datasets likely come from similarly shaped distributions. - Visual methods are useful heuristics but do not provide a precise quantitative distance. ## Kolmogorov-Smirnov Distance - The KS distance compares empirical cumulative distribution functions (CDFs). - It is the largest absolute difference between the two CDFs at any point. - It is a true metric: - It is nonnegative. - It is zero only for identical distributions. - It is symmetric. - It satisfies the triangle inequality. - The distance is bounded between 0 and 1. - This makes it convenient for determining whether distributions are similar, but less useful for measuring how far apart very different distributions are. - For normal distributions with equal standard deviations and increasingly separated means, KS quickly levels off rather than growing proportionally. ## Earth Mover’s Distance - Earth Mover’s Distance (EMD), or the first Wasserstein distance, measures the minimum work required to transform one distribution into another. - “Work” is the amount of mass moved multiplied by the distance it travels. - EMD is equivalent to the area between the two empirical CDFs. - It is particularly useful for distributions with long or significant tails because distant mass contributes substantially to the result. - Unlike KS, EMD is unbounded and can grow with the physical separation between distributions. ## Cramér-von Mises Distance - The Cramér-von Mises (CM) distance sums the squared differences between empirical CDFs and takes the square root. - Its relationship to EMD resembles the relationship between L2 and L1 norms. - CM is less sensitive to isolated local changes than KS but more sensitive than EMD. - For normal distributions with separated means: - EMD grows linearly with the separation. - KS rapidly reaches a plateau. - CM grows approximately like the square root of the separation. ## Sensitivity to Distribution Changes - KS focuses on the maximum CDF difference, making it highly sensitive to local deformations. - EMD averages absolute CDF differences, so localized changes may have little effect. - CM provides a compromise, detecting local changes without reacting as strongly as KS. - In examples involving shifted or localized probability mass: - KS can increase dramatically from a small local deformation. - EMD may barely change. - CM usually shows a moderate increase. - Conversely, changes in distant high-percentile regions may have little effect on KS while producing much larger changes in EMD and CM. Choose the distance according to the type of distributional difference you need to detect: KS for maximum local deviation, EMD for overall displacement and long tails, and CM for a balanced sensitivity to both global and local changes.

Read original

Anomaly Detection Data Visualization Statistical Distances Kolmogorov Smirnov+3