Statistical Analysis | Techlist.io

cloudflareAug 13, 20262 min readCurated summary

Total eclipse of the Internet: traffic impacts in Iceland, Spain, and Portugal

The August 12, 2025 total solar eclipse caused a measurable, temporary decline in Internet activity across Europe. Cloudflare Radar data shows that HTTP traffic dropped most sharply when the eclipse reached maximum obscuration, especially in countries along the path of totality. Traffic generally returned to normal within minutes as people resumed using their devices. ## Traffic Drops Matched Eclipse Timing - Cloudflare analyzed HTTP requests in five-minute intervals across affected countries. - Traffic reductions aligned closely with each location’s moment of maximum eclipse. - The strongest declines occurred in Iceland, Ireland, the UK, France, Spain, and Portugal. - Countries with only shallow partial eclipses, including Sweden, Denmark, Poland, and Switzerland, saw little or no decline. - Regions experiencing deep eclipses recorded traffic drops of roughly 15% to 30%. - Traffic typically rebounded shortly after maximum obscuration. ## Eclipse Depth Predicted Internet Activity - Researchers compared each country’s peak solar obscuration with its average traffic change during the surrounding 15-minute window. - The results showed a clear downward relationship: greater obscuration generally produced larger traffic declines. - Local factors such as population density, cloud cover, and time of day caused some variation, but the precise timing supported the eclipse as the primary cause. - Solar obscuration was calculated geometrically using the apparent sizes and positions of the sun and moon, measuring how much of the sun’s disk was covered every five minutes. ## Iceland, Spain, and Portugal Saw the Largest Declines - Country-level traffic changes ranged from a 9.3% increase to a 46.7% decrease. - Iceland, Spain, and Portugal experienced the most dramatic reductions. - Norway and Sweden saw slight increases above normal levels. - Denmark experienced the smallest overall change, while Poland quickly returned to baseline. - Eclipse-day traffic was compared with the median traffic from the three previous Wednesdays, using matching times of day to reduce the effect of unusual weekly patterns. ## Physical Events Reshape Digital Behavior - The findings show that Internet traffic reflects where people direct their attention. - The eclipse reduced online activity because people temporarily stopped using their devices to observe it, not because of technical network problems. - Traffic normalized quickly afterward, demonstrating how a shared real-world event can create a continent-wide but short-lived shift in digital behavior. - Cloudflare Radar can be used to study similar changes during major global events.

Read original

Statistical Analysis Data Visualization Cloudflare Radar Http+2

discordApr 24, 20263 min readCurated summary

Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters

Discord argues that experiments should measure fewer, higher-quality metrics rather than automatically collecting every potentially useful signal. Large metric sets increase compute and cognitive costs while creating a tradeoff between false positives and missed real effects. Multiple-testing corrections such as Benjamini–Hochberg reduce false discoveries but also lower recall, so the most effective solution is selecting metrics that represent distinct, important concepts. ## The Cost of Measuring Too Much - Discord’s “Default Metric List” gradually expanded as teams added metrics and rarely removed them. - More metrics create: - Higher compute costs - More difficult experiment readouts - Increased risk of false positives - With 100 metrics and an uncorrected significance threshold of 0.05, roughly five metrics may appear significant purely by chance. - Correcting for multiple comparisons reduces false alarms but makes genuine changes harder to detect. ## The Multiple Comparisons Problem - Discord uses the Benjamini–Hochberg (BH) procedure to control the false discovery rate at 5%. - BH ranks p-values and compares each one with a rank-specific threshold: `i × α / n` where `i` is the metric’s rank, `α` is 0.05, and `n` is the total number of metrics. - A metric with an unadjusted p-value of 0.038 might be significant without correction but fail after BH adjustment. - BH treats all metrics equally because it has no information about which ones are more likely to reflect a real effect. - The resulting tradeoff is: - Fewer false alarms - Lower recall for real changes - The article notes that Bayesian methods could potentially incorporate prior knowledge, but Discord’s default system is frequentist. ## Simulation Results - Discord simulated 50,000 experiments containing: - Twenty null metrics generated from `N(0, 1)` - One metric with a real effect centered at `z = 2.8` - The simulations tested how metric count affects: - Experiment-level false alarm rates - Recall of the metric with the known effect - Without correction, false alarm rates rose sharply as more metrics were added—approximately from 23% with five metrics to 93% with 50. - BH kept false alarm rates near 5%, but recall declined as the metric pool grew, falling from roughly 60% to 30% across the same range. - These results demonstrate that adding metrics makes statistical correction stricter and makes genuine effects harder to identify. ## Fewer, Higher-Quality Metrics - Reducing the metrics automatically included in experiments improves the balance between false alarms and recall. - Metrics should be selected for quality and conceptual distinctness rather than added “just to be safe.” - The article’s central conclusion is that no sophisticated statistical method eliminates the underlying tradeoff created by excessive measurement. Teams should maintain a focused default metric set, regularly remove low-value or redundant metrics, and reserve specialized metrics for experiments where they are genuinely relevant.

Read original

Statistical Analysis A B Testing Experiment Design Benjamini Hochberg Correction+3

lineAug 29, 2025Original article

Extracting Trending Keywords from OpenChat (opens in new tab)

To enhance user engagement on the LINE OpenChat main screen, LY Corporation developed a system to extract and surface "trending keywords" from real-time message data. By shifting focus from chat room recommendations to content-driven keyword clusters, the team addresses the lack of context in individual messages while providing a more dynamic discovery experience. This approach utilizes a combination of statistical Z-tests to identify frequency spikes and MinHash clustering to eliminate near-duplicate content, ensuring that the trending topics are both relevant and diverse. **The Shift from Chat Rooms to Content-Driven Recommendations** * Traditional recommendations focus on entire chat rooms, which often require significant user effort to investigate and evaluate. * Inspired by micro-blogging services, the team aimed to surface messages as individual content pieces to increase the "main screen visit" KPI. * Because individual chat messages are often fragmented or full of typos, the system groups them by keywords to create meaningful thematic content. **Statistical Detection of Trending Keywords** * Simple frequency counts are ineffective because they capture common social fillers like greetings or expressions of gratitude rather than actual trends. * Trends are defined as keywords showing a sharp increase in frequency compared to a baseline from seven days prior. * The system uses a Z-test for two-sample proportions to assign a score to each word, filtering for terms with at least a 30% frequency growth. * A seven-day comparison window is specifically used to suppress weekly cyclical noise (e.g., mentions of "weekend") and to capture topics whose popularity peaks over several consecutive days. **MinHash-based Message Deduplication** * Redundant messages, such as copy-pasted text, are removed prior to frequency aggregation to prevent skewed results and repetitive user experiences. * The system employs MinHash, a dimensionality reduction technique, to identify near-duplicate messages based on Jaccard similarity. * The process involves "shingling" messages into sets of tokens (primarily nouns) and generating $k$-length signatures; messages with identical signatures are clustered together. * To evaluate the efficiency of these clusters without high computational costs, the team developed a "SetDiv" (Set Diversity) metric that operates in linear time complexity. By combining Z-test statistical modeling with MinHash deduplication, this methodology successfully transforms fragmented chat data into a structured discovery layer. For developers working with high-volume social data, using a rolling weekly baseline and signature-based clustering offers a scalable way to surface high-velocity trends while filtering out both routine social noise and repetitive content.

Statistical Analysis AI Machine Learning Natural Language Processing+5

datadogJun 3, 20253 min readCurated summary

Detecting faulty deployments: Our journey from unlabeled data to supervised learning

Deployments are a major source of software incidents, making rapid detection of faulty releases essential. Datadog developed Automatic Faulty Deployment Detection to identify releases associated with significant, deployment-related increases in error rates, despite having no reliable labeled dataset. Their solution evolved into an iterative, unsupervised ensemble of statistical checks designed to balance precision, recall, and the diverse behavior of customer applications. ## Challenges in Detecting Faulty Deployments - No universal ground truth exists because teams define “faulty” differently depending on their applications. - Faulty deployments are rare, creating severe class imbalance: - Random manual labeling would produce few useful examples. - Even a low false-positive rate could result in poor precision. - Applications have widely varying traffic and error patterns: - Seasonal applications naturally experience periodic changes. - Low-traffic services need longer observation periods. - Frequent deployments can make it difficult to identify which release caused an incident. ## Defining a Faulty Deployment Datadog focused on deployments that caused a significant and sustained increase in error rate. The definition relied on three attributes: - **Impact** - The total number of errors must be meaningfully higher than the baseline. - The increase must be significantly worse than in previous versions. - **Temporal correlation** - The error increase should align with the introduction of the new version. - **Persistence** - The elevated error rate must continue over time rather than reflecting temporary deployment noise. ## Building an Iterative Detection Framework - The initial system applied simple statistical rules to the first 60 minutes after each deployment. - Manual annotation was used to estimate precision, but this required substantial effort and did not reveal recall. - Datadog created an iterative framework composed of checks for different deployment requirements. - Checks included: - Comparing error rates before and after deployment. - Comparing a release with previous versions. - Accounting for periodic traffic and errors. - Handling sparse traffic patterns. - The checks were combined into a unanimous-voting ensemble: a deployment was flagged only when every check classified it as faulty. - The process began with a high-recall model, then: - Manually reviewed predicted faults. - Analyzed false positives. - Added new checks and adjusted thresholds to improve precision and recall. - Incident data and version rollbacks provided additional signals for finding faulty deployments the model had missed. ## Balancing Detection Speed and Recall - The model used the first hour after deployment to gather enough data to determine whether increased errors were persistent. - Increasing the observation period can improve confidence but delays detection. - The framework became progressively more sophisticated, adapting to: - Periodic error and traffic patterns. - Sparse traffic. - Multiple concurrent application versions. The practical recommendation is to begin with simple, high-recall statistical rules, then iteratively improve them through targeted manual review, false-positive analysis, and additional operational signals such as incidents and rollbacks. This approach can support other anomaly-detection problems where labels are scarce, failures are rare, and application behavior varies significantly.

Read original

Statistical Analysis Database Design Application Performance Monitoring Incident Management+4

discordFeb 27, 20253 min readCurated summary

Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries. ## Why Traditional A/B Testing Was Difficult - Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile. - The feature inherently involves networks: one user sends a message and another receives it. - Network effects violate the assumption that treatment and control users behave independently, known as SUTVA. - Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization. - User-level A/B testing risked cross-group interactions. - Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior. ## How Synthetic Controls Work - Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units. - Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from: - 50% Argentina - 30% Uruguay - 20% Chile - The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes. - This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country. - It also produces a result that may be more representative than learning only how users in one specific country respond. ## Benefits and Evaluation - Synthetic controls can account for both observable and unobservable differences between regions. - They require: - Outcome data for the treated unit before and after treatment - Data from multiple untreated control units over the same periods - An analytical library, such as `Synth` in R or `SyntheticControlMethods` in Python - Discord evaluates the fit using Mean Squared Prediction Error (MSPE). - A close pre-treatment fit indicates that the synthetic control is a credible counterfactual. - A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region. - Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention. Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.

Read original

Statistical Analysis Python Ab Testing Causal Inference+3