Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages
Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries. ## Why Traditional A/B Testing Was Difficult - Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile. - The feature inherently involves networks: one user sends a message and another receives it. - Network effects violate the assumption that treatment and control users behave independently, known as SUTVA. - Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization. - User-level A/B testing risked cross-group interactions. - Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior. ## How Synthetic Controls Work - Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units. - Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from: - 50% Argentina - 30% Uruguay - 20% Chile - The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes. - This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country. - It also produces a result that may be more representative than learning only how users in one specific country respond. ## Benefits and Evaluation - Synthetic controls can account for both observable and unobservable differences between regions. - They require: - Outcome data for the treated unit before and after treatment - Data from multiple untreated control units over the same periods - An analytical library, such as `Synth` in R or `SyntheticControlMethods` in Python - Discord evaluates the fit using Mean Squared Prediction Error (MSPE). - A close pre-treatment fit indicates that the synthetic control is a credible counterfactual. - A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region. - Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention. Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.
Read original(opens in new tab)