Measuring Product Impact Without A/B Testing: How Discord Used the Synthetic Control Method for Voice Messages

Discord used the Synthetic Control Method to measure the impact of Voice Messages when network effects made traditional A/B testing unreliable. Because users’ behavior is interconnected, randomizing individuals could contaminate treatment and control groups, while country-level comparisons could introduce geographic bias. Synthetic controls offered a stronger alternative by constructing a weighted “synthetic” comparison region from multiple untreated countries.

Why Traditional A/B Testing Was Difficult

• Discord launched Voice Messages in 2023 for text channels, DMs, and Group DMs on mobile.
• The feature inherently involves networks: one user sends a message and another receives it.
• Network effects violate the assumption that treatment and control users behave independently, known as SUTVA.
• Randomizing entire networks would be ideal, but Discord’s testing platform did not support cluster randomization.
• User-level A/B testing risked cross-group interactions.
• Country-level testing could reduce network contamination, but comparing countries directly would conflate the treatment with differences in language, culture, history, and user behavior.

How Synthetic Controls Work

• Synthetic controls compare one treated unit, such as Brazil, with a weighted combination of untreated units.
• Instead of comparing Brazil only with Argentina, Discord might construct a synthetic Brazil from:
  • 50% Argentina
  • 30% Uruguay
  • 20% Chile
• The weighted combination is designed to better reproduce the treated country’s pre-treatment outcomes.
• This approach addresses omitted-variable bias more effectively than selecting a single “similar” control country.
• It also produces a result that may be more representative than learning only how users in one specific country respond.

Benefits and Evaluation

• Synthetic controls can account for both observable and unobservable differences between regions.
• They require:
  • Outcome data for the treated unit before and after treatment
  • Data from multiple untreated control units over the same periods
  • An analytical library, such as Synth in R or SyntheticControlMethods in Python
• Discord evaluates the fit using Mean Squared Prediction Error (MSPE).
• A close pre-treatment fit indicates that the synthetic control is a credible counterfactual.
• A substantial increase in MSPE after rollout suggests that the feature changed outcomes in the treated region.
• Additional placebo checks can test whether the method tracks outcomes accurately during periods without an intervention.

Synthetic controls are a practical choice when network effects prevent conventional experimentation. For geographically distributed products like Discord, constructing a weighted counterfactual from multiple untreated regions can provide a more credible and generalizable estimate than either user-level A/B tests or simple geo-tests.