Measure Less to Learn More: Using Fewer, Higher-quality Metrics to Capture What Matters

Discord argues that experiments should measure fewer, higher-quality metrics rather than automatically collecting every potentially useful signal. Large metric sets increase compute and cognitive costs while creating a tradeoff between false positives and missed real effects. Multiple-testing corrections such as Benjamini–Hochberg reduce false discoveries but also lower recall, so the most effective solution is selecting metrics that represent distinct, important concepts.

The Cost of Measuring Too Much

• Discord’s “Default Metric List” gradually expanded as teams added metrics and rarely removed them.
• More metrics create:
  • Higher compute costs
  • More difficult experiment readouts
  • Increased risk of false positives
• With 100 metrics and an uncorrected significance threshold of 0.05, roughly five metrics may appear significant purely by chance.
• Correcting for multiple comparisons reduces false alarms but makes genuine changes harder to detect.

The Multiple Comparisons Problem

• Discord uses the Benjamini–Hochberg (BH) procedure to control the false discovery rate at 5%.

• BH ranks p-values and compares each one with a rank-specific threshold:

  i × α / n

  where i is the metric’s rank, α is 0.05, and n is the total number of metrics.

• A metric with an unadjusted p-value of 0.038 might be significant without correction but fail after BH adjustment.

• BH treats all metrics equally because it has no information about which ones are more likely to reflect a real effect.

• The resulting tradeoff is:

  • Fewer false alarms
  • Lower recall for real changes

• The article notes that Bayesian methods could potentially incorporate prior knowledge, but Discord’s default system is frequentist.

Simulation Results

• Discord simulated 50,000 experiments containing:
  • Twenty null metrics generated from N(0, 1)
  • One metric with a real effect centered at z = 2.8
• The simulations tested how metric count affects:
  • Experiment-level false alarm rates
  • Recall of the metric with the known effect
• Without correction, false alarm rates rose sharply as more metrics were added—approximately from 23% with five metrics to 93% with 50.
• BH kept false alarm rates near 5%, but recall declined as the metric pool grew, falling from roughly 60% to 30% across the same range.
• These results demonstrate that adding metrics makes statistical correction stricter and makes genuine effects harder to identify.

Fewer, Higher-Quality Metrics

• Reducing the metrics automatically included in experiments improves the balance between false alarms and recall.
• Metrics should be selected for quality and conceptual distinctness rather than added “just to be safe.”
• The article’s central conclusion is that no sophisticated statistical method eliminates the underlying tradeoff created by excessive measurement.

Teams should maintain a focused default metric set, regularly remove low-value or redundant metrics, and reserve specialized metrics for experiments where they are genuinely relevant.