Michael Kaminski, a developer running a custom LinkedIn scheduler, discovered that his A/B tests for post timing were statistically doomed before they started. By analyzing 182 posts with logged impressions, he found that detecting a meaningful 20% lift on raw impression data would require 7,747 posts per arm, translating to 738 weeks of experimentation at his current cadence. The breakthrough came from switching the power calculation to a log scale, which reduced the required duration to just 30 weeks for the same statistical power.

The Problem With Raw Impression Data

The scheduler, a 1,171-line Python file using a Thompson sampler, was making decisions based on engagement rates and impression volumes. However, the underlying distribution of impressions was heavily skewed. With a mean of 223 impressions but a standard deviation of 983, the data exhibited a sample skewness of 12.1. One viral post accounted for 31.9% of all logged impressions, creating a long tail that inflated variance and made traditional t-tests on raw values inefficient and misleading.

Why Log Scale Works for Multiplicative Metrics

Reach is a multiplicative metric, not an additive one. When a post performs well, it doesn't just add impressions; it triggers an algorithmic cascade that multiplies exposure. This process naturally produces log-normal outcomes. By taking the natural log of impression counts, Kaminski normalized the variance. On this scale, a 20% lift becomes an additive constant (ln(1.2) = 0.182), allowing for a standard power calculation that yielded a required sample size of only 314 posts per arm.

The Trade-Off: Geometric vs. Arithmetic Mean

The log-scale approach optimizes for the geometric mean, answering whether a change improves reach for a typical post. It does not directly optimize for the extreme tail events that often drive total reach. The outlier post that skewed the raw data sits 13 standard deviations from the mean on the raw scale but only 5.9 on the log scale. This means the test becomes partly blind to the very outliers that might carry the most business value, a trade-off Kaminski accepts to make experimentation tractable.

Key Takeaways

  • Check the ratio of your largest observation to your median; if it exceeds 10:1 for non-negative metrics, assume a log-normal distribution.
  • Use log scales for multiplicative metrics like reach, revenue, or latency tails to reduce required sample sizes by orders of magnitude.
  • Power calculations should be run before experiments begin to determine if they are feasible within current operational constraints.
  • Optimizing for the median (geometric mean) may miss tail-driven value, requiring manual review of outliers.

The Bottom Line

If you are running experiments on multiplicative metrics and haven't checked your power calculations, you are likely running decorative tests that will never conclude. Switch to the log scale or accept that you are wasting time.