A/B testing Calculator: Sample Size

Sample size depends on the desired confidence level, margin of error, population size, and expected effect size (MDE). Larger effects require smaller samples, while smaller effects require larger samples to detect.

Confidence level (typically 95%) indicates how confident you are in your results. Margin of error is the range of uncertainty around your estimate. A higher confidence level or smaller margin of error requires larger samples.

Use a conservative approach - an equal split into two groups in 50/50 proportions.

If the population is limited, use a finite population correction and try to maximize coverage. In the case of very small samples, use non-parametric analysis methods and consider reduced statistical power.

If you choose too small an MDE, a huge sample will be required, which may be impractical. If you choose too large an MDE, you might miss small but important effects. The optimal MDE should reflect the business or research value of the effect.

Statistical power is the probability of detecting a real effect (if it exists) and avoiding a Type II error (a false negative). Higher power (typically 80% or more) requires a larger sample size to ensure you don't miss a significant result.

The greater the variability in the population data (standard deviation), the larger the sample size needed to achieve the same precision. This is because greater variability makes it harder to obtain an accurate estimate of the population mean.

For comparative studies, you need a sample size calculation that considers the Minimum Detectable Effect (the smallest difference you deem important) and statistical power to ensure your experiment is capable of detecting that difference.