A/B testing Calculator: Two Sample T-test

A two-sample t-test compares the means of two independent groups to determine if they are significantly different. Use it when comparing means between two groups with continuous data.

Paired t-tests compare related measurements (before/after, matched pairs). Unpaired t-tests compare independent groups. Choose based on your study design and data structure.

T-tests assume: normal distribution (or large sample size), independent observations, and equal variances between groups. The tool may provide tests for these assumptions.

For small samples (usually n < 30), it is important that the data in each group be approximately normally distributed or that there are no significant outliers. As sample size increases, the impact of deviations from normality is reduced by the central limit theorem.

Look at the p-value, confidence interval, and effect size. A p-value <0.05 typically indicates a significant difference. The confidence interval shows the range of plausible differences.

Effect size (e.g., Cohen's d) measures the magnitude of the difference between group means, beyond whether the difference is statistically significant. It helps understand the practical importance of the difference.

A Z-test is used when you know the population variance and have a large sample size. A t-test is more appropriate when the population variance is unknown and estimated from the sample, especially for small samples.

The null hypothesis (H0) in a t-test typically states that there is no significant difference between the means of the two groups being compared. The t-test assesses whether there is enough evidence to reject this hypothesis.