A/B testing Calculator: Sequential Sampling

The calculator for sequential A/B test analysis is a tool that helps determine the optimal moment to stop an experiment based on statistical data. It takes into account the baseline conversion, minimum detectable effect, statistical power, and significance level to help you make an informed decision about continuing or stopping the test.

The calculator uses statistical methods to estimate the minimum sample size based on baseline conversion, minimum detectable effect, statistical power, and significance level. It shows when the experiment can be stopped.

Absolute effect is a percentage difference (e.g., a 5% increase in conversion). Relative effect is a percentage change from the baseline conversion (e.g., a 25% increase from a 20% baseline conversion).

Statistical power (60-95%) determines the probability of detecting an effect if it exists. The significance level (1-10%) determines the probability of a false positive. Higher power requires a larger sample size. You can adjust the statistical power and significance level in the calculator.

Sequential testing is ideal for A/B tests with high observation costs, where it is important to save resources. It is also useful for quickly obtaining results in marketing campaigns.

A Type I error (alpha) is when you reject a true null hypothesis (conclude there's a difference when there isn't). A Type II error (beta) is when you fail to reject a false null hypothesis (conclude there's no difference when there is).

Sequential sampling is particularly useful when the cost of each observation is high or when you want to get results faster. However, it requires constant monitoring of results, which can be more challenging to implement than a fixed-size test.

Baseline conversion is the current or expected conversion rate of your control group (the original variant). It's the reference point from which you measure the potential impact of your new variant (the test group).