What is the Confidence Level for A/B Testing?

Question

When Marketing Cloud picks a winner for its AB tests, what confidence interval does it use? I'm assuming 90% but it'd be nice to know for sure what it is.

Answer 1

I think you mean confidence level that you are assuming is 90%. Confidence interval would be a range such as x to y. This will vary based on sample size you are taking to perform A/B test.

The confidence interval is a range within which the true value can be found at a given confidence level. Confidence interval depends on factors such as confidence level, conversion rates and sample size we are using to perform A/B testing.

It is important to perform test on an adequate sample size (number of visitors) prior to doing any A/B test, in order to establish the time that the test should be allowed to run before evaluating the results. Stopping the test after some time of monitoring once you have received one sample higher opens than the other causes the confidence interval to be vastly underestimated, making the test unreliable. It is desirable to use the highest possible confidence level, so that the test will yield few false positives. However, a higher confidence level requires a larger number of visitors, which increases the time required to do the test.

Confidence Level: A CL of 99% would mean that the results will meet the expectation 99 times out of 100.

For example, suppose that two offers (A and B) have true conversion rates of 10% and 15%, respectively. If these offers are shown to 100 visitors each, there is a 95% chance of observing conversion rates in the range 4% to 16% for offer A and 8% to 22% for offer B due to the stochastic nature of conversions. These ranges are known as confidence intervals in statistics. They represent the confidence in the accuracy of the estimated conversion rates. The larger the sample size (more visitors) the more confident you can be that the estimates of the conversion rates are accurate. Because of the large overlap between the two ranges, the test cannot determine whether the conversion rates are different. Therefore, this test with 100 visitors cannot distinguish between the two offers. However, if we expose the offers to 5,000 visitors each, then there is a 95% chance that the observed conversion rates will fall in the ranges of 9% to 11% and 14% to 16%, respectively. In this case, it is very unlikely that the test will come to a wrong conclusion, so the test with 5,000 visitors can distinguish between the two offers. The test with 5,000 visitors has a confidence interval of approximately +/-1%. This means the test can detect differences of about 1%. Therefore, even more visitors would be needed if the true conversion rates of the offers were, for example, 10% and 10.5% instead of 10% and 15%.

Hope it helps!