The Multi-Armed Bandit Sue A/B Testing Over Bayesian Regret

If you are in digital or performance marketing, and you test different ad copy, messaging headline etc, and optimise budgets to the best performing, this is not A/B testing! At the very best, this is a simplified version of Multi-Armed Bandit (MAB) testing.

This is because A/B testing and statistical significance are linked. You cannot have one without the other. If you cannot determine statistical significance, you have not done an A/B test. MAB testing on the other hand is an optimisation algorithm that maximises a specific metric.

MAB testing and A/B testing complements each other, and as sophisticated marketers or product managers, you should be using both.

With A/B testing, the objective is to collect data to make critical business decisions, and to determine the impact of all variations with statistical significance. With MAB testing, all you care about is maximising the number of a specific conversion.

MAB testing should be used when the window of opportunity for optimisation is small and there is not enough time to gather statistically significant results.

When you run a A/B test or multivariate test, you will be spending equal amount of money on the optimal solution as well as all the other options you are testing. This loss in conversions due to running the low performing variations is called Bayesian Regret. Secondly, statistical significance also comes with a time cost. It may take days, weeks, or months for test to run to get statistical significance.

MAB testing should be used when the opportunity cost of lost conversions is high, optimising revenue when traffic is low, when there’s need for continual optimisation or to optimise things such as Click-Through-Rate (CTR). MAB testing is quick to optimise and aims to reduce Bayesian Regret. However, the fastest way to statistical significance is A/B testing.

Factors to consider:

1. You may not be able to carry out post experimental analysis with statistical significance with MAB test data, because the algorithm sends more traffic to the optimal variation.

2. MAB testing only factor in one goal. This is not a suitable option when your goal is to optimise for multiple variations, whereas A/B testing is used to incorporate learnings from variations to make business decisions.