Mar 8

Is running A/A Test before A/B testing beneficial?

A/B testing

Being an eCommerce marketer, you must have heard or even practiced A/A testing before A/B testing. The idea or the question to A/A test itself invites conflicting opinions. The majority of the companies face a dilemma when it comes to deciding whether to do this process or not.

Well, understanding what A/A testing is and what are the pros as well as cons doing it would give a better insight into whether to do it or not to do it.

What is A/A test?

A/A test is an A/B test, where the two variations that form the user experience are identical. The test is conducted to help the marketer analyze the accuracy of the setup and authenticity of the A/B testing platform.

Why do some organization invest in A/A test?

Just like it is mentioned above, there are some other reasons too for conducting A/A test on your website. They include:

  • The authenticity of the new A/B testing tool
  • Setting a measure for future A/B tests; and
  • Decide the minimum size of the sample

Let us now look at the three reasons in brief below.

The authenticity of the new A/B testing tool

The organizations who are planning to invest in a new A/B testing tool and have asked yourself some essential questions before actually taking the step generally conduct A/A tests. Why? To check whether the A/B testing tool/software that they bought is working fine and whether or not the setup is done correctly.
So, when you conduct an A/A test on two identical versions, the results should come in identical. However, if in the end, you have a clear winner, there’s definitely some problem with:

  • The setup of the tool
  • The test that was conducted; or
  • The unworthy testing tool

Setting a measure for future A/B tests

Before running any A/B test, you need to know the conversion rate that you will be benchmarking the performance results against. This benchmark is your baseline conversion rate.

The A/A test is what will help you to achieve that conversion rate measure.

Decide the minimum size of the sample

Testing your variables with the A/A test is also great to decipher what your minimum sample size should be from your website traffic. This is no secret that a smaller sample size is insufficient for accurate results. In order to get accurate results, a bigger sample size is necessary. The smaller size is what makes you miss out on the smaller segments that have a potential impact on the test results. Hence, it is always better to have a larger sample size in order to get clearer results.

Disadvantages of A/A testing

The following is the disadvantage of A/A testing.

The large sample size

Though having a large sample size adds in more chances of getting accurate results, but there are disadvantages of having such a big size too. The larger the size, the more consuming the test would be. Do you have enough time?

Element of randomness

If you check up, earlier in the blog we talked about the authenticity of the testing tool which of course is the purpose of conducting the test in the first place. However, what if you find that there’s a difference between the conversions of control and the identical variation? Would you refer it as a bug in the A/B testing tool?

Well, the problem with A/A testing is that there’ll always be an element of randomness involved. In few incidences, the experiment grabs the statistical significance just by chance. Hence, it actually means that the change in the conversion rate is merely by chance and there’s no certainty about it.


If you really want your A/B test to be accurate and give clear results, make sure your tools are in check. And, in order to implement the new tool, you must run the same thoroughly. One of the methods to check the authenticity of the new software is A/A testing. There are other companies investing in other methods as well. While some experts believe that A/A tests are a waste of time, others find it quite helpful. You can see if it suits your business or not.


Post A Comment