Apr 17

5 Sins You Must Avoid While A/B Testing Your Website

Do you want your precious efforts to get wasted?

Just as important it is to know how to improve website’s functionality to see great results, it is equally crucial to know what to avoid to ensure that your efforts are not being wasted. Have a look into these 5 sins that marketers generally do while conducting A/B Tests. For your advantage, we have provided a quick description so that you can let them go off in your next round of A/B Testing.

1. Never do A/B Testing on pages with too less traffic

Running A/B Tests on websites that are less trafficked will definitely prevent you from gathering enough of data from statistical point of view. Make sure to run tests on pages that have enough traffic otherwise you may end up with tests that have no significant result or you will have to wait for quite a long time to get enough data points collected by the AB Testing software to reach any conclusion. The website goals that you set up for an experiment determines the statistical metrics by which a test is considered successful. Therefore, if your website has a fewer conversions or traffic, it will definitely take time to determine a winning variation, especially if there is a slight difference in conversion rate between the original and variation.

2. Failing to understand the need for MVT and A/B Tests

 What do you think is the major difference between A/B Test and MVT. An A/B Test measures the performance of two completely different versions of a web page whereas a multivariate test measures the performance of different elements in a web page. So, before going for any of the two, you need to take care of-

How soon you want to gather results for your website- A/B Testing is a great choice if you want to acquire results quickly and with a lesser traffic. The changes delivered are so sharp that it will easily tell you which page will generate the maximum result for your business.

How different elements of a web page interact individually or in combination with each other- MVT or Multi Variate Testing is a perfect pick if you want to understand how certain elements interact with one another to improve on an existing web page design. Since large number of items are involved in MVT Test, it takes longer to run and complete. It is suggested to run MVT on the pages which have enough traffic.

Target pages based on the quality of traffic your business receives- Although site traffic is an important factor to consider, but there are other important aspects like the different pages within a site as “product page”, “landing page”, “home page” etc. that needs to be visited regularly so as to have some data to analyze. This will further help you decide the type of test to opt for your website.

3. Avoiding the importance of statistical significance

Never be in a hurry to reach a conclusion. Statistical confidence in A/B testing helps you get an idea whether your test results are significant or not. It prevents you from reading too much into the results if you have only a fewer conversions for each variation. Generally 95% confidence interval is good to choose a winner, but people got for 90% as well as 85% confidence interval. Lower the confidence interval faster the conclusion will be reached but this is at the cost of accuracy. Remember lower the confidence interval lower the quality of the result and it means that you are leaving things to ‘chance’ and not statistically supported conclusion.

For example, you have A/B tested your product pages and realized that A is better than B. Now, how would you determine whether or not this was something real or just a coincidence. Let’s talk about a real time example, what if it rained every time you went out for shopping in a month? How will you conclude that going out for shopping brings rain? Here is where statistical significance comes into picture. It determines the probability of a result being a chance or reality.

4. Stop Running multiple tests simultaneously with overlapping traffic

A/B testing helps us know more about our customers and conversion ratio. Before running a test, you need to make sure that it is easy to analyze and understand. Conducting multiple tests at the same time with the same amount of traffic is the biggest mistake people normally do. If you are doing so, it’s time to stop. Mark my words, it is going to be harder to get conclusions from the tests and understand which elements or variable worked and which did not.

Running multiple tests is not a bad idea if you have huge amount of traffic but if you want to learn how individual elements perform, make sure running it with a different traffic.

 5. Wasting time on testing minor changes

Well, it’s quite true that even a small change sometimes leads to wonders in A/B Testing but it is not likely that it will always. So, if you are thinking that changing a word in the headline text or the color of buttons in your website will bring magic is something that is not acceptable for an eCommerce entrepreneur. Please note that I am not saying that small changes are not to be done, but do them only when major changes are all complete and you have reached optimum level with the page and now small changes are being tested to further push the results.

All you need to do is focus more on testing major changes that an average consumer would find easy spotting in. This will help you gain significant data and drastically increase your website conversion rate.

In short, A/B Testing is a long way to go for website optimization. Doing it correctly means you are continuously optimizing it and analyzing your conversion funnel. Analyzing data is one thing and knowing what can be done with it is something different. So, if you are really looking forward for significant results, try coupling both hand in hand.


Need a platform to run your AB tests and increase conversions, don’t be hesitant to talk to us. We are here to analyze data and optimize your website.