Recommendation Engines: How to Get Them Right With A/B Testing

Fill the form below to subscribe to our newsletter!

Recommendation Engine - How to get it right with A/B Testing

Table of Contents

Have you ever wondered how Netflix can get you to binge watch a whole series that eventually becomes your favorite? Or maybe you’ve wondered about the way e-commerce websites such as Amazon use the rather famous “Customers Who Bought This Item Also Bought” trick?

Amazon - You may also like

Now, just imagine applying these methods to your website. How impactful would that be for your business?

All this is possible with the help of a recommendation engine (RE).

Recommendation Engines and A/B Testing

A recommendation engine is the best way to offer a user-friendly service to ensure customer satisfaction, which eventually results in higher revenue — just like Amazon or Netflix.

Most websites don’t use this feature due to its complexity. It’s also difficult to judge whether to apply it permanently or not.

But what if there were a way through which you could test it beforehand and make sure it would be worth it? You can — with A/B testing. A test will help you decide if you need an RE or not, as it provides the comparative stats of conversion hikes before and after it has been applied to your site.

A/B testing is not new, and advanced websites have been utilizing it for a long time. There are many tools which help develop a solid A/B test. Industry leaders, including us at Brillmark, use tools like Optimizely, Adobe Target, VWO and Google Optimize to build successful tests.

Optimizely Logo VWO Logo Adobe Target Google Optimize Logo

To discover more about how A/B testing can help improve a recommendation engine, let’s first examine REs in general.

What Is a Recommendation Engine?

As the name suggests, a recommendation engine is a system that works as an agent to connect the products and services of your website with the nature of the product a user is already interested in, based on what another user with similar behavior did, or on a popularity basis of the products of the same genre, etc.

How streaming sites like Netflix may produce “Hot Picks”:

Recommendation ProcessImage source: freecontent

For example, whenever we shop online, we generally either purchase something we like, or something is recommended to us by others because we trust that person. A recommendation engine works like that person for online shopping. The suggestions shown can be a list of complementary products you want to show along with the main one, which we will use and apply in your RE. It is basically a way to use the data you already have to work in your favor.

Here, I added a television in my cart from BestBuy:

Amazon - You may also like

Now, instead of showing me another TV, the recommendation system suggested a wall mount and sound system, which are accessories that often accompany a TV purchase. Pretty smart move, right? Good web experts can help you add an RE like this one to your site, too.

Why Do You Need a Recommendation Engine?

Online shops need REs as part of their marketing strategy. If you set it up properly, it can boost conversions, click-through rates, and revenue.

It can also:

  • Change the way the website communicates with customers
  • Promote your product to potential buyers
  • Increase the amount of the sale, as the user can buy two or more items easily
  • Enhance your consumer satisfaction graph by providing personalized service to every client
  • Save your company time and money by automatically selling more items
  • Increase revenue by using resources you already have: each customer’s shopping history.


Is a Recommendation Engine Right for Your Site?

This feature is appropriate for all e-commerce websites that provide a large number of related products. In particular, it helps increase each order value by letting customers add related products they might need.

This is slightly different from upselling and cross-selling, as it is not only about price but focuses on the customer’s taste, which is a better approach to selling.

However, if you have a website which sells only one or two unique products, then it is not necessary for you to use this feature on your website. getmagicbullet, is a single product based website which sells a kitchen appliance and does not need an RE. But if your website offers more than one product, it should only help your site.

Amazon credits 35% of its revenue to its recommendation system. They show the user items which are purchased along with the items that user has in their cart, products similar to the ones they already viewed, etc.

How to Evaluate and Measure a Recommendation System With the Help of A/B Testing

Like any other step in the e-commerce world, the recommendation system also needs some kind of analysis before applying it. The best way to know that is to do A/B testing, which will help you to know whether your RE will give you more clicks, sales, and increased usage.

A/B testing is the best way to do online user evaluation, and we at Brillmark have clients who ask us to create a recommendation system for their website with the help of A/B test development

For Example:
Sometimes a client asks us to build a recommendation engine for their website. They already have the data, but they don’t know how to use it to build an RE. That’s where we come in.

Optimizely overall revenue result

Optimizely Overall Revenue Result

Optimizely cart result page visit result

Optimizely Cart Result Page Visit Result

As you can see from the above data, A/B testing allows you to judge the result between the original and a variation. The data shows clearly what effect a change can or cannot give to your website if shown live.

A/B testing plays an important role in decision making. You’ll get better results from using an RE, including cart conversions, thank-you page improvement, order review growth, etc.

It’s our job to help our customers know what changes to make by building an A/B test for them. We use their lists to apply an RE, which we run on their live site. We then compare the site with and without the RE to show the RE’s conversion power. With this information, our team can help drastically improve web sales metrics.

It’s a simple process in theory, but it takes a lot of research and knowledge to create an ideal RE to increase your site’s revenue. Here’s how it works: You give us a list of recommended products you want to show with a particular product, and we apply a recommendation engine design to run an A/B test. We run the RE A/B test variations on your live site for a day or two. Once we have the results, we apply the winning design to your live site permanently.

OptimizelyImage Source: Optimizely

If applying a recommendation engine can increase revenue so much, you may be wondering why we should run A/B test at all and not just apply the RE directly. A recommendation system needs to be tested first because, sometimes, putting a recommendation engine on a cart page may redirect the customer to other products, which could kill the chances of a sale on the initial product. In order to be sure of the benefits before permanent implementation, you can see the results from the A/B test and make a decision based on its stats.

Because when it comes to increasing sales and numbers — which should always go up — you want to be sure every move you make is the right one.


Now you know how to increase your user-base, get more clicks, and provide better personalization to each of your users in an effective manner. This can be one of the most useful technical and financial investments you can make for your website. A high-quality A/B test — resulting in the best recommendation engine — can elevate your business to a whole new level. You just need to find the right web experts to do that for you.

We at Brillmark are just the experts to handle your A/B testing needs. If you’re ready to take your online business to the next level, we’re ready to talk testing. Or, grab a spot on our calendar to schedule a 30-minute consultation.

Share This Article:

Skip to content