Understanding A/B Split Tests

Leading online retailers continually test different aspects of their websites to help drive sales. This can be as simple as testing various fonts and colors, or more sophisticated tests, such as pricing and product descriptions.

Optimizely is a leading testing platform for online businesses, small and large. It provides an interface for them to easily alter an item on a website, to see which version shoppers best respond to. The process is called “split testing,” or “A/B testing.”

We spoke recently with Optimizely’s co-founder and CEO, Dan Siroker.

Practical eCommerce: Optimizely offers A/B testing. What is A/B testing?

Dan Siroker: “A/B testing is the simple idea of running an experiment on your website. You create a variation of what your existing website looks like and then you have some of your traffic go to that variation and see whether they are more or less likely to convert into customers. The process of A/B testing helps you improve your website by making data-driven decision around what’s the most effective way to turn your visitors into customers.”

PEC: Our readers are ecommerce merchants — mainly smaller ecommerce merchants. What sort of A/B test would you run if you owned a small ecommerce company without much budget?

Siroker: “The best place to start is by creating an experiment on one of your most highly valued pages. That might be on your checkout page, the place where shoppers might enter their credit card to complete their purchase, or it might be the product page where they’re going to add something to their shopping cart. But the experiment I recommend you start with is a dramatically new layout. It’s a very common temptation to just make a small, incremental change but often those are the changes that don’t have as big as impact as a dramatically new layout.

“The reality is most websites aren’t very good. It’s hard to hear that as a small business owners, but it gives you the opportunity to make a dramatically new website. If you run an experiment [on a totally new site], you can see whether those changes are actually having an impact.”

PEC: In what respect are most websites not good?

Siroker: “Most websites are not good in the sense that on average across the entire web, only 2 percent of visitors convert into customers. That means 98 percent are not turning into customers, or they are not turning into something that’s valuable for you as an organization. The opportunity for you to improve the effectiveness of your website is pretty large. If you can increase that 2 percent to 4 percent, you have now doubled the effectiveness of your website and hopefully doubled your revenue as well.”

PEC: Tell us a success story or two of a company that has used your service or has engaged an A/B testing to improve its business.

Siroker: “One example is Electronic Arts, which ran an experiment on their ecommerce store. They are selling video games. They have a popular new game that just came out called SimCity and in the most recent version of SimCity they had a product page that looks very similar to most ecommerce sites. It has big ‘buy now’ button. You add SimCity to your shopping cart and you can buy it online.

“When they first launched this product, they had a giant banner at the top of this website that says, ‘Pre-order and get $20 off on your next purchase.’ So EA effectively gave you a discount on your next product if you bought SimCity. That big banner at the top of this page actually moved down all the content that was necessary for you to buy — basically the ‘buy now’ button.

“They ran a simple experiment. They removed the big ‘Pre-order and get your $20 off on your next purchase’ banner and that improved the conversion rate — the percentage of people who actually turned into customers — by 43 percent. They went from 5.8 percent conversion rate to a 10.2 percent conversion rate, which is a dramatic impact. What was so amazing about this is that was just by removing something from the page, not adding something. They removed a number of things that were on the page that might distract the user or a choice that would prevent them from actually finishing the entire purchase process.

“That really proves the value of ‘less is more.’ Reduce choices and your customers are more likely to take the choices you want them to take.”

PEC: You’re co-founder of Optimizely. Tell us about that company.

Siroker: “We launched Optimizely about four years ago and the idea behind Optimizely was rooted in my experience on the Obama campaign in 2008. Before my work on the campaign, I was working at Google as a product manager and in November of 2007, I snuck in to see Barrack Obama speak when he came to Google. He gave a talk about how he wanted to take what we are doing at Google — with evidence in science and feedback — and bring that to the government.

“That got me really excited and I decided to leave my job at Google for the campaign and start off as a volunteer that eventually turned into a full time job as the director of analytics. In that job, I was asked to figure out how to use data to help make better decisions. And naturally that lead to website optimization and A/B testing and in that process I used Omniture, Test&Target, and Google Website Optimizer. I got a ton of value out of that but I was constantly bottlenecked and requiring a developer to be part of the process. Optimizely is a product I wish we had in 2008 to make it easy for anybody to do A/B testing.

“Since then we have launched and we have actually achieved that goal. We had Obama in 2012 and Mitt Romney use this extensively in the last election cycle as well. And 5,000 other businesses both big and small, including Starbucks, Salesforce, and Crate & Barrel, use it. We know that the number one reason people don’t use Optimizely today is because they haven’t heard about us yet. Our long-term vision is to enable businesses to show exactly the right thing to the right person at the right time and we think A/B testing is a great first step towards that vision.”

PEC: Say a merchant wants to test something on a site, perhaps a product description for a single product. Could you walk us through how that process works using Optimizely to test two different product descriptions for a single product?

Siroker: “That process would be pretty straightforward, and it’s something that marketing or product management or even the CEO of the company can do. It starts by going to and entering the URL of the page you want to test.

“What that will do is load your page in context of our visual editor. You will see your page at the bottom of our site and you can move your mouse over different parts of the page. A blue rectangle will show up around any part of the page that you can test and by clicking on that, you will get a series of options. In this case, for the example you gave, you would click the edit text option and then you are given a form field very much like Microsoft Word where you can just enter into a what-you-see-is-what-you-get fashion the new text or the new description for that product.

“Once you do that, you go and implement Optimizely. The process of implementing Optimizely is as simple as a one-time copy and paste of a single line of code. Once that’s on your website, you can create as many experiments and create as many variations as you like.”

PEC: In that example, how does the person doing the test track the results?

Siroker: “What actually happens is it’s all still hosted on your site. We actually never host the variations ourselves. What we do is that snippet of code that you included on your site, in this case if you have two variations, will randomly put 50 percent of your traffic into this variation bucket, and so randomly, they will see that variation that you created in Optimizely. Because the way we were implemented as a single line of code of a JavaScript file, these changes all happen client side. They happen in your end user’s browser so he doesn’t see the URL change.

“It still looks like they are on your site. They don’t actually know that they are part of an experiment. What will happen is if they click on different parts of the page, maybe they add that product to their shopping cart that will send a tracking call to the Optimizely servers and then on the Optimizely results page you will be able to see the conversion rate for your original page and for the variation you have created. You will also see a difference in those two variations, and if that is statistically significant, we also show you an indication. You don’t have to worry about the math or underlying analysis. We will just tell you variation B did better than variation A.”

PEC: How much does your service cost?

Siroker: “Optimizely starts as low as $19 per month and goes on up from there. We have several tiers of service from bronze, silver, gold, and platinum and we would like to think that we have a product offering that fits with any size business and budget. We also have a 30-day free trial.”

PEC: Anything else?

Siroker: “Ecommerce is a great opportunity for you to use data you have about an individual to give her a better experience. There’s probably no better example of this than Amazon, and our vision at Optimizely is to enable the rest of the web to do what Amazon does with hundreds of engineers. Amazon has a huge army of engineers who help them day in and day out to improve the effectiveness of We don’t think that’s something just Amazon should be able to do.”

PEC Staff

PEC Staff

Bio   •   RSS Feed