Optimizely Exec on A/B Split Tests

Leading online retailers continually test different aspects of their websites to help drive sales. This can be as simple as testing various fonts and colors or more sophisticated tests, such as pricing and product descriptions.

Optimizely is a leading testing platform for online businesses, small and large. It provides an interface for them to easily alter an item on a website to see which version shoppers respond to. The process is called “split testing” or “A/B testing.”

We spoke recently with Optimizely’s co-founder and CEO, Dan Siroker.

Practical Ecommerce: Optimizely offers A/B testing. What is A/B testing?

Dan Siroker: A/B testing is the simple idea of running an experiment on your website. You create a variation of what your existing website looks like, and then you have some of your traffic go to that variation and see whether they are more or less likely to convert into customers. The process of A/B testing helps you improve your website by making data-driven decisions around what’s the most effective way to turn your visitors into customers.

PEC: Our readers are ecommerce merchants — mainly smaller ecommerce merchants. What sort of A/B test would you run if you owned a small ecommerce company without much budget?

Siroker: The best place to start is by creating an experiment on one of your most highly valued pages. That might be on your checkout page, the place where shoppers might enter their credit card to complete their purchase, or it might be the product page where they’re going to add something to their shopping cart. But the experiment I recommend you start with is a dramatically new layout. It’s a very common temptation to make a small, incremental change, but often those are the changes that don’t have as big an impact as a dramatically new layout.

The reality is most websites aren’t very good. It’s hard to hear that as small business owners, but it allows you to make a dramatically new website. If you run an experiment [on a new site], you can see whether those changes are having an impact.

PEC: In what respect are most websites not good?

Siroker: Most websites are not good in the sense that, on average, across the entire web, only 2 percent of visitors convert into customers. That means 98 percent are not turning into customers, or they are not turning into something valuable for you as an organization. The opportunity for you to improve the effectiveness of your website is pretty large. If you can increase that 2 percent to 4 percent, you have now doubled the effectiveness of your website and hopefully doubled your revenue as well.

PEC: Tell us a success story or two of a company that has used your service or has engaged an A/B testing provider to improve its business.

Siroker: One example is Electronic Arts, which ran an experiment on their ecommerce store. They are selling video games. They have a popular new game that just came out called SimCity and, in the most recent version, they had a product page that looks very similar to most ecommerce sites. It has a big “buy now” button. You add SimCity to your shopping cart and buy it online.

When they first launched this product, they had a giant banner at the top of this website that said, “Pre-order and get $20 off on your next purchase.” So EA effectively gave you a discount on your next product if you bought SimCity. That big banner at the top of this page moved down all the content necessary for you to buy — basically the “buy now” button.

They ran a simple experiment. They removed the big “Pre-order and get your $20 off on your next purchase” banner. That improved the conversion rate — the percentage of people who turned into customers — by 43 percent. They went from a 5.8 percent conversion rate to a 10.2 percent conversion rate — a dramatic impact. What was so amazing about this was just by removing something from the page, not adding something. They removed several things on the page that might distract the user or a choice that would prevent them from actually finishing the entire purchase process.

That proves the value of “less is more.” Reduce choices, and your customers are more likely to make the choices you want them to take.”

PEC: You’re the co-founder of Optimizely. Tell us about that company.

Siroker: We launched Optimizely about four years ago, and the idea behind Optimizely was rooted in my experience on the Obama campaign in 2008. Before my work on the campaign, I was working at Google as a product manager. In November of 2007, I snuck in to see Barack Obama speak when he came to Google. He gave a talk about how he wanted to take what we are doing at Google — with evidence in science and feedback — and bring that to the government.

That got me excited. I decided to leave my job at Google for the campaign and start as a volunteer that eventually turned into a full-time job as the director of analytics. In that job, I was asked to figure out how to use data to help make better decisions. And naturally, that led to website optimization and A/B testing. In that process, I used Omniture, Test&Target, and Google Website Optimizer. I got a ton of value from that, but I was constantly bottlenecked and required a developer. Optimizely is a product I wish we had in 2008 to make it easy for anybody to do A/B testing.

Since then, we have launched and have achieved that goal. We had Obama and Mitt Romney in 2012 use this extensively. And 5,000 other businesses, big and small, including Starbucks, Salesforce, and Crate & Barrel, use it. We know that the number one reason people don’t use Optimizely today is that they haven’t heard about us yet. Our long-term vision is to enable businesses to show exactly the right thing to the right person at the right time, and we think A/B testing is a great first step towards that vision.

PEC: Say a merchant wants to test something on a site, perhaps a product description for a single product. Could you walk us through how that process works using Optimizely to test two different product descriptions for a single product?

Siroker: That process would be pretty straightforward, and it’s something that marketing or product management or even the company’s CEO can do. It starts by going to and entering the URL of the page you want to test.

What that will do is load your page in the context of our visual editor. You will see your page at the bottom of our site, and you can move your mouse over different parts of the page. A blue rectangle will show up around any part of the page that you can test, and by clicking on that, you will get a series of options. For the example you gave, you would click the edit text option, and then you are given a form field very much like Microsoft Word where you can enter into a what-you-see-is-what-you-get fashion the new text or the new description for that product.

Once you do that, you go and implement Optimizely. The process of implementing Optimizely is as simple as a one-time copy and paste of a single line of code. Once that’s on your website, you can create as many experiments and create as many variations as you like.

PEC: In that example, how does the person doing the test track the results?

Siroker: What happens is it’s all still hosted on your site. We never host the variations ourselves. What we do is that snippet of code that you included on your site — if you have two variations — will randomly put 50 percent of your traffic into this variation bucket. So randomly, they will see that variation that you created in Optimizely. Because we were implemented as a single line of code of a JavaScript file, these changes all happen client-side — in the end user’s browser — so he doesn’t see the URL change.

It still looks like they are on your site. They don’t know that they are part of an experiment. If they click on different parts of the page, maybe they add that product to their shopping cart that will send a tracking call to the Optimizely servers. Then on the Optimizely results page, you will be able to see the conversion rate for your original page and for the variation you have created. You will also see a difference in those two variations, and if that is statistically significant, we will also show you an indication. You don’t have to worry about the math or underlying analysis. We will tell you variation B did better than variation A.

PEC: How much does your service cost?

Siroker: Optimizely starts as low as $19 per month and goes up from there. We have several service tiers from bronze, silver, gold, and platinum. We like to think that we have a product offering that fits any size business and budget. We also have a 30-day free trial.

PEC: Anything else?

Siroker: Ecommerce is a great opportunity for you to use data about an individual to give her a better experience. There’s probably no better example of this than Amazon, and our vision at Optimizely is to enable the rest of the web to do what Amazon does with hundreds of engineers. Amazon has a huge army of engineers who help them day in and day out to improve the effectiveness of We don’t think that’s something just Amazon should be able to do.

PEC Staff
PEC Staff
Bio   •   RSS Feed