Practical Ecommerce

How to Test Multiple Variables on an Ecommerce Site

Ecommerce merchants strive to increase the number of visitors to their sites that actually purchase products. Increasing those conversions frequently requires experimenting with new approaches. Merchants can test which of these experiments increase sales, and which do not. The testing can be automated, sophisticated — and it produces results. To explain multi-variable ecommerce testing to us, we recently spoke with Mark Simpson, the founder and president of Maxymiser, a testing, personalization and conversion-optimization firm.

Practical eCommerce: What are common split and multivariate tests that merchants need to be thinking about?

Mark Simpson

Mark Simpson

Mark Simpson: “We see a vast range depending on the website that we’re working with. A good place to start is looking at your site analytics and seeing where you get drop off points — try to fix those areas to increase conversion rates. Commonly with people new to testing, they generally start doing A/B split test across landing pages, maybe it’s complete page changes, maybe it’s just changing a button on the page, but very quickly I think people move towards testing wider through their site.

“The real return on investment you get through testing is when you start getting towards the checkout area of the site, purely because checkouts have already been largely untouched over the years. So, we’re seeing huge increases when we actually start testing within checkout processes and the like.

“Ultimately, our more advanced retail clients are using testing on a continual basis. It’s really testing every area of the site and not making any changes without testing. Sites are moving from websites where you used to have redesigns every couple of years to websites that are being iterated on a continual basis. So in two years’ time, yes, the site does look very different, but every single change that has been made has been tested, has been proven to increase conversion rates and the site is performing a lot better as well. So, less guess work, less gut feel and much more of that iterative measured approach to running a website. It’s something that the leaders in this space, the Amazons, the eBays, the Googles of this world have been doing since their inception. They haven’t gone through big website redesigns. They’ve really focused on testing with their user base and improving on a continual basis.”

PEC: Give us some common variables that larger merchants test on the checkout pages.

Simpson: “There are now a number of key areas in checkout functionality. One is how much information you’re taking. Obviously, there is a standard, a basic amount of information you’re taking, and it depends on the site as to the drop off rates when you start taking more than just the information that is needed. Testing on things like having to be logged in versus just letting people go through the checkout process. Testing the number of pages that you put your checkout process through and also the layouts of those pages is absolutely crucial and vital in the number of tests we’ve seen.

“Even simple things like the buttons, the position of them, the color of them and the wording of them can really have quite dramatic differences to people actually flowing through the process. Also, spelling out in a breadcrumb trail how many stages of the process there are. So, there are a number of key areas that can be tested in the checkout process, all of which will differ depending on a site in terms of what works and what doesn’t, but vital nonetheless in getting more business root through that checkout and less drop off.”

PEC: If you’re testing, say, a half-dozen variables at any one time, how do you know which if those variables is causing the benefit, or the problem?

Simpson: “This is where multivariate testing really comes into its own over A/B or A/B/C split testing. With multivariate testing, you’re actually getting into the detail of which variables are controlled in the uplift and which ones aren’t, which ones are most relevant and how will those really interact with each other to form the very best converting page, which you just don’t get that level of detail with A/B and split testing. With multivariate testing, ultimately you need a platform to run it from. The good news is the platforms really vary. They range from very expensive to costing nothing; they range from very easy to use to very hard to use, as well.

“So, there’s obviously a trade-off in the platform that’s used, but once you have a testing platform and you’ve integrated that with your site — normally that’s just a piece of JavaScript code that you need to put on the site — once that’s in, the testing can get underway and the platform controls what content is pulled onto the page when the visitor gets to the page. So, it keeps the page as is at the moment. You don’t need to develop new pages. You just need to develop new items of content, new variants that get fed into the page areas in which you’re testing and the platform will ultimately control all of that. The platform will also measure how that interacts, what the feedback is, what the conversion rates or different variants are. You then feed that back in useful reports so you can see what works and what doesn’t — with ease.”

PEC: For Maxymiser’s customers, are their site visitors actually routed to Maxymiser’s servers and those servers then replicate the merchant’s site? Is that how that works?

Simpson: “No. This is where some platforms do differ. With Maxymiser, you would integrate [our platform] with a single global line of code. If you put that into a global header or a template, it means you can then test on any page through the site without ever having to integrate more code again, without ever having to go back to IT teams. Once that code is implemented, you would go on to the Maxymiser user interface, which is just a web interface. You would then pull up the page you want to test, whether it’s a checkout process, whether that’s a landing page, and then you can start moving items around the pages, changing the different number of pages or adding different content into those pages through the Maxymiser user interface. Once that is done, we then give you a portal that allows you to test those different pages. Once you’re happy with that — the testing is all in a live environment — you would then publish the test and the test goes live.

“Now, the test still happens and the content is still on the merchant’s site. So, nothing is routed to Maxymiser. Maxymiser is like a rules engine over the page, controlling the presentation there of the page, but the functionality of the page and the way the page is loaded and where it sits and the way it works still stays with the merchant and all the content can still stay on the merchant’s secure server — or we can host them in the cloud. It really depends on how a partner would like to work.”

PEC: Does your solution work with different ecommerce platforms? Readers of this interview are using dozens, if not hundreds, of ecommerce platforms.

Simpson: “Absolutely. We’ve got over a hundred ecommerce clients now, and we’ve come across everything from home built platforms to IBM WebSphere and it really does not matter what platform is being used. As I said, Maxymiser is there controlling the presentation layer of the content. We’re not there to alter the fundamentals of the platform. So, we’re completely agnostic in terms of any platform usage.”

PEC: In the example of conducting a test across multiple variables, how does your platform determine which customer sees which variable?

Simpson: “There are two different options for that, really. One, which is probably a preferred option, to get the fastest result, is to have Maxymiser serve the different content or we give the rules to serve the different content on a random basis. So, ultimately, it’s completely random on who receives which variants. That person is clicking to make sure when they come back to the page they see the same variants — very important — and then Maxymiser can automatically start optimizing to better performing pages and better performing variants through the course of the test. It’s an automated way of optimizing.

“The other way of doing it, if you have particular rules and you want to show test as specific segments of the market or limit the number of people who see the test, different percentages of traffic and the like, that’s all configurable through our platform’s user interface. So, there are really choices on that, but I think the preferred route — if you want to get the fastest result — is use the randomized sample that’s automatically starting to optimize through that multivariate testing process.”

PEC: Could give us some actual examples of conversion increases that you have seen from ecommerce merchants that have implemented multivariate or split testing, and how that actually impacted their business?

Simpson: “Obviously, the purpose is to increase conversion rate. I think it’s worth saying that defining what that conversion rate is, is fundamental of the outset. While it might seem obvious that sales is the ultimate conversion rate for an ecommerce site — and it is — but there are multiple different conversion rates that go through the site, things like reducing bounce rate of landing pages.

“We have a very large retail client in North America where we reduced the bounce rate of its landing pages by 19 percent through one test, through a redesign of the navigation of that page. Things like bounce and click-through rates are quite important. Specifically, how to get someone to the checkout fast, whether you’re presenting a checkout offer, adding something into the basket, whether you’re giving them an option where the basket is displayed on the page.

“We’ve had widely varied increases in conversion rates through the different tests we’ve done on retail, anything from very small single figure percentages. I think the largest test that we’ve recorded on a retail site was actually to a newsletter signup, but nevertheless it’s still relevant. We had over a 1,000 percent increase — which is 10 times the amount of people actually signing up to a newsletter — through changing the content and the way in which that subscription page worked, which is phenomenal. It is a bit of an outlier to get those sorts of percentages, but certainly low single figure percentage uplifts right through to about 50 percent can be quite common across single tests that companies are doing.”

PEC: The subscription newsletter signup, can you elaborate on that a little bit? Was that a landing page that a merchant’s consumers went to, and your company changed it up?

Simpson: “Yeah, exactly. So, this was after clicking on a newsletter signup button on the home page of the retailer, and once that happened, you went through to a subscription form so you need to fill out your details and your email address and all those elements to subscribe to the newsletter. We looked at testing fundamentally the complete layout of the pages, how much information was taken versus not taken across that. We limited some of the information that was taken, but got massive bumps in conversion rate by doing so.

“So, really think about what is fundamentally important when you’re asking for a newsletter signup. The pitch in terms of what the customer was going to get for signing up to a newsletter was also very important, and how that all worked together with the call to action buttons, the elements that stood out on the page.

“We probably produced up to about 250 or 300 different versions of that page, which is very quick and easy to get to if you look at changing, for example, three areas of a page with just three different versions of the content for each. One of those is for the default. So, you’re only producing two new bits of content for each. You look at three to the power of three, so 27 times, 27 different page combinations. You can multiply that out very, very quickly to get to the hundreds. They’re monitoring all of those. The winning page gave a massive uplift of 1,000-and-something percent — very, very pleasing.”

PEC: What sort of information do you see merchants asking for on a newsletter subscription that they really shouldn’t be?

Simpson: “Different merchants are asking for various different amounts of information. Other than just email address, people are looking at first names, last names, sometimes physical addresses, sometimes phone numbers. You get quite a wide variety of different types of forms out there. I think limiting some of that information, which is not necessary, is vastly important to the actual newsletter subscription, and giving that flat pitch in terms of what they’re going to get from that newsletter subscription is very important as well.”

PEC: Another service your company provides is personalization, where a merchant can change the appearance or change the offer of a site based on the characteristics of a particular consumer. Could you explain that?

Simpson: “We do have a personalization part of our solution. I think personalization is a bit of an overused term, actually. Personalization means many things to many different people. It’s probably worth just spelling out the different areas that we work with retailers on. One is product recommendations, which Amazon really pioneered and mastered and there are many different product recommendations platforms out in the market. So, people who look to these items also look to these, people who bought this also bought that, and those sorts of areas that everyone is very familiar with, with Amazon.

“There are segmentation solutions. We have a segmentation part of our personalization platform where, ultimately, you can tailor different content to different segments of the market, different visitors that come on. You can have similar attributes that might react to different content or is very likely to react to different content, things like people living in a hot climate right now in November versus a cold climate. You should be affecting the different types of clothes you offer them, for example.

“What we mean by true personalization is using mathematics to model what is the very best content to show that individual based on all of their attributes and how they’ve interacted with your brand in the past, so where they live, their zip code, their demographic information, whether they bought things or not in the past, their interest areas on the site and so on and so forth. You can really start to tailor the messaging on the site, the offers on the site, maybe the offers you use to attract them to actually convert on that day, the different products you’re going to push to them. Maybe it’s completely different pages you’re landing people on. You can really adapt the experience for an individual, so your site is the most relevant to them and they’re more likely to transact with you and they’re more likely to be loyal customers for you as well.”

PEC: The more customer visits a site, presumably the more it can be personalized for that customer.

Simpson: “Yes, the algorithms that are used to do this are self-learning. They learn over time, so the more interactions we have with different people on the site — the more traffic a site has — means that the algorithms learn quicker. But they’re pretty good out of the gate as well in terms of being able to show more relevant content to an individual that comes to the site.”

PEC: What can a merchant personalize about a visitor coming to that site for the very first time?

Simpson: “You might want to personalize the content you’re showing them. Maybe you’re recognizing where that person is via their zip code. Maybe you’re recognizing the demographics of that person. You can start putting different products in front of them for that. It might be that you’re just offering completely different products, a completely different product set, or different type of navigation or landing them on a different landing page based on that person’s attributes — such as what they searched on before they came into your site, where they’re located, the time of day or day of week they come onto the site, the weather outside.

“All of those things will affect the attitude of a person when they land on a site and all of those can be used in terms of educating the model and pushing out more relevant content to that individual visitor.”

PEC: How can you determine an attribute like economic status or educational level?

Simpson: “That’s actually a lot easier than most people think. Once we have an IP address here in North America, we can relate that to a zip code. That zip code — through various companies like Mosaic and Experian — will give you a profile of the individual that lives within that zip code and from that you can start seeing a lot of depth. Things like what newspapers people read, what they do on weekends, what types of car they drive — really the type of person that individual is — and using that information can be very powerful in targeting specific content and offers for them.”

PEC: Tell us more about Maxymiser. When was it founded, where is it located, that sort of thing.

Simpson: “I founded the company five years ago in London, England. We expanded through Europe before coming to the U.S. I’ve been living in New York now for about 18 months. The company’s been here a couple of years in North America with active clients. We’re based in New York and San Francisco. We have some small offices in the middle of the country as well. It’s a company that’s growing very quickly at the moment. The split testing and multivariate testing part of the market is really booming right now, so we’re very keen to help more and more retailers in increasing conversion rates to their site.”

PEC: Anything else?

Simpson: “With the holiday period coming up and increased traffic to websites — increased business being done on ecommerce sites — it’s a great time to start optimizing right now. Use that increase in traffic to get faster results, to get quicker education into personalization and really use it to start driving even greater sales through this holiday period. I think it’s a great time right now where people can grasp the amount of business that’s going on and significantly improve that right now.”

Practical Ecommerce

Practical Ecommerce

Bio   •   RSS Feed


Sign up for our email newsletter

Comment ( 1 )

  1. Joe Vlcek December 19, 2011 Reply

    I wanted to add that multivariate testing should be used in a very conservative way. You don’t want to have hundreds of page variations if your site has lower traffic volume. Each variation needs at least several hundred visitors and at least 10 days of testing to get correct results. If you have 300 page variations then you will need around 100,000 visitors to that page and at least 10 days of running the test to really know what version is best.

    We were testing a page for a client recently with only a few variations and one variation had a 300% improvement over the original in the first couple days of testing with over 500 visitors to each variation. The testing software said enough data was used to determine which variation was the winner.

    I could have ended the test at that point and shouted that I got a 300% increase, but that is not entirely true. I left the test running for at least 10 days and sent 4000 visitors to each variation. In the end, the variation with a 300% improvement turned out to be a 25% improvement.

    Imagine if we had hundreds of variations and just threw a few visitors to each variation and listed to the software tell us the winner after a couple days. This is why you will need to extend your tests to 10 days and send at least several hundred visitors to each variation. You will see a much more clear picture and see who the real winner is. The more variations and the fewer the visitors, the more inaccurate your results will be. You could easily see loosing variations as winning variation in the first few days of testing. I have seen this many, many times.

    When it comes to split testing, you don’t want to miss the best path early on. You could easily head down the wrong path and miss the path to the highest revenue. So, take your time with initial tests.

    If you don’t have traffic volume to test hundreds of variations, then you need to research other sites to see what they are doing. What works on other sites might not work on your site, but if you think a specific change will fit on your site and makes sense, you can add it to your testing. Always start testing with your own eyes. Look at your site from a customer’s point of view and ask your customers their opinion.

    You can’t start adding several variations just for the sake of testing or your first test will never get a true result if you don’t have the traffic volume. Use A/B or A/B/C testing initially, then do small multivariate testing to tweak the page. The number of variation used for multivariate testing should depend on traffic volume. You need to have enough traffic for each variation or the test is worthless.

    Sorry for the long reply. I just see many companies getting carried away with conversion testing and they wonder why their revenue does not increase when all of their testing is showing massive improvements in conversions. Test results can be way off if they are not done right.