A Crash Course in Optimization

Split Testing 101: A Crash Course in Optimization


Growing up, I always wanted to be a professional split tester.

Although, I thought this meant I’d be eating banana splits all day.

Split tests in the business world involve a lot less ice cream, but a lot more profit. I’d say they about even out.

If you’re looking to improve your profits per lead or customer, understanding the fundamentals of split testing is the first, and arguably most important, step on your journey.

I’m going to break down what split testing is and what role it plays in, not only conversion optimization, but in increasing your business’ bottom line.

Think of it this way: Conversion rate optimization is the “what,” but split testing is the “how.”

Split Testing 101: CRO Fundam­entals Explained…with @­dalecudmore

Click to tweet

Why Split Test?

Not many people enjoy creating and running split tests, but what they do enjoy are the results.

Split tests are a statistically backed method that tells you if a particular change, usually on your website, will lead to more profit.

Even small improvements in your conversion rate can lead to tens of thousands in extra profit every year for the foreseeable future.

Imagine if you could improve your revenue by 400% by simply removing an image? While that’s an extreme case, most businesses can improve their conversion rate by X% much easier than increasing their lead quantity or quality by 8%.

Essent­ially, conversion rate o­ptimization can be a relatively low effort, high reward activity that relies on split testing.

Click to tweet

Going Deeper: What is a Split Test?

The most basic split test is an A/B split test.

You have an A version (the original), and a B version (the variation) of whatever you wanted to test.

You “split” your traffic, and send half to the A version of your page, and half to the B version.

Here’s a basic example, where the test is to see if a red button will convert better than the original grey button:

basic button split test

Over time, you measure and determine which version converts more visitors into sign-ups.

In the case of the image of the test above, the variation converted at 4.5%, which is much better than the original 1%.

Because variation B “won,” it becomes the new standard, and will be “variation A” in any future tests. Make sense?

An Alternative: Multivariate Split Tests

Okay, so changing button color seems like a good thing to test, but what if you also want to test a different headline?

The simplest option is to do the button test and then, after you have a winner, you can start the headline test.

However, another option is to do a multivariate test. It’s essentially the same as any other split test, but you can test multiple changes at the same time.

The downside of this is that it takes much longer to run.

Suppose you wanted to test one variation of both the button and headline. Now you have 4 total variations:

Button Color Headline
Variation A Grey Headline 1
Variation B Red Headline 1
Variation C Grey Headline 2
Variation D Red Headline 2

And of course, you need to split your traffic equally 4 ways, which is why reaching a significant conclusion takes longer.

A Few Important Definitions

There are a few terms I mentioned above that you may or may not be familiar with. This is a good time to define a few important terms that you will see over and over as you study conversion optimization.

Conversion rate: The percentage of people who take a desired action. For example, if 100 people arrived on your homepage and 3 people signed up for a demo, that would be a 3% conversion rate.

Overall conversion rate: Whenever possible, optimize your overall conversion rate, which is the amount of customers that actually purchase your service or product. If 1 person purchases your offer out of 100 visitors, you have a 1% overall conversion rate, regardless of the steps in-between.

Note: In many cases, it’s difficult to measure the impact that a change has on your overall conversion rate. In these cases, you have to resort to optimizing the intermediate steps (like requesting a demo) that should lead to increased sales.

Sales funnel: Your sales funnel breaks down all the actions a user takes, starting from landing on your website, all the way to making a final purchase. Note that you can have multiple sales funnels. See a basic funnel below:

basic website conversion funnel

Statistical significance (or just significance): All tests have variation. Variance will even out in the long run, so as long as you reach a high level of statistical significance (or confidence), you can be confident that you’re picking the right variation as the winner.

Significance is usually stated as a percentage. If a split test result has a confidence level (significance) of 95%, that means that 19 out of 20 times, you’ve chosen the right winner. However, that means that 1 out of 20 times you falsely picked the wrong variation as the winner.

Sample size: The sample size refers to the amount of traffic sent to each variation in your split test. The higher your sample size, the more accurate your results will be. Most split testing tools have a sample size calculator built in, but there are also many stand-alone sample size calculators you can use.

What Should You Split Test?

Conversion test hypothesis

You’re probably a bit excited to get going, but hold on a second.

What you don’t want to do is start split testing elements of your website at random.

To decide what you need to split test, you need to first develop a conversion rate optimization (CRO) plan.

With a properly thought-out CRO plan, you’ll know where the biggest opportunities for improvement are , where any big leaks are, and you’ll be able to come up with a hypothesis to split test.

Don’t worry if you don’t know how to come up with a hypothesis right now, you’ll learn that over time. In general, these are some questions that will get you started about the right things that might be having a negative effect on your conversion rate:

  • are any copywriting messages simple and clear to understand?
  • is the user experience good? (i.e., everything is where the user expects it to be)
  • do the most important elements stand out?
  • what objections do visitors have when they come to this page? Do we address them?

The more you think about and clarify user interaction in general when it comes to conversion, the better you will get at diagnosing leaks costing you customers and profit.

How Do You Split Test?

Say you do all that work and come up with a split test hypothesis… what comes next?

Actually doing the split test is fairly simple in most cases nowadays, even if you can’t code to save your life.

Click to tweet

There are tons of split testing tools that will not only help you create a variation of a page and split traffic between your options, but will also record the results and compute the statistics to tell you which variation won.

All you’ll need to do is modify the single element that you want to test, and then follow the simple instructions on one of those tools like Optimizely or Visual Web Optimizer to implement the test on your site.

At the end, all you’ll need to do is record the result, make any necessary changes to your site, and then start the process all over again.

Remember, split testing is a long-term strategy , so even though you’ll have tests that don’t produce any improvement, overall you will increase your conversion significantly. When you first start out, don’t judge the results of your efforts for at least 6 to 12 months.

Case Study Example: Does a Security Seal Hurt Conversions?

split test case study

(image source)

Let’s look at a real-life example to help clarify the split testing process.

The picture shows a basic email optin form. Originally, it had a security seal, the TRUSTe image, right beside the submit button.

The variation was the exact same form, minus the security seal.

The results: Conversions increased by 12.6% and variation B was declared the winner.

After every test, you need to try to explain why the winning result won. In this case, it’s likely that variation B won because users were put off by the security seal.

Usually, security seals are displayed when there is some sort of monetary transaction. Even though this is just a simple email signup form, this association was likely enough to make users pause and reconsider what exactly they were filling out. As a result, it scared away a significant fraction of potential leads.

Conclusion and Next Steps

Conversion rate optimization can be intimidating when you’re first starting out, but it always comes back to the fundamental concepts we looked at here.

I recommend going through our Beginner’s Guide to Creating a Conversion Rate Optimization Plan to discover what else you need to learn about before getting started.

All this effort will be worth it, as a solid CRO plan can lead to consistent, significant growth for as long as you run your business.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s