A/B Testing: A Step-by-Step Guide

Want to improve your website’s conversion rate? Or the open rate on your email newsletter? Or one of a dozen other marketing channels your business is active in?

You’re in luck. This article is all about how to do exactly that, through A/B testing methodology.

Digital marketers are always talking about analytics and improving your results compared to competitors. But how does that actually happen?

Maybe they’re magically better at writing copy, or designing awesome graphics. Usually, though, if Company A is better at getting results than Company B, it’s because they know how to properly test, analyze, and act on the results of their efforts.

If good A/B testing were easy, everyone would do it and have amazing results. Yet most brands don’t. So let’s figure out how to separate your business from the pack, shall we?

What is A/B Testing, or Split Testing?

A/B testing is when you use two (or more) different versions of a marketing piece to compare results between the two. This is also sometimes called split testing.

You could test a very large element, say, two entirely different emails sent to newsletter subscribers. Or it could be something very small, such as the header text on a web page that is otherwise entirely identical.

We usually think of A/B testing for websites, where you’ll test two different versions of a page to see which one performs better. This is far from the only use for it, even though it’s a popular one.

Split URL Testing

A simple A/B test can be to have two versions of a web page. You then run identical ads for your product or service, but approximately half of the ads link to one page or the other.

If one of them is markedly better (or worse) than the other at converting the leads you get to the page, you have a strong data point on which you should use, and also the type of landing page you should create in the future to replicate those results.

Multivariate Testing

Multivariate testing is more or less what it sounds like. It’s a split test that includes more than a single variable and/or more than two different versions.

An ad run on social media might have four different versions: two each with a different color scheme and central image, and two each with different copy and call-to-action text. Mixing and matching each of these will result in four variations on the same ad.

As we’ll talk about a bit later, multivariate testing can be trickier than regular A/B testing, because you’re testing multiple things at once. It can be harder to identify which elements are driving the change. Using our example just above: Did Version C of your social ad perform the best because of the copy or color scheme? Do the results from the other variations give us information to form a reliable conclusion? Sometimes the answer is no.

However, there are still times when multivariate testing can work wonders, and speed up the process of A/B testing. This is what we’ll cover in the analysis portion of this article below.

Segmentation in Testing

Another concept that’s important to understand is audience segmentation. You can’t necessarily do this with website visitors, but many ad platforms, email platforms and other digital marketing channels will allow you to split your audience into groups.

Audience segmentation is important, because it can be another variable that invalidates your test.

Let’s say you have two versions of an email and you want to test open rates for both. You then split your audience roughly down the middle…

…but the split is between men and women. Or those over the age of 50 and under 50.

This can confound a test, because maybe the over 50 audience opened at a higher rate. You don’t necessarily know if it was the email that was better, or if your older subscribers are simply more likely to open your emails.

Different demographics have different digital habits, so when you segment an audience, it’s important to create groups that are as similar as possible to one another.

Conversely, maybe you want to test email open rates only for women, who have been opening at lower rates in recent months. Here, segmentation based on gender could be exactly what you need to draw actionable insights.

Offline Testing

We’re focusing on digital marketing and websites in this article, but A/B testing isn’t limited to the digital world.

For an example of how A/B testing can make the jump offline, let’s say you are running a local print ad in two similar townships. Assuming the demographics of these townships are similar (otherwise it could be a faulty test), some publishers will allow you to run different ads on a city, township, zip code or even neighborhood level.

If you were to link those different ads either to different web pages or phone numbers, you could then track and compare the results between them.

Reasons to A/B Test

Testing without a plan can be wasteful. Having specific goals in mind can help. Here are some of the major reasons you might want to methodically A/B test your website or marketing platforms.

1. Increase ROI - who doesn’t want better revenue? You could even test for this with successful marketing channels, to make them even better!
2. Customer Retention - making more engaging content, on your website or otherwise, is more likely to keep existing and potential customers invested in your brand. That potential customer isn’t any good to you if they remove the bookmark from your website or unsubscribe from your emails, even if they’re not looking to purchase from you right now.
3. Create Word of Mouth Marketing - the best marketing is the kind you don’t need to promote yourself. Your audience does it for you. Make more engaging content and this effect will start to snowball throughout your audience and their digital peers.
4. Compare Between Marketing Channels - You have a limited budget, and only so much time in a day to devote to your marketing efforts. Where is that effort best put to use? By testing and refining, you can understand the limits of each marketing channel and make better decisions on where to allot your time and energy.
5. Gather Data For Future Campaigns - looking to redesign your website? What’s working now? What isn’t? You may not know, and A/B testing can help you, so that a new design keeps the best of your current online presence.

That list isn’t comprehensive, but it should give you a good idea of some of the end results of good A/B testing.

Creating Specific Goals

You’ve probably heard about SMART goals: Specific, Measurable, Attainable, Relevant and Time-Bound.

This is a good framework for creating goals for your A/B tests, which should be done before you do the tests.

What do you want to accomplish? Multiply revenue by 10 times via Google Ads? That’s probably not attainable. Nor is it time-bound.

A better example of a goal along these lines could be: “We’ll increase ROI in our Google Ads efforts by 10% over the next 6 months via a series of A/B tests within our core demographics.”

This gives your team something realistic to shoot for, rather than simply saying that we want to improve as much as possible.

How to Split Test Your Website

We talked a little bit about this above, but split testing a website is something everyone should be doing who is serious about generating consistent results from their website.

There are different types of tests you can do, and we’ve listed some common ones before.

Many website platforms, such as Hubspot or various Wordpress plugins, will allow you to split test individual elements on web pages. If you are visiting a website that was created in one of these platforms, chances are you’ve seen a version of a page that not everyone else is seeing. And your behavior on the page was one data point among thousands or even millions that are used to make iterative changes to the site.

So what types of things are being tested, and why?

Navigation

It’s well-known that the fewer clicks visitors have to make on a website to take a desired action, the more likely they are to follow through with that action. So it makes sense to streamline your navigation.

A/B tests can be run to see which paths through a website create the highest conversion rate. This means testing different navigational links on key pages on your site.

But what about when a visitor isn’t looking to purchase or take an action, but is still interested in information from your brand? A good test can be to alter navigation to see which pieces of content on your site have the highest retention rate.

Stated differently, what content keeps people on the site the longest? This is good for your site’s SEO as well as what it means about the user’s experience with your brand.

Imagery

Should you have that landscape photo of a sunrise or a picture of a smiling baby?

Should the large button that says “Buy Now” be green or red? Or should it say “Get Yours Today” instead?

Should the entire look and feel of your website be warmer or use earth tones?

These are among the things you can A/B test. That last one even hints at something larger: namely, A/B testing variables like color schemes that run across your entire website, not just a single page.

Because make no mistake, each of these questions has a “right” answer for your brand. But you won’t know which is right and wrong until you gather the data for it.

Content Length

Are you losing customers because they don’t know enough about your products or services, and therefore are unwilling to trust your brand enough to make a purchase?

Conversely, are you bombarding potential customers with too much information, overloading them until they decide to go elsewhere?

The answer probably depends on things like the education level of your average customer, the amount they’re being asked to spend, and how complicated the product or service you’re offering is.

Which means there isn’t one correct answer, but there’s a correct answer for your brand and suite of products and services. And you can discover it through split testing.

Calls to Action

Calls to action, or CTAs, are some of the most important items on your website.

What’s a CTA? In brief, it’s anything that prompts a user to take a specific action. This could be a “Subscribe” option for your newsletter, a “Download” button for an eBook, a “Schedule Demo” option for a service you offer, or a “Buy Now” button for a product. Or a lot of others!

The placement of CTAs on a page matters. The size of them matters. The color matters. The frequency with which you present users with CTAs matters. The copy on them matters. The visuals and copy surrounding the CTA matter.

It’s not an embellishment to say that a website with poor CTAs and one with optimized, thoroughly tested CTAs, can be the difference between a wildly successful business and one that’s filing for bankruptcy.

A/B Testing Marketing Channels

A/B testing doesn’t have to be limited to websites. We’ve mentioned email marketing a couple times now, and below we talk more in-depth about this. But there are other common marketing platforms that can benefit from A/B testing.

Paid Search Ads

Google or Bing ads can be excellent opportunities to A/B test different copy.

Increasingly, these platforms are even starting to build A/B functionality into their interface, so that you don’t have to manually alter ads when you want to test a new one.

It’s important to note, however, that you want to be careful about running concurrent ads that are trying to market to a similar audience and/or selling the same product or service.

You can end up competing with yourself for the same audience hurting both ads’ performance or, worse, driving up the cost of both of them.

The flip side of this is that if you run ads back-to-back in different time windows, you need to make sure that the results aren’t based on seasonality. If you’re an eCommerce site, your ads are likely to have better results as we near the winter holidays. If you’re a heating and cooling company, you’ll see spikes in results during the hottest and coldest months of the year.

These things don’t mean that the ad is better, just that the outside conditions are better for the ad to flourish. Managing your variables in this manner is extremely important to avoid incorrect insights.

A/B Testing Emails

The two biggest metrics that you’ll always have in email marketing are open rate and clickthrough rate.

RELATED: Email Marketing: Tools, Strategies and Best Practices

A/B testing to increase your open rate usually involves testing out different subject lines, since this is all people will see before they decide to open your email or not.

Clickthrough rate refers to how often someone clicks on something within the email, usually taking them to your website. To improve this, you’ll want to experiment with different layouts and calls-to-action, or perhaps varying the type and volume of CTAs you use.

Social Media Posts

Social media has metrics that don’t exist on other platforms, so occasionally you have to create tests that are specifically catered to your social media channels.

Likes, Subscribes, Shares and Comments are all valid metrics to look at, depending on what your goals are.

How do you test things things? Change the time of day you post to social channels. Or the tone of voice that you use. Does a question or poll increase engagement? Or an image your audience can identify with? Does video work best? What types of video? How long?

You could run hundreds of A/B tests and still have more data that you could collect. The point, though, isn’t to do every possible test, but to figure out a formula that consistently achieves the results you’re looking for.

Analytics of Evaluating A/B Tests

Properly preparing tests and analyzing results is half the battle. All the good intentions in the world won’t lead to better business decisions if your data is flawed. To that end, what should you be aware of before evaluating an A/B test?

Collect Prior Metrics

What’s your conversion rate on your website currently? What’s the conversion rate of a particular page? How many newsletter sign-ups do you receive in a month, and what is this number as a percentage of total web traffic?

Whatever your test, you first need to establish a baseline set of data to compare it against. Otherwise, you have no way of tracking progress (or lack thereof).

Avoiding Common Data Errors

Working with statistics is difficult. Even intelligent people can be swayed by biased or incomplete statistics, or believe they have insights that the data doesn’t corroborate.

So how to avoid this?

The first is making sure you’re acquiring enough data, which means enough to determine statistical significance. There are statistical significance calculators, for instance, that can help with this.

This also involves proper segmentation, as we discussed earlier, and running a test for a long enough duration to avoid any unexpected spikes in data that can happen on a day-to-day basis.

The second is making sure you’re not testing too many variables at once. As we mentioned earlier, even the time of year can change results in many industries. And testing multiple variables

Multivariate testing is still viable in the right setting. But if you’re just starting out, you should probably only be looking at a single variable at a time, to make sure you are properly interpreting the results.

Lastly, admitting that a test has failed is important (and some of them will!). The test might fail because results go down, or because the results aren’t statistically significant. Either way, this is an iterative process that will take time.

A/B Testing Conclusions

By now you should have a good idea of how to prepare and execute a series of A/B tests that will increase your website’s results, or the results of other digital marketing channels.

Establish your goals and benchmarks, isolate the variables you want to test for, then ensure you’re interpreting results in ways that the data supports.

Then you repeat, over and over, until you have improved significantly.

For some companies, this process never ends. Their website will be running A/B tests 24/7/365, and they’re slowly improving as they analyze the results from the thousands of small tests they run.

You might not need this level of automated execution, but it’s possible, and is a clear sign of the power that A/B testing can have for a business and its long-term success.