A/B Testing, or Split Testing, is a method of testing one change on a page – the wording of a headline or the placement of a button, for example – to see how it performs compared to the original. Users follow one URL and are either shown the original (version A, the control) or redirected to an edited version (version B, the treatment). For information on testing multiple changes in different combinations, see Multivariate Testing.
SCHEDULE TIME & GATHER MATERIALS
Preparation: 1 Day (depending on your organization)
Testing Time: 1-3 Weeks
Analysis: As quick as looking at a graph or table
An editable version of your live site (this will depend on the platform on which your site is built and what testing tool you use)
A/B testing software subscription (see below for links)
CARRY OUT THIS METHOD
Determine your success metric. This may be time spent on your site, sales, conversions, e-mail signups, or number of support tickets submitted.
Determine the page element that needs to be altered and tested. This may be headline wording, call-to-action visibility, images, or any page element that has been ambiguous to test users.
Formulate a hypothesis regarding how to fix your problematic page element. Apply that fix to a new version (“version B”) of your site, leaving the original version (“version A”) unchanged.
Use an A/B testing tool to serve version B to a randomly chosen subset of users.
Run the test for approximately two weeks in order to reach a statistically significant number of users.
TIPS AND RESOURCES
Try these tips
Serve the original version of the page to Google’s bots to avoid being penalized in search results.
Avoid the temptation to stop the test as soon as you see a statistically significant difference between the two pages. Continue the test until its scheduled end. This is especially important if you plan to publish results or your research is funded with grant money.
Serve both versions to users concurrently in order to account for seasonal variables.
Explore more resources
Wikipedia: A/B testing
Visual Website Optimizer: The Complete Guide to A/B Testing
Smashing Magazine: The Ultimate Guide To A/B Testing
Google Webmaster Central Blog: Website testing & Google search
MeasuringU: 10 Things To Know About A/B Testing
MeasuringU: A/B Test Calculator
Conversion Rate Experts: Split Testing Software