top of page
A/B Testing

A/B Testing, or Split Testing, is a method of testing one change on a page – the wording of a headline or the placement of a button, for example – to see how it performs compared to the original. Users follow one URL and are either shown the original (version A, the control) or redirected to an edited version (version B, the treatment). For information on testing multiple changes in different combinations, see Multivariate Testing.

SCHEDULE TIME & GATHER MATERIALS

Schedule Time

  • Preparation: 1 Day (depending on your organization) 

  • Testing Time: 1-3 Weeks 

  • Analysis: As quick as looking at a graph or table

Gather Materials

  • An editable version of your live site (this will depend on the platform on which your site is built and what testing tool you use) 

  • A/B testing software subscription (see below for links)

CARRY OUT THIS METHOD

  1. Determine your success metric. This may be time spent on your site, sales, conversions, e-mail signups, or number of support tickets submitted.

  2. Determine the page element that needs to be altered and tested. This may be headline wording, call-to-action visibility, images, or any page element that has been ambiguous to test users.

  3. Formulate a hypothesis regarding how to fix your problematic page element. Apply that fix to a new version (“version B”) of your site, leaving the original version (“version A”) unchanged.

  4. Use an A/B testing tool to serve version B to a randomly chosen subset of users.

  5. Run the test for approximately two weeks in order to reach a statistically significant number of users.

TIPS AND RESOURCES

Try these tips

  • Serve the original version of the page to Google’s bots to avoid being penalized in search results.

  • Avoid the temptation to stop the test as soon as you see a statistically significant difference between the two pages. Continue the test until its scheduled end. This is especially important if you plan to publish results or your research is funded with grant money.

  • Serve both versions to users concurrently in order to account for seasonal variables.

bottom of page