Overview

Split tests allow you to discover which version of an email is most effective. You can create different variations on an email (three versions with different subject lines, for example) and send them out to your contact list at random, in equal numbers. The statistics that are returned let you see which version has the highest open or click-through rate. Lead Liaison also supports automatic split testing, in which the “candidate” email versions are sent to only a small proportion of your contact list, and the “winner” version is then sent to the remaining contacts in the list.

Split testing, also known as “A/B testing”, is a scientific method of testing your email campaigns’ effectiveness using statistical analysis. When you perform a normal campaign send, you send one email to your whole contact list. You can then see how well it performed using the email software’s statistical analysis tools. However, when you perform a split test you “split” your contact list into different parts and send a different email campaign to each part. Then, you can see how well each email performed and select the most effective.

Split testing is essentially a competition between email campaigns, in which the losers are discarded and the winner is used in future campaigns. You can select the criteria on which to judge the winner (open rate or click-through rate) and the email software tracks each email’s performance and decides the winner.

Split tests are usually performed on variations of the same basic email campaign, not on vastly different emails. For example, you might split test two emails that are identical except for their slightly different subject lines, in order to see which variation is opened most often. However, you would not split test one email advertising a seasonal discount on store merchandise and another offering a free holiday; this would not produce meaningful results.

The term “A/B testing” suggests a split test must be a competition between two emails. However, this is not the case with the email software, which allows you to create tests using any number of emails. This means that you can perform highly detailed and efficient split tests that compare multiple email variations simultaneously.

Manual and Automatic Split Testing

Lead Liaison supports both manual and automatic split testing.

• When you use manual split testing, the email software sends the email campaigns you have included in the test, tracks each email, and lets you view their effectiveness in the Manage > Emails > Statistics > Split Test Statistics screen. Manual split tests have no cut-off time.

Manual Split Testing:

Once you have discovered the most effective email campaign, it’s up to you what you do next. You could use the winning email for your next campaign, send the winning email to another contact list, or perform further split tests on variations of the winning email (an example of which is shown in the following figure) in order to further refine its effectiveness.

Multi-Level Split Testing:

When you use automatic split testing, the test emails are sent to only a percentage of your contact list’s contacts (you define the percentage). You also define the test’s duration. The email software sends the test emails and tracks their performance. Once the test duration has elapsed, the email software declares a winner based on the criterion you selected, and automatically sends the winning email to the rest of your contact list.

Automatic Split Testing:

Split Testing Tips

If you have not used split testing before, here are some tips that might be helpful:

1. Don’t “compare apples and oranges”. Make sure that the emails you are testing are variations of one email, not completely different emails. Split testing is designed to help fine-tune your email marketing, not make broad decisions on marketing direction.

2. Keep it simple. As with any scientific test, results are only meaningful if they compare two things that differ only in a single feature. For example, test two emails that differ only in their font colors, or two emails that differ only in their subject lines. Don’t test one email against another that has a different subject line, graphic design and body text; the results would not let you know which element is affecting performance. Lead Liaison lets you test an unlimited number of emails simultaneously. For the most informative results when testing multiple variations, create an email campaign for each variation or combination of variations.

3. Include a “control”. In scientific testing, a “control” is an element of the test that provides a base result against which other results can be compared. For instance, in a pharmaceutical test a section of the test group is given a placebo instead of the real drug. This allows the scientist conducting the test to exclude the influence of other variables on the results. In email split testing this means that you should always include the original version of the email, which has had no changes or variations applied to it. This is particularly important if you are split testing variations of an email campaign that you have used in the past. In order to accurately gauge the effectiveness of each new variation, you must compare it against the control email (which is sent at the same time as the variations) and NOT the email you sent in the past. If you compared the new variations’ performance against that of the email you sent in the past, your results could be affected by a multitude of variables (time of day the email was sent, the day of the week, the season, economic conditions, and many more) that would severely impair the comparison.

4. Use a large test group. Like any statistical analysis (political polls, for example) the larger the test group you use, the more accurate and reliable your results will be.
5. Use the correct metric. The email software lets you base your split test on open rates or click-through rates (links clicked). Make sure you select the metric that makes sense based on the type of variation you’re testing. If you are testing variations in your subject line, it makes sense to use the open rate (since recipients open - or don’t open - emails based largely on the subject line). If you are testing variations in your email’s graphic design, it makes sense to use the click-through rate; it would be useless to base the test on the emails’ open rate, since recipients do not see an email’s design until after they have opened it.

Viewing and Managing Split Tests

• To view details about the email campaign split tests configured in the email software, click Content > Emails in the navigation bar, then click the Split Tests tab.

• Each area is described in more detail below:
  • New = Click this to begin configuring a new split test. 
• Highlighting one of the split tests will give the options to:
  • Delete Selected = Select one or more split test’s checkboxes and click this to remove the split test from the email software. Deleted split tests cannot be retrieved.
  • Checkboxes = Select a box belonging to an email campaign before choosing an option from the Choose an action list. Select the box at the top of the column to select all campaigns.
  • Name = This displays the name of the split test.
  • Campaigns = This displays the name of each email campaign in the test. Note: This is the name that displays internally Lead Liaison, not the email subject line.
  • Type = This displays the type of test:
    • Split test displays if you selected Find a winning email and show me the results when you created the test. The campaigns are sent to the whole contact list in equal proportion, and the winner is displayed to you.
    • Best performing displays if you selected Find a winning email then send it to my list automatically when you created the test. The campaigns are sent to a small part of your contact list, and the winner is automatically sent to the rest of the list.
      Note: If there are multiple newsletters with the same score in the best category then the rest of the split test campaign turns into an ‘evenly distributed’ campaign with a rotation between newsletters being sent.
  • Created = This displays the date on which the test was created.
  • Last Sent = This displays the date on which the test was last performed. If the test has never been performed, Not sent displays.
• The following actions can be taken:
  • Click Waiting to send to see when a scheduled test will be sent. The View scheduled email queue screen displays (see Viewing Scheduled Email Campaigns). Note: This option displays only when a test is scheduled and has yet to be sent.

  • Click a test’s Pause link to stop sending the test. Click its Resume link to begin sending the test again. Note: These options display only when a test is currently being sent.

  • Click a test’s Send link to perform the test. Note: Before you send Best performing tests, you need to enable Split test campaigns in the Settings > Cron settings screen.

  • Click a test’s Edit link to make changes to the test. See Creating and Editing Split Tests for information on the screen that displays.
  • Click a test’s Copy link to create a new test with the same properties as the original. The new test displays in the list with Copy of prepended to its name.
  • Click a test’s Delete link to remove the test from the email software. Deleted tests cannot be retrieved. Note: When you delete a test, its associated email campaigns are not deleted.

Performing a Split Test

To send a split test to members of your contact lists:

• Click Content > Emails  in the navigation bar, then click Split Tests tab. Click the relevant split test’s Send link. Alternatively, while creating or editing a split test, click its Save and send button (see Creating and Editing Split Tests).
• The first screen that displays allows you to setup the split test campaign.

• Basic Info
  
  • Reply Email - Configures the reply-to address for the campaign.
• From 
  • Sender - Configures the email address you will be sending the campaign from
• To Email 
  • Send to Email - Lets you configure if you should send the campaign to a prospects Personal, Work, or known Primary email address
• To 
  • Select the contact list(s) or segment(s) to which you want to send the email campaign. Use the search box below the list to search for contact lists or segments.
• Send Time
  
  • Configures sending the email immediately, or at a specific time
• Advanced Options
  • Notify list owner on start / stop - Select this if you want to send an email to the list owner when a scheduled campaign starts sending, and another when it finishes sending.
  • Sending Server - If you have multiple email sending servers you can pick which one will be used
  • Personalize To - Lets you setup / configure what name will be shown as part of the To address
  • Send as multi-part - Select this to send both HTML and text elements of a multipart email campaign to recipients. This allows a recipient’s email client to choose the correct display method. Note: Use this option if you are unsure. Also, use this option if you don’t provide your recipients with the ability to choose HTML or text emails.
  • Embed Images in content - Select this to embed the images you send in HTML emails, rather than linking to them. This allows recipients to view the email campaign offline, but can increase the email size significantly.
• Click Next

• In the screen that displays, review your split test.
  

• Click Schedule when you are ready to send your test.
• Your Split Test will now appear in the email sending queue:

Congratulations, your Split Test has been sent!

Viewing Split Test Statistics

To view statistics about the split tests you have performed, including the winner of each test, click Reports > Emails  in the navigation bar, You will be able to see all the campaigns that were part of the split test along with it's statistics. Note: This screen displays information about all campaigns, not just split tests. 

Creating a Split Test

To create a new split test, click Content > Emails > Split Tests tab in the navigation bar. Click the New button. To edit an existing split test, click Split Tests in the navigation bar. Then click the relevant test’s Edit link. Note: This screen allows you to set up your test; you can choose the people to whom you want to send it later on.

• Each area of the screenshot above is explained in detail below:
  • Name = Enter a name for this split test. The name only displays in the configuration interface, and not to your contacts. 
  • Email Campaigns = Select the email campaigns that you want to include in this test. Note: You can select as many email campaigns as you want.
  • Based On = Use this field to specify how the winning email campaign is chosen. Note: When you select this option, the email software counts the total number clicks on all links in each email campaign, not any one link in particular.

    • Select Number of people who open the email to choose the email campaign that is opened most often.

    • Select Number of people who click a link in the email to choose the email in which a link is clicked most often (the one with the highest click-through rate).

  • Test Type = Use this section to define whether to perform a manual test or an automatic test, and to configure how the test should proceed:
    • Select Find a winning email and show me the results if you want to perform a manual test. When you select this option, the email software splits the contact list (or lists) into equal parts and sends each email campaign to one part. You can then view each email campaigns performance in the Statistics > Split test statistics screens. For example, if you select two email campaigns and have 1000 people in your contact list, each email campaign is sent to 500 people. Note: If you choose this option, you may want to first set up a new contact list for test subjects, rather than sending it to your complete contact list. If so, see Creating and Editing Contact Lists.
    • Select Find a winning email then send it to my list automatically if you want to perform an automatic test. When you select this option, you can specify the percentage of your contact list’s contacts that are test subjects. The email software performs the test, then waits the amount of time you specify before declaring a winning email campaign (based on the criteria you select). The email software then sends the winning email campaign to the remainder of your contact list’s contacts (those not in the test group). In the Emails will be sent in equal groups to (...) % of your list field, enter the percentage of your contact list’s contacts that should be in the test group. The default is 10%. In the The best performing email will then be sent to the rest of your list (...) later field, select the number of Hours or Days after which the email software decides the winning email. Note: The minimum test time is two hours, and the maximum is thirty days. For example, if you have a contact list of 5000 people, select five email campaigns, specify a test group of ten percent and a test time of one day, each email is sent to 100 people, and the winning email is sent to the remaining 4,500 people the next day.

  • Save = Click this to save your changes to the test and return to the View split tests screen.

  • Cancel = Click this to return to the View split tests screen without saving your changes.