Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 7 Next »

Overview

Split tests allow you to discover which version of an email is most effective. You can create different variations on an email (three versions with different subject lines, for example) and send them out to your contact list at random, in equal numbers. The statistics that are returned let you see which version has the highest open or click-through rate. Lead Liaison also supports automatic split testing, in which the “candidate” email versions are sent to only a small proportion of your contact list, and the “winner” version is then sent to the remaining contacts in the list.

Split testing, also known as “A/B testing”, is a scientific method of testing your email campaigns’ effectiveness using statistical analysis. When you perform a normal campaign send, you send one email to your whole contact list. You can then see how well it performed using the email software’s statistical analysis tools. However, when you perform a split test you “split” your contact list into different parts and send a different email campaign to each part. Then, you can see how well each email performed and select the most effective.

Split testing is essentially a competition between email campaigns, in which the losers are discarded and the winner is used in future campaigns. You can select the criteria on which to judge the winner (open rate or click-through rate) and the email software tracks each email’s performance and decides the winner.

Split tests are usually performed on variations of the same basic email campaign, not on vastly different emails. For example, you might split test two emails that are identical except for their slightly different subject lines, in order to see which variation is opened most often. However, you would not split test one email advertising a seasonal discount on store merchandise and another offering a free holiday; this would not produce meaningful results.

The term “A/B testing” suggests a split test must be a competition between two emails. However, this is not the case with the email software, which allows you to create tests using any number of emails. This means that you can perform highly detailed and efficient split tests that compare multiple email variations simultaneously.

Manual and Automatic Split Testing

Lead Liaison supports both manual and automatic split testing.

  • When you use manual split testing, the email software sends the email campaigns you have included in the test, tracks each email, and lets you view their effectiveness in the Manage > Emails > Statistics > Split Test Statistics screen. Manual split tests have no cut-off time.

Manual Split Testing:

Once you have discovered the most effective email campaign, it’s up to you what you do next. You could use the winning email for your next campaign, send the winning email to another contact list, or perform further split tests on variations of the winning email (an example of which is shown in the following figure) in order to further refine its effectiveness.

Multi-Level Split Testing:

When you use automatic split testing, the test emails are sent to only a percentage of your contact list’s contacts (you define the percentage). You also define the test’s duration. The email software sends the test emails and tracks their performance. Once the test duration has elapsed, the email software declares a winner based on the criterion you selected, and automatically sends the winning email to the rest of your contact list.

Automatic Split Testing:

Split Testing Tips

If you have not used split testing before, here are some tips that might be helpful:

  1. Don’t “compare apples and oranges”. Make sure that the emails you are testing are variations of one email, not completely different emails. Split testing is designed to help fine-tune your email marketing, not make broad decisions on marketing direction.
  2. Keep it simple. As with any scientific test, results are only meaningful if they compare two things that differ only in a single feature. For example, test two emails that differ only in their font colors, or two emails that differ only in their subject lines. Don’t test one email against another that has a different subject line, graphic design and body text; the results would not let you know which element is affecting performance. Lead Liaison lets you test an unlimited number of emails simultaneously. For the most informative results when testing multiple variations, create an email campaign for each variation or combination of variations.

  3. Include a “control”. In scientific testing, a “control” is an element of the test that provides a base result against which other results can be compared. For instance, in a pharmaceutical test a section of the test group is given a placebo instead of the real drug. This allows the scientist conducting the test to exclude the influence of other variables on the results. In email split testing this means that you should always include the original version of the email, which has had no changes or variations applied to it. This is particularly important if you are split testing variations of an email campaign that you have used in the past. In order to accurately gauge the effectiveness of each new variation, you must compare it against the control email (which is sent at the same time as the variations) and NOT the email you sent in the past. If you compared the new variations’ performance against that of the email you sent in the past, your results could be affected by a multitude of variables (time of day the email was sent, the day of the week, the season, economic conditions, and many more) that would severely impair the comparison.

  4. Use a large test group. Like any statistical analysis (political polls, for example) the larger the test group you use, the more accurate and reliable your results will be.
  5. Use the correct metric. The email software lets you base your split test on open rates or click-through rates (links clicked). Make sure you select the metric that makes sense based on the type of variation you’re testing. If you are testing variations in your subject line, it makes sense to use the open rate (since recipients open - or don’t open - emails based largely on the subject line). If you are testing variations in your email’s graphic design, it makes sense to use the click-through rate; it would be useless to base the test on the emails’ open rate, since recipients do not see an email’s design until after they have opened it.

Viewing and Managing Split Tests

  • To view details about the email campaign split tests configured in the email software, click Manage > Emails > Email Campaigns in the navigation bar, then click View split tests.

  • Each area is described in more detail below:
    • Create a Split Test button = Click this to begin configuring a new split test. 
    • Delete Selected = Select one or more split test’s checkboxes and click this to remove the split test from the email software. Deleted split tests cannot be retrieved.
    • CheckboxesSelect a box belonging to an email campaign before choosing an option from the Choose an action list. Select the box at the top of the column to select all campaigns.
    • Name = This displays the name of the split test.
    • Campaigns = This displays the name of each email campaign in the test. Note: This is the name that displays internally Lead Liaison, not the email subject line.
    • TypeThis displays the type of test:
      • Split test displays if you selected Find a winning email and show me the results when you created the test. The campaigns are sent to the whole contact list in equal proportion, and the winner is displayed to you.
      • Best performing displays if you selected Find a winning email then send it to my list automatically when you created the test. The campaigns are sent to a small part of your contact list, and the winner is automatically sent to the rest of the list.
    • Created = This displays the date on which the test was created.
    • Last Sent = This displays the date on which the test was last performed. If the test has never been performed, Not sent displays.
  • The following actions can be taken:
    • Click Waiting to send to see when a scheduled test will be sent. The View scheduled email queue screen displays (see Viewing Scheduled Email Campaigns). Note: This option displays only when a test is scheduled and has yet to be sent.
    • Click a test’s Pause link to stop sending the test. Click its Resume link to begin sending the test again. Note: These options display only when a test is currently being sent.
    • Click a test’s Send link to perform the test. Note: Before you send Best performing tests, you need to enable Split test campaigns in the Settings > Cron settings screen.
    • Click a test’s Edit link to make changes to the test. See Creating and Editing Split Tests for information on the screen that displays.
    • Click a test’s Copy link to create a new test with the same properties as the original. The new test displays in the list with Copy of prepended to its name.
    • Click a test’s Delete link to remove the test from the email software. Deleted tests cannot be retrieved. Note: When you delete a test, its associated email campaigns are not deleted.

Performing a Split Test

To send a split test to members of your contact lists:

  • Click Manage > Emails > Email campaigns in the navigation bar, then click View split tests. Click the relevant split test’s Send link. Alternatively, while creating or editing a split test, click its Save and send button (see Creating and Editing Split Tests).
  • The first screen that displays allows you to specify to whom you want to send the test.

  • Select the contacts to whom you want to send the test in the I want to section:
    • To send the campaign to the entire list you select in the Contact list field, select Send the selected split test to all contacts in the selected list(s) below.
    • To send the campaign to the segment you select in the Segments field, select Send an email to all contacts in the selected segment(s) below. Note: Segments define a sub-section of a contact list. In order to select a segment here, you must first create segments.
  • Select the contact list(s) or segment(s) to which you want to send the email campaign. Use the search box below the list to search for contact lists or segments.
  • Click Next
  • The screen that displays allows you to configure your campaigns’ settings.

  • The areas shown in the screenshot above are discussed in more detail below:
    • Send From This Name = This displays the default “from” name for the contact list you selected. If you want to use a different name, enter the new name in this field.
    • Send From This Email Address = This displays the default “from” email account for the contact list you selected. If you want to use a different email account, enter the new address in this field.
    • Send Reply Emails To = This displays the default reply email address for this contact list. When a contact replies to your email campaign, this is the address to which the reply is sent. If you want to use a different address, enter the new address in this field.
    • Send Bounced Emails To = This displays the default bounced email address for this contact list. This is the address to which undeliverable emails are sent. If you want to use a different address for handling bounced emails, enter the new address in this field.
    • Send Your Split Test NowSelect this if you want to send the split test campaign immediately. Deselect this to schedule a date and time to send the email. Select the date, month and year in the first three lists, and select the hour, minute and AM/PM in the last three lists.
    • Notify Owner About Sending = Select this if you want to send an email to the list owner when a scheduled campaign starts sending, and another when it finishes sending.
    • Send Your Email as Multipart?Select this to send both HTML and text elements of a multipart email campaign to recipients. This allows a recipient’s email client to choose the correct display method. Note: Use this option if you are unsure. Also, use this option if you don’t provide your recipients with the ability to choose HTML or text emails.
    • Track Open Rates for these EmailsSelect this to have the email software track the proportion of HTML emails opened by their recipients. Note: This option cannot be deselected, as it is necessary for split testing campaigns to work.
    • Track Links Clicked in these EmailsSelect this to have the email software track the proportion of links in this split test that are clicked by the recipient. Note: This option cannot be deselected, as it is necessary for split testing campaigns to work.
    • Embed Images as Attachments = Select this to embed the images you send in HTML emails, rather than linking to them. This allows recipients to view the email campaign offline, but can increase the email size significantly.
  • Click Next. In the screen that displays, review your split test.

  • Click Schedule My Split Test Campaign when you are ready to send your test.
  • Your Split Test begins to send:

Do Not Close Window

As stated in the warning that displayed previously, do NOT close this window, otherwise your test will not be sent correctly. The browser window closes automatically when the split test has been completely sent.

  • Congratulations, your Split Test has been sent!


Viewing Split Test Statistics

To view statistics about the split tests you have performed, including the winner of each test, click Manage > Emails > Statistics in the navigation bar, then click Split Test Statistics. Note: This screen displays information about both manual and automatic split tests. This screen shows an overview of your split tests, and the winner of each (if the test has finished sending, and tracking information has been returned). You can:

  • View a split test’s detailed statistics. These include information about the winning email campaign (based on the criteria you selected), the number of opened emails, links clicked, bounced emails, and unsubscribed contacts.
  • Export a split test’s statistics to a CSV (Comma-Separated Value) file.
  • Print a split test’s statistics.
  • Delete a split test’s statistics from the email software. The statistics cannot be retrieved.

  • The following areas of the screenshot above are explained in more detail below:
    • Choose Action = Select one or more split tests’ checkboxes, select an option from the list and click Go to perform the selected action.
    • CheckboxesSelect a box belonging to a split test before choosing an option from the Choose an action list. Select the box at the top of the column to select all split tests.
    • Split Test Name = This displays the name of the split test.
    • Split Test Type = This displays the type of test:
      • Split test displays if you selected Find a winning email and show me the results when you created the test. The campaigns are sent to the whole contact list in equal proportion, and the winner is displayed to you.
      • Best performing displays if you selected Find a winning email then send it to my list automatically when you created the test. The campaigns are sent to a small part of your contact list, and the winner is automatically sent to the rest of the list.
    • Sent To = This displays the name of the contact list, contact lists, or contact list segments to which this test was sent.
    • Emails Sent = This displays the name of each email campaign that is part of this split test.
    • WinnerThis displays the name of the winning email campaign (based on the criterion you selected when you set up the test). If no tracking information has been received, or all campaigns are performing equally, None displays. If the test has not yet finished sending, Still sending displays. Note: Place the mouse pointer over the winner’s name to see more information.
    • Finished Sending = This displays the date and time at which the split test completed sending the test emails.
    • Action = The following options are available to take action on the Split Test:
      • Click View to see detailed statistics about the split test. See Viewing Detailed Split Test Statistics.
      • Click Export to save a split test’s statistics as a CSV (Comma-Separated Value) file. 
      • Click Print to select the split test’s statistics you want to print, view a preview of the printed document, and send the print job to your printer. 
      • Click Delete to remove a split test’s statistics from the email software. The statistics cannot be retrieved.

Viewing Detailed Split Test Statistics

To view detailed statistics about a split test, click Manage > Emails > Statistics in the navigation bar, then click Split Test Statistics. Click a test’s View link. The Detailed split test statistics screen displays at the Statistics snapshot tab:

The following table describes the tabs in the screenshot above:

  • Statistics SnapshotUse this tab to see general information about the split test, including the best-performing email campaign, as well as:
    • The number of opened emails of each type.
    • The number of clicks in each email (the click-through rate).
    • The number of bounced emails of each type.
    • The number of contacts unsubscribing from each email.
  • Open Statistics = Use this tab to see information about the number and percentage of unique email opens for each email campaign in this test.
  • Link Statistics = Use this tab to see information about the number and percentage of links in emails of each type in this test that were clicked by their recipients.
  • Bounce Statistics = Use this tab to see information about the number and percentage of undeliverable emails of each type in this test.
  • Unsubscribe Statistics = Use this tab to see information about the number of contacts who have unsubscribed using a link in emails of each type in this test.

Creating and Editing Split Tests

To create a new split test, click Manage > Emails > Email Campaigns in the navigation bar, then View Split Tests. Click the Create a Split Test button. To edit an existing split test, click Email campaigns in the navigation bar, then View split tests. Then click the relevant test’s Edit link. Note: This screen allows you to set up your test; you can choose the people to whom you want to send it later on.

  • Each area of the screenshot above is explained in detail below:
    • Give Your Split Test a Name = Enter a name for this split test. The name only displays in the configuration interface, and not to your contacts. 
    • Choose Which Emails to SendSelect the email campaigns that you want to include in this test. Note: You can select as many email campaigns as you want.
    • Preview Selected = Click this to see all the email campaigns you selected in the Choose which emails to send list. Each email campaign opens in a new browser window.
    • Choose a Winner Based onUse this field to specify how the winning email campaign is chosen. Note: When you select this option, the email software counts the total number clicks on all links in each email campaign, not any one link in particular.
      • Select Number of people who open the email to choose the email campaign that is opened most often.
      • Select Number of people who click a link in the email to choose the email in which a link is clicked most often (the one with the highest click-through rate).
    • The Kind of Test to RunUse this section to define whether to perform a manual test or an automatic test, and to configure how the test should proceed: 
      • Select Find a winning email and show me the results if you want to perform a manual test. When you select this option, the email software splits the contact list (or lists) into equal parts and sends each email campaign to one part. You can then view each email campaigns performance in the Statistics > Split test statistics screens. For example, if you select two email campaigns and have 1000 people in your contact list, each email campaign is sent to 500 people. Note: If you choose this option, you may want to first set up a new contact list for test subjects, rather than sending it to your complete contact list. If so, see Creating and Editing Contact Lists.
      • Select Find a winning email then send it to my list automatically if you want to perform an automatic test. When you select this option, you can specify the percentage of your contact list’s contacts that are test subjects. The email software performs the test, then waits the amount of time you specify before declaring a winning email campaign (based on the criteria you select). The email software then sends the winning email campaign to the remainder of your contact list’s contacts (those not in the test group). In the Emails will be sent in equal groups to (...) % of your list field, enter the percentage of your contact list’s contacts that should be in the test group. The default is 10%. In the The best performing email will then be sent to the rest of your list (...) later field, select the number of Hours or Days after which the email software decides the winning email. Note: The minimum test time is two hours, and the maximum is thirty days. For example, if you have a contact list of 5000 people, select five email campaigns, specify a test group of ten percent and a test time of one day, each email is sent to 100 people, and the winning email is sent to the remaining 4,500 people the next day.
    • Save And Send = Click this to save your changes to the test, and send it immediately. 
    • Save And Exit = Click this to save your changes to the test and return to the View split tests screen.
    • Cancel = Click this to return to the View split tests screen without saving your changes. 


  • No labels