A/B Testing

A/B testing is a method used to split test your email templates to determine if one variation performs better than another. Running a test is very simple, and can help you improve the conversion rates for your emails.

Most A/B tests will want to test only one metric, open rate or click through rate. Open rate is tested by changing the subject or from sender, whereas click through rate is generally tested by adjusting the body copy and calls to action.

Running an A/B Test

Creating a Draft A/B Test

To get started A/B testing an email, select “Run A/B Test” from the settings drop down on the template list page (the ‘Edit’ tab).

Run an A/B test

You’ll be taken to the template editor, but it will look slightly different. Firstly, you’ll see a ‘Draft A/B Test’ badge next to the template name. You’ll also see two tabs below the template name, ‘Original’ and ‘Alternative’. These tabs toggle between the two versions of your email that will be sent. You’ll probably only want to edit the alternative, but you might have been itching to make some changes to the original template – so you can make these changes now.

AB Test Editor

Note: At this point, the email being sent has not been changed. While in draft mode, an A/B Test is not active and sending. You can save the changes you make and it will not affect the live email until you start the A/B Test.

Starting your A/B Test

To actually start sending your A/B Test, you just need to select ‘Save & Start Test’. Now when your app requests to send the template, one of the versions will be chosen at random to send to the end user and we track the delivery statistics for each version separately.

While your A/B Test is running, a dashboard will show on both the editor and analytics pages for that email template. At first there won’t be much information available, since there is no data available for the test, however as your app starts to send emails you’ll start to see results show.

AB Test Editor

A brand new A/B Test

AB Test Editor

An A/B Test with results

Tip: Once you’ve started a test, you cannot make any edits to your templates since that would likely skew your test results.

Selecting a Winner

Once your A/B Test has been running for a while, you’ll be able to determine a winner. Sometimes this is a clear cut decision and other times it‘s not. If you’re not seeing a clear winner, you can either let your test run for a longer period of time and hope a winner becomes obvious, or you can finish your test and create a new one with a new version that you think may generate a better split result.

Once you’ve selected a winner the A/B Test is complete and from now on only the winner will be sent.

Selecting a winner

Selecting a winner