A/B Testing Email Marketing

Remember the golden rules of direct marketing:

“If it ain’t broke…fix it!” so it will work even better in the future.
“Test the BIG things” i.e. the big 6 marketing variables

Email marketing continues to be a powerful tool for businesses to engage with their audience, drive conversions, and build brand loyalty. However, not all email campaigns are created equal. What works for one audience may not resonate with another. That’s where A/B testing comes in. A/B testing allows marketers to experiment and optimise their email campaigns by comparing two or more variants and using data-driven insights to make informed decisions. In this article, we’ll explore the concept of A/B testing in email marketing and how it can help you achieve better results.

When A/B testing email marketing, use these six marketing variables:

• Product / service
• Target Audience (lists)
• Offer (price, terms)
• Timing
• Format
• Creative

A/B Testing Email Marketing Best Practices

1. Define clear goals: Before starting an A/B test, clearly define your goals and what you want to achieve. Whether it’s increasing open rates, click-through rates, or conversions, having specific objectives will help you measure the success of your tests accurately.
2. Test a significant sample size: Ensure that your sample size is large enough to produce statistically significant results. Testing with a small sample size may lead to inaccurate conclusions. Utilise statistical tools or calculators to determine the required sample size for your tests. Alternatively, use our formulas below.
3. Monitor and analyze metrics: Track and analyze key metrics such as open rates, click-through rates, conversions, and unsubscribe rates for each variant. Pay attention to statistically significant differences and trends to make data-driven decisions.
4. Implement changes based on results: Based on the insights gathered from your A/B tests, implement the changes that yielded better results. Continuously refine and optimise your email campaigns to deliver improved performance over time
5. Test only one variable at a time: To isolate the impact of each element, focus on testing one element at a time. Whether it’s the subject line, email design, CTA, or sender name, changing multiple elements simultaneously can make it difficult to determine which change influenced the results.With that in mind, and if you want to do some testing quickly then the most important variable is the “audience” – the list. When compiling a list it is necessary to test to minimise the financial risk of buying all the available names. Below is a simple sampling model that we know will help you.There is no “right sample size” or right and wrong way of sampling. Sampling can be as simple as picking records at random until you have a reasonable number. However the most successful sampling technique in our experience relies on a degree of confidence and accuracy. It works as follows:

A/B testing direct email marketing sample size calculation

Sample Size = Z² x R x (100 – R)
E²

Where:

Z = degree of confidence
R = estimated % response rate
E = limit of error

The degree of confidence (Z) is a commonly used constant. The most used constants are
1.281 for an 80% confidence level
1.645 for a 90% confidence level
1.960 for a 95% confidence level
2.575 for a 99% confidence level

So if you estimate a 2% response rate and want a 95% confidence that the roll out will be within +/- 0.3% (i.e. between 1.7% and 2.3%) then the sample size formula would look like this:

SAMPLE SIZE = 1.96² x 2 x (100-2)
0.3²
=8,366

In other words, if you take a sample of 8,366 and the response rate is 2% then you can be 95% confident that the rollout response from the total audience will be between 1.7% and 2.3% subject to consistent conditions.

In practice, rollout conditions are rarely consistent. However after a test has been done and actual % response rates are known, the formula can be converted to adjust the confidence intervals.

Please send your enquiries via the contact us form.