For instant results to subject line tests, use Touchstone, our new open rate prediction and subject line optimization tool. Access Touchstone at subjectlinegold.com.
In this Q&A, Dela Quist shows that the A/B Split email test is not the only way – or always the best way – to assess the effectiveness of your campaigns…
I need to do an email test – tell me a bit about A/B split testing.
The A/B split email test is the classic way email marketers test the performance of their new email activity. It looks at 2 different variables of email response, typically subject line, content or creative. Most often the test will look at an email’s performance for just a short time period, to see which subject line / creative treatment etc achieves the best open and click rates. But at Alchemy Worx, we’ve seen that isn’t always the best way to assess which approach will be most successful overall.
So what’s the issue with A/B split testing?
The problem is that the short time period distorts the result. Again and again, our research shows that the test cell in the lead after a few hours rarely ends up as the most best performing cell after a few days, especially when sales are taken into account. This is because the people who interact with an email first are a very specific subset of your customer base – very different to (and commercially less significant than) the sort of people who will interact with an email over an extended period. Of course, there will be some people who will be actively waiting for your email and will open immediately – but this is not typical behaviour. The usual behaviour is not to open the email at all, meaning there are no guarantees that the people who act first are going to act the same way through the length of the campaign. Furthermore, although A/B split testing may uncover the best “average” approach, the reality is that there is no such thing as an average subscriber. When analysing your data, you may find that some things work best for some users, whilst another approach works better for others.
So how else can I test the impact of different factors?
It’s not as sexy, perhaps, but the most effective way to assess a new email campaign’s performance is to crunch historical data. After all, a test – however sophisticated – is only a prediction. Analysing historical data, on the other hand, is to look at the factual record of what actually DID happen – and that should be the basis of analysis for future testing. So use your historical data to check what you have already tested. You will most likely have sent mailings at different times of the day, different days of the week etc. Use that knowledge to fine-tune your future test plan, and only test what you need to test. Analyse your individual subscriber’s behaviour to group subscribers into segments of people who respond in certain ways. For example, it may be worthwhile developing an additional program for your active subscribers.
http://www.youtube.com/watch?v=CYKfItiOrOk
Top Tips
Some tried-and-trusted methods to reinvigorate interest in your email activity:
• Make the most of what you already know about your subscribers by analysing your historical data.
• Consider grouping subscribers into segments according to response.
• If you’re A/B split testing, make sure to leave enough time to get truly representative results.
Last updated: Apr 20, 2015 admin
