“How do I improve my Subject Lines?” is arguably the most frequently asked email question of all time. The obvious answer is A/B testing.
But when we looked at data from Subject Line Pro, a virtual test platform for Subject Lines, the answer wasn’t that simple. When it comes to testing effectively, A/B is not enough.
Subject Line Performance Curve – No test
This bell curve taken from the campaign data held with Subject Line Pro shows the distribution of open rates for every Subject Line in the database. All open rates are computed relative to the sender’s average open rate, so an open rate of +20% here means that it performed 20% above average.
How Many Subject Lines Should You Test?
Let’s start with how many Subject Lines you should test to ensure you get a significant lift. The term A/B test would imply that two is ok, but the data would suggest otherwise. The data shows us that any given Subject Line has roughly 1 chance out of 3 of performing at least 20% above average. In other words, to be sure to get a 20% lift in your open rate, you should test at least 3 Subject Lines on every single campaign.
We can also see what happens if you don’t test, or test too little. If you use only one subject line, there is a 1/3chance that it will under perform the average by at least 20% and a 1/10 chance that it will perform 42% below average. So, if you select only two Subject Lines in your test, then each has a 1/3 chance of underperforming by >20%. In other words, there is a 1/9 chance that both will be 20% or more below average. Subject Line “A” is bad, “B” is worse – bad beats worse, but they are both rubbish!
The more Subject Lines you test the greater the chance that at least one of them will be one of those great Subject Lines to the right of this chart and reduce the chances that you’ll send one of the less effective Subject Lines on the left.
As you will see from the next 3 charts every incremental Subject Line you test improves your chances of getting a lift.
Predicted Performance from 2 Subject Line(AB) Test
The same is true whether you decide to optimize for Click Rate or CTO rate as this handy chart taken from the data demonstrates.
The Answer is C – More is Better
If you test 3 Subject Lines the chances are you can expect a 21% improvement in your open rate, 42% in click rate, and 25% improvement in your CTO. The need to test is something that even the AI language generating services such as Persado or Phrasee cannot sidestep. Both services promise/deliver an uplift – but require you to test between 10 and 20 Subject Lines. So, it doesn’t matter who or what writes your subject lines: If you want to guarantee a lift, you must test more.
