We recently looked at how using AI to create subject lines could increase lift. We found that using an AI writing tool to rapidly create ten or more subject lines for testing could significantly increase email opens and clicks. However, the key was using a testing platform that could test the effectiveness of subject lines at volume and speed.
As a follow-up, we decided to see just how fast we could create 50 subject lines using ChatGPT and then test them using our virtual testing platform, Subject Line Pro.
Why ChatGPT?
There are several AI writing tools that offer benefits such as time saving, data-driven results, objectivity, personalization, constant improvement through machine learning, and cost effectiveness. ChatGPT offers all these advantages but is gathering a huge amount of interest thanks to its ability use natural language prompts to generate truly human-like responses.
In addition, while most other AI writing tools require a subscription,ChatGPT is currently free to use. Its combination of features, human-like responses, and low barrier to entry make ChatGPT a great option for marketers interested in giving AI a try.
Why Subject Line Pro?
To take advantage of the speed of AI writing tools, you need a testing platform that can keep pace. Traditional A/B testing is limited by the number of subject lines that can be tested simultaneously and the speed at which results can delivered – typically hours or longer. That’s because traditional A/B tests are based on responses from live audience segments. This form of testing can cause a bottle neck that negates the efficiencies associated with using AI to create subject lines. It can also be costly in terms of wasted subscribers.
Subject Line Pro is a virtual testing platform that enables you to enter ten subject lines at a time for testing against a simulation of your own data or a database of more than 500 billion subject lines across different sectors. Its proprietary algorithm delivers results accurate to within 1 to 2% of actual clicks and opens in just seconds. Users can step-and-repeat the process to quickly aggregate the results of multiple tests. (Disclosure: Subject Line Pro is a product of Alchemy Worx.)
The Test
We conducted the test using a Macbook Pro running macOS Ventura and the latest version of Chrome (110.0.5481.100) on Wi-Fi network with 53.29 Mpbs download and 20.15 Mbps upload speeds. In other words, a system you might typically use in your home office.
We asked ChatGPT: “Write 50 email subject lines on the topic of why it is important to test the effectiveness of subject lines created by AI.” It took 90 seconds for ChatGPT to deliver 50 responses.
Armed with the subject lines, we entered them into the testing platform. Subject Line Pro is currently limited to entering 10 subject lines at a time. It took an average of 75 seconds to copy and paste 10 subject lines from ChatGPT into Subject Line Pro. The process was repeated five times to complete the entry of 50 subject lines in 6 minutes 15 seconds.
Subject Line Pro works by testing each group of 10 subject lines as soon as they are entered. Each subsequent group of ten subject lines is tested with the results being combined with all previous entries. The test times for each of the five groups was as follows:
It took just 11 seconds to get results for the first group of 10 subject lines. Each subsequent group added about 10 seconds to the result time. The total time Subject Line Pro took to deliver results for all 50 subject lines was 2 minutes 49 seconds.
The Results
The entire process from generating 50 subject lines with ChatGPT to getting test results from Subject Line Pro took a total of 9 minutes and 34 seconds.
Our test demonstrated that is possible to use an AI writing tool and a virtual subject line testing platform like Subject Line Pro to create and test 50 subject lines in less than 10 minutes. This represents a tremendous time saving over traditional copywriting and A/B testing that can take many hours or even days.
That’s Fast, But Are They Effective?
Based on the results from Subject Line Pro, the top ten subject lines generated by AI registered positive lift rates with the top five ranging between 34 to 35% better than average. We compared these results against a similar test we performed on 50 subject lines created by copywriters. In that test, the top ten subject lines also registered positive lift, but the results for the top 5 were significantly lower at 3 to 19%. Here are the results of the AI subject lines:
Bonus
As a bonus, we will be using the top result from our test of AI subject lines for an email promoting a blog on why it is important to test AI subject lines. We will follow-up with results of that email in a later post.
