This installment of We Test Everything highlights the importance of repeat tests to see if the results hold up across various seasons or audiences.

Alchemy Worx recently ran an A/B split test on behalf of foot-care-products merchant Dr. Scholl’s, pitting emails with pricing against emails without.

Dr. Scholl’s typical customers tend to be highly price conscious. The idea behind the test was to figure out if it was best to show pricing up front or to get them to click to the site first.

In this test, clicks were almost identical, but the email with pricing drove 3x the revenue of the email without. That’s a phenomenal result. 

So it would seem putting prices in the email was the way to go. Well, not so fast. The folks at Alchemy Worx questioned the results. While the audience size was substantial enough for statistical significance, the number of orders was a little bit lower than we thought was ideal. 

Alchemy Worx ran three repeat tests pitting pricing versus no pricing over three days to a total audience more than four times the size of the audience in the first test. Here’s one example:

In these tests, emails without prices consistently drove more clicks and an average of 18% more revenue across the three campaigns.

“This highlights the importance of repeat testing,” says Allan Levy, CEO of Alchemy Worx. “If we had just gone out after the first test and said: ‘Great, we got a phenomenal response. Let’s go with it’ and we hadn’t repeated the test with a larger audience, then we would have thought we should include the prices and we would have potentially been losing revenue.”

In the first test, while the results were lopsided in favor of including pricing, the actual number of sales was too low to inspire confidence in a rollout based on them.

“We said: ‘That first test was interesting, but let’s run it again with a greater audience so we can be statistically significant on the number of orders’ and we got a different result,” says Levy. “But the different result was pretty clear in that it showed we just didn’t have enough of an audience in the prior one to make a call.”

Apparently with Dr. Scholl’s customers, creating interest in the products before getting them to the site and giving them prices is more effective than presenting prices up front.

“It’s going to be different with every company, but this is apparently the formula that works for Dr. Scholl’s,” says Levy.

For now.

“We’ll probably retest this in the future to see if anything has changed around price sensitivity,” says Levy.

At Alchemy Worx, we test (and retest) everything. You should, too. For more information on how to partner with Alchemy Worx, contact us.