Results for:

Which subject line converted best?



WhichTestWon's Analysis: (Click back to see versions A & B)

– Case Study –


Opportunity:

After conducting an initial email marketing campaign with lackluster results, Virgin Holidays, the online travel site, started using Phrasee, the artificial intelligence (AI) language software to create winning email campaigns.

Phrasee’s software attempts to take the guesswork out of creating copy by generating different subject lines that best resonate with audience sentiment, emotion, and language to increase email open rates increase.


Test Details:

For this updated email campaign, five different subject lines were generated through the Phrasee platform.

The subject lines were:

Version A: The Virgin Holidays Sale – hot offers. . .cool prices!

Version B: Where shall we go today? Just for you

Version C: Open me for holidays worth getting excited about

Version D: Florida, the Caribbean and the Middle East are waiting for you. . . Surprising holidays

Version E: The world is waiting to see where you’ll visit next! Unwind and browse holidays


Goal

To test which subject line worked best at increasing open rates, the five subject lines were sent out to a database of 299,294 existing Virgin Holidays email subscribers.

The test ran for 14 hours. Results were tracked through Zeta Global’s big data and analytics platform.


Hypothesis:

The testing team suspected the subject line with the most emotional and appealing language would work best to increase email open rates. But, the jury was out on which subject line that would be.


Results:

Winner: Version A – the subject line stating, “The Virgin Holiday Sale – hot offers. . .cool prices!” far outperformed all other variants.

Compared to the losing variant, which stated, “Where shall we go today? Just for you,” the winning subject line achieved a 6.4% higher open rate.

While revenue can’t be publicly released, the winning subject line brought Virgin Holidays a 10,000% increase in ROI!


Analysis:

The winning variant, “The Virgin Holiday Sale – hot offers. . .cool prices!” far outperformed all other variants. It especially outperformed the losing variant, which stated, “Where shall we go today? Just for you”.

Why?

The word “sale” likely generated interest, especially when followed by the phrase, “cool prices.”

The cost and the benefit were clearly stated to the reader. It was apparent that there was a sale and the prices to book a vacation were low.

In comparison, with the losing subject line, the text is somewhat unclear. The phrase, “Where shall WE go today?” doesn’t address the reader. It talks about “we” — not “me”.

The next phrase, “just for you,” uses the word “you” in attempt to personalize the text, but leaves the reader unclear. The use of the personal pronouns “we” and “you” in the same context may have created some confusion.

As well, the reader wasn’t told what they’re getting in the offer. There’s no closure. The reader may have been left thinking: just for you. . . what?

The what question is never answered, so uninterested readers liked moved on, scanning their inbox — rather than clicking to find out what the offer was. The subject line doesn’t indicate any benefit, or give any reasons why the user should click.

These results show clear language that describes immediate benefit to the reader may convert best.


Actionable Takeaways:

  1. Always split test your subject lines. The difference between a bad subject line and a good one can be huge. By testing, you’re limiting the potential damage of a bad subject line and increasing the potential revenue gains of a good one.
  2. Don’t use personal bias to choose your subject lines. Consider using artificial intelligence to do the hard work for you. Machine may beat man when it comes to short-form marketing campaigns and copy.
  3. Communicate benefit to your reader. The word “you” is often a powerful way to draw-in your reader. However, it’s effects are minimized when using the pronoun “we” in the same context. To be most powerful, clearly address your audience and tell them exactly what they’ll get from following-up on your offer.

Tell Us Your Thoughts:

Why do you think the winning subject line far outperformed all others?

Let us know by tweeting @BEHAVEdotorg.

Got a great test you’d like to see featured on the next Test of the Week? Send it on in!

Winning Version

A

Reader Guesses:

Which Test Won?

  • Version B
  • Version A
Loading ... Loading ...