Results for:

Is it best to feature your app installer banner? Or leave it off?

WhichTestWon's Analysis: (Click back to see versions A & B)

– Case Study –

Taking a digital-first approach to the customer journey, financial institution Royal Bank of Scotland (RBS) ran this ‘app-ealing’ test to understand whether adding an app installer banner increased or decreased app installations. The study was conducted in-house, by RBS’ testing team, for one of its subsidiary brands, NatWest.

Splitting traffic 50/50, a total of 35,403 banking customers were directed to the page either with or without the app installation box. App installs were tracked across both versions for two weeks, using Adobe Target.

Although customers could still perform all their banking needs without the app, the team wanted to measure demand for the banking app.

They suspected presenting a banner promoting the app would increase installs since it would clearly communicate there was an easy-to-install app available for customers.

Winner: Version A – As expected, the version with the app installer drove the highest impact. Compared to Version B, without the banner, Version A resulted in a 58% increase in mobile app registrations.


Test Evidence
As marketers, we tend to think that, because we know about a new feature our dev team just launched, everyone else does too. In reality, however, most customers don’t know much about your company’s products or features, and why would they? Your customers are not likely to be on the lookout for product or website updates. That said, if you tell them about it, bringing your new feature to their attention, chances are they’ll take a look. As the old saying goes, “You can lead a horse to water, but you can’t make it drink.” Likewise, while you can’t force your customers to engage with your products, you can make those products readily and easily accessible, thereby giving them the clear opportunity to try it.

Prioritizing your testing pipeline
Okay, great — you’ve promoted your new app. People are aware of it and, hopefully, installing it. You’re convinced the app will improve conversion rate, but it’s still in beta. And you’re awaiting approval to develop and launch the finalized version.

Even more frustrating is the fact that you and your colleagues are not on the same page. Management keeps telling you the update is not a priority. The development team is overloaded, so unless it can dramatically improve mobile conversion rates, it will have to wait.

What do you do?

Why not rollout the beta version of the app through an A/B test? It’s a low-risk way to determine customer appetite and measure the conversion rate impact. Once you’ve run the test, you’ll have definitive results to prove demand. You can then go back to the management and show, with quantitative evidence, the need to prioritize your app in the dev workflow.

The point: while we tend to think testing is only useful for optimizations, this week’s example illustrates that A/B testing can, in fact, fulfill another purpose. It can help determine whether a specific app or feature is worth the time and development investment required. In the case of RBS, it proved true — validating their hypothesis and confirming that they should put resources behind the app’s development. In other scenarios, it may prove not worth the time, but either way, you’ve just gained additional insights into the needs of your traffic.

Actionable Takeaways:

Communicate with your traffic. If there’s something you want your traffic to do, don’t expect they’ll figure it out for themselves. Tell them what they should do and why it’s important.

Take testing beyond tests. With every test you run, you’re learning a little more about your traffic. Be sure to consider how you can use your insights to inform your overall business strategy.

What has worked for you when telling customers about a new feature on your site? And, what kind of testing have you done to get your idea prioritized in the development stack?

Let us know by tweeting @BEHAVEdotorg.

Got a great test you’d like to see featured on the next Test of the Week? Send it in here!

Winning Version


Reader Guesses:

Which Test Won?

  • Version A
  • Version B
Loading ... Loading ...