Discovering Hidden Segments

Discovering Hidden Segments

A/B and multivariate testing has finally come of age is now considered a mainstream digital marketing and website management practice. However, though testing has been implemented by many organizations, few organizations has embarked on the personalization aspects of website optimization. Most of them mistake personalization for user-customization and hence did not make full use of the power of personalization.

What A/B and multivariate testing can reveal

Most people think of A/B testing or multivariate testing as means to check for winning versions that lead to the most amount of conversion. However, testing to figure out the winners is only half the story. Discovering why they are winners or losers is something else.

Let me simplify the test result for example. During the test for an effective campaign page, you discover that creative A has a conversion rate of 10% versus creative B that has a conversion rate of 5%. At first glance, most analysts would say that it is obvious that A is the winner with 100% uplift and recommend that A, the winning creative, be used as the campaign page. Nothing is wrong here, or so it seems.

However, if you examine the data to see what it hides, things can get really interesting. You may discover something unusual if you start segmenting the data. Segmenting the conversions by sex, you may discover that for winning creative A, 80% are female. However, for the losing creative, female shoppers convert badly and 90% of all conversions are male. Thus, the right approach is not recommending creative A to be a permanent campaign page. The right recommendation would be a targeting strategy, creative A for female and creative B for male.

You may not have advance analytics and may not be able to tell the age or sex of the visitors. Does segmenting helps? The answer is still a resounding yes. You can use existing data in your analytics program to segment the users. Exhibit 1 below shows the segmentation of A/B testing result. Although A/B testing shows that the winning creative is B, segmentations tell us a different story. It appears that creative B is winning because of high number of users on mobile devices converting. However, for creative A, the mobile users aren’t really converting. Switching to creative B, the winning creative, may have some incremental uplift. But using a targeting strategy, you can have a massive uplift.


Exhibit 1: Winning creative B is performing better than creative A because of mobile users. A targeting strategy that target desktop users with creative A and mobile users with creative B would yield far better result than merely switching to creative B.

Where do I begin to look for segments?

In an ideal world, we would have data scientists who would crunch the numbers and identify many complex segments that we didn’t know exist (eg. Women age 20-30 who are from location X and use sports keywords to enter site through an iPhone etc.), you will gain quite a few mileage if you concentrate on segmenting base on age, sex, browser type, behavior (pages that they visit or referring sources). If you try segmenting them, you would probably find some good segments to do targeting after each A/B and multivariate tests unless the losing creative is inferior in all segments.

Targeting is key to optimization

To conclude, segmenting your data would yield a lot of results. You may discover that different segments behave differently to different creative and require a targeting strategy. If you dive deep enough, you may also discover complex segments that you would never have imagined.

If you are interested in more advance segmentation methods or are keen to do some serious data crunching, do contact us to learn more.