The success of several product changes (such as new features, user interface improvements, content and design concepts) at Yahoo! is measured scientifically, by rolling out the changes to a sample set of population instead of the entire user population, via experimentation a.k.a. bucket testing. This is done keeping various things in mind, like — to test product features, user interface design algorithms on a smaller scale before exposing it to larger audience, to test out ideas to optimize the tradeoff between monetization and user experience, forecast the impact of product changes to the overall page performance, gain insights into user behavior and more.
Ashish Vikram, VP- Engineering heads the User Data and Analytics (UDA) group at Yahoo! India R&D, that analyzes Yahoo!’s customer data to provide deep insights into user behavior, which is then leveraged to personalize digital content and enhance Yahoo!’s products. In a candid conversation he talks about the bucket testing process and its advantages.
Why is bucket testing important?
User data is really critical to us for we need to constantly analyze it, show the ROI to the advertisers on the website. The advertisers need the statistics to ensure that by placing ads on Yahoo!, they are able to reach the required demographics. The data is also used to improve the engagement on our website and bucket testing becomes very important for this.
When you have a site that is getting millions of page views a day, it is quite risky because if you make changes that cause your audience to go away, they may never comeback. So at Yahoo!, the way we make changes first introducing the new design to a small percentage of the users, say about five percent. The changed UI is run for a set period of time, varying from weeks to several months, depending up on the data required. The system automatically collects the data from the five percent bucket and with statistical analytics UDA group confirms if the new changes are working better or not and if so how much better.
Is bucket testing useful only for Internet based products?
Yes. Bucket testing works best for Internet based products because it is here that you have millions of users from which you can pick a small percentage to test on and significant results on the changes one wants to make. These tests can be anything from a slight change in UI to seeing traction for Ads within specific user demographics to user reaction to any specific change in terms of features or services introduced on a website.
For example, every Yahoo! page, especially the UI of the homepage has been derived based on experimentation. In fact, the home page layout differs across geographies, user gender and age group. All this is derived by analyzing the usage pattern of the Yahoo! user based on the frequently visited pages, search interests and more. Even the size of fonts, pictures, placement of widgets, and choice of background color is decided after doing the bucket test.
In case of online ads, one can decide what layout of the ad that works best. One of the examples of a successful experimentation is the Yahoo! search box. Through bucket testing we learnt that a wider and pronounced search box captivates the attention of the user and would increase their desire to search, hence there will be an increase in the click-through-rate of the search link. The team actually saw user engagement going up after implementing their learning from bucket testing.
One of the features we plan to roll out in the near future is allow advertisers to test different creatives and decide which one to finally publish, depending on the user traction.
What are some of the trends that you have identified in the behavioral pattern through the tests?
One of the things we have realized through the tests is "the bigger the better". If you put a bigger photograph with less white space around, users prefer it. Similarly, we have seen that while in India users are more drawn to brown as the background color, in European countries, green is the preferred color. This helps us to customize the websites accordingly.
How are other web players adopting this methodology?
I think only the bigger Internet players are using bucket testing and experimentation to plan their website as of now. This is because to do such experimentations, one needs to build a robust infrastructure and technology which is quiet expensive. This method of testing also requires a large user base which as of now is only available to big players like Amazon, Zynga, Facebook and more. For example if you are a young e-commerce portal, you will have a very small number of visitors and doing a test on five percent of the users means you are looking at only a few hundred visitors. Statistically, the results will be insignificant.
Hence, keeping in mind the cost limitations of small players, we are looking at providing bucket testing as a platform so that other publishers, portals and more can use it to design or improve their websites.
What are the other things that the UDA group is focusing on?
We are responsible for analyzing data, so experimentation is one of the things that we use data for, but we also use it to personalize the content that we deliver to people on the web. Today, we serve 14 million different versions of our home page every day to people across the world. The Yahoo! homepage delivered varies from user to user. The personalization is done based on country, region, location, gender, age, and most importantly on what some has clicked before.
Also the ads that one sees is personalized which again is based on the users gender, age and interests. The other thing for which we used data for is to keep the statistics of how many people is visiting each website, how much time they are spending there, what they are clicking on, and that is used by the executives to make some decisions.
(As told to Vimali Swamy)