Big data in retail: Navigating the challenges ahead Date:   Wednesday , October 08, 2014 Retail industry is witnessing rapid change as more and more players embrace online channels to widen their customer base and drive revenue. Working across multiple channels, online and offline retailers are looking to provide customers with individualized offers, delivered anytime, anywhere and on any device. Recent reports suggest mega trends like big data, social media, mobility, and analytics are significantly shaping the retail landscape. Retailers deal with a highly informed breed of digital consumers who leverage technology for affordable offerings, which are easy to order and delivered at the place and time that the customer finds most convenient. For instance, some retailers allow customers to place orders simply by scanning the quick response (QR) codes of the merchandise they need on a smartphone. The goods are delivered a short while after the consumers get home from work, not earlier, not later. Convenience isn\'t enough; new-age consumers look for personalized marketing that is strictly tailored to their need. Retailers must build the big picture of their customers, containing details of their buying history, age, marital status, salary, lifestyle, habits, and more before they can deliver on the personalization mandate. This is where big data comes in handy by providing a massive pool of granular data on customers from across touch points in the retail organization, which can help the retailer gain a competitive advantage in the market. While that sounds pretty cool on the surface, there are hidden problems in accessing big data. Traditionally, the major part of the data retailers use is structured data, which is easily captured, queried and analyzed. All that is changing with the high-volume of largely unstructured data flooding the organization at an unprecedented speed, that too in every conceivable format – email, blog post, audio, video, phone calls and social media postings. This is popularly described as \"big data.\" The volume of this data is set to grow 800 percent over next 5 years and 80 percent of this will reside as unstructured data, says a recent report by Gartner. Processing this assortment of data is going to require considerable time and effort. Big data has the potential to help retailers get inside knowledge of customers, including their buying trends, location, age as well as implement targeted marketing campaigns. Targeting niche groups of like-minded individuals is the new rage. Last year, as part of a micro targeting initiative, a multinational grocery sent a £5 discount offer to women aged 25 to 54 living in the catchment areas of its stores. Nearly 40,000 women clicked on a single day to redeem the coupon from the store. In another campaign a few months later, the top-end retailer sent mobile messages to people within the proximity of its stores, gently reminding them to order their turkey early enough for Christmas. Apart from improving operations and overall merchandising, tapping into rich stores of big data enables retailers to present customers with real-time location-based offerings like in the example above, which are also highly relevant. While big data is a blessing, there are challenges in getting the most value out of big data pileups within retail organizations. First of all, for the most part, the data is all over the place – in spreadsheets, databases, and apps, and, of course, between the ears (of people). Effective metadata management can help in cataloging this flood of data and making it more understandable and accessible for business users. Being able to analyze and act upon this tidal wave of data at almost the same speed as it is collected has become a necessity, not a nicety, at big retail chains. With a traditional extract-transform-load (ETL) approach to data usage, it can take hours or even days to process the data stream and gain the accurate and deep understanding required to do personalized marketing in real time. It will also help rid the retail organization of operational inefficiencies. To come to grips with the gigantic amount of data as well as support current and future needs, any technology architecture needs to be extensible and scalable. Alongside, the architecture must be cost-friendly. Open-source data processing platforms like Hadoop marry high scalability with low cost. With Hadoop, the cost of processing data is one-third lower; so much so that farsighted retail outfits are trailing out different use cases of big data for retail (such as personalization, energy and maintenance analytics and for operational efficiency) within the Hadoop platform. We at Tesco have created a framework using Hadoop and other technologies for unlocking the power of social media data and this framework can take in any category, product, promotion and/or service to provide deeper& hidden insights using unstructured data (customer sentiments). This framework can also stitch the structured data with unstructured data for it to get the reality validated and for more sophisticated analysis. At Tesco, we use video analytics in our stores for assessing completeness of fresh bins in the stores. Our continuous replenishment algorithms are the best in class by feeding various parameters (both structured and unstructured) to forecast the orders, sales and promotions. Consolidating big data at the organizational level rather than at the internal business unit level is a pain point for many retail businesses. At Tesco, given the number of business units & processes we have, it is a huge task to consolidate the data and especially big data. However, we have started to proliferate the framework that we have created within the organization and also aiming to have leadership level information governance to help us get there. Talent is the not-so-secret weapon in retail\'s big-data-based marketing offensive; a lot depends on how players attract resources, teach additional skills, and retain them. It\'s a tight market, no doubt, and talent hunt, especially for the \"cream of the cream,\" is often in stealth mode. Certainly, there are thoughtful steps retailers can take to get ahead of the talent crunch. Like maintaining a somewhat deeper resource pool than is normally required to take care of shortages. Rather than hiring a few superhero employees with a broad brush approach to analytics issues, often lacking in detail, it pays in the long run to build a strong team with deep-dive capabilities, which comprises experts in statistics, consumer psychology, marketing and visual merchandising, campaign management, data mining, predictive analytics, and so on.