Jhunjhunwala on The Untapped Value of Predictive Technology and Big Data
Data is the new oil of the digital economy of the 21st century. Just like oil in the last couple of centuries, it has immense untapped value. But just like oil, it also has no significant value in its crude form; it needs to be refined, broken down and converted into meaningful insights. With the data storage & processing costs decreasing rapidly and the proliferation of IoT-enabled devices producing huge amounts of data, most of the data sets today are very large & complex to deal with, more commonly referred to as Big Data.
Big Data has permeated our everyday lives in unimaginable ways. On one hand, it’s being used in the agricultural sector to consolidate data from weather, satellite & IoT devices to provide insights to farmers about new-age crop practices; while on the other hand, it’s the new game-changer in elections, contributing both in the recent Trump win in US and BJP’s 2014 victory in India. It has far-reaching applications in fields as varying as science, healthcare, finance, advertising, education, sports & retail. Taking decisions based on data is the new holy grail, but as we all know, it is easier said than done.
Big Data is typically characterized by the widely used Gartner’s model of 3 Vs –
1. Volume – As of 2016, roughly 2.5 exabytes (1 billion GBs) of data is produced every day.
2. Velocity- Data velocity can range from real-time, to near real-time to periodic batch dumps.
3. Variety – Data diversity, ranging from text, images, audio/video, to complete unstructured bits.
As you would guess, the challenges of handling Big Data increase by expansion on all of the 3 factors, and not just volume alone. And that is why Big Data is so relevant today! It can help organizations & people to combine dissimilar data sets, harness them and drive insights to identify new opportunities. Be it venturing into new markets, driving cost reduction initiatives, or launching of new products – all of them are backed by crunching Big Data.
Brands looking to invest in Big Data technologies would do it for various challenges being faced by them – including but not limited to – data capture, data storage, data processing, data search/transfer, data querying and data visualization.
One of the best and most resonating examples of Big Data being applied in the consumer tech industry is the ‘Surge Pricing’ Model of cab aggregators like Ola or Uber. These companies employ sophisticated predictive modeling techniques to preempt rise in demand at certain times at certain locations. They continually analyze regular traffic trends of weekday office goers or weekend pub hoppers, spurt in demand due to weather events like rains, or some event like a sports match or a music concert. All of this then gets matched to the supply & demand patterns being seen in real time to decide whether surge pricing would be applied or not, and if yes, by what factor. This happens in real time in thousands of location clusters just in a single city like Bangalore, multiply that by hundreds of cities and you have a real Big Data Application at hand.
In the ad tech industry, one of the most widespread applications of using Big Data and Machine Learning is for fraud detection. Companies are using advanced real-time techniques to detect bot traffic, install hijacking, click flooding, etc. which results in savings of billions of dollars annually.
India too has seen tremendous progress in Big Data analytics and applications, both by the public sector and the private companies alike. It is already in the top 10 Big Data markets globally, while NASSCOM predicts that it would be among the world’s top 3 in the next few years. It is poised to see a double-digit growth of more than 25 percent in the coming years, making it a USD 16 billion industry by 2025. There are more than 650 companies in India in this sector, many of them young start-ups at the forefront of innovation. These numbers alone indicate the importance of this rising new kid on the block.
Big Data and analytics would continue to remain the poster boy of the India media in the coming years and for valid reasons that is. But it is important to understand that getting the maximum potential out of any data is much more about management of it than the technology used. You may employ the best tools & team to generate important insights, but it will die its death if the relevant stakeholders don’t understand its implications. Big Data will become ubiquitous even more so than now once organizations start converting their data into insights then putting those into action!