The Smart Techie was renamed Siliconindia India Edition starting Feb 2012 to continue the nearly two decade track record of excellence of our US edition.

BIG DATA: Making Sense in Real Time

V R Ferose
Friday, September 3, 2010
V R Ferose
What if we found a ‘needle’ in a hay stack that is ten times the size of planet Jupiter, in less than a second? Think about it. Time is money. Information from various sources is increasing at a mind-boggling rate begging to be analyzed correctly and made sense of. Traditional way of running business is passé. To stay ahead of competition, you require tools that will handle these large volumes of data in real-time helping you take informed business decisions. You need to know the status as-is as of ‘now’. Not as of July 31, 2010.

Big Data. Slice n’ Dice!

‘Big Data’ is the buzz word from the high performance computing niche of the IT market. Big Data is suddenly the focus of most presentation from suppliers of processing virtualization and storage virtualization software. But what is Big Data?
According to a recent report from McKinsey, data are flooding in at rates never seen before—doubling every 18 months— this is all because of greater access to customer data from public, proprietary, and purchased sources, as well as new information gathered from Web communities and newly deployed smart assets. These trends are broadly known as ‘Big Data’. But if I were to explain this more simplistically, then any data is big when one has to really sit and take decisions on how to organize it, manage it and most importantly analyze it to get some desired results. In other words, the phrase refers to the tools, processes and procedures allowing any organization to create, manipulate, and manage very large data of the size of many gigabytes to terabyte, petabyte or even larger collections of data.

Data accumulated and not managed can pose a great problem. There are many reasons for data to grow. From government regulation needs where in data is stored for future reference to data needed for critical research and analysis for example in sectors like health/pharma, energy or weather environment. While on the other hand big data can really provide a huge competitive edge for those companies who can manage it, analyze it and use it for optimizing operations or any other beneficial task. There are examples cited by big companies like Google who clearly demonstrate how one can have an edge over its competitors simply by analyzing the data that gives you information on the ground you are operating.

Today Big Data management stands out as one point challenge for IT companies and increasingly the solution is moving from providing hardware to more manageable software solutions. But before actually talking about the solutions, let’s take a step back and understand where all this extra data is coming from. Well, the sources are many; the web itself is one source which is giving out a lot of valuable and critical information that needs to be analyzed and therefore stored as data. Facebook, in just over two short years, has quintupled in size to a network that touches more than 500 million users. Twitter, since its creation in 2006 has grown to over 100 million users worldwide and is now attracting 190 million visitors per month and generating 65 million tweets a day. Computing and sensor networks itself is becoming more dynamic and needs extra data. People are using more than before data to predict behavioral pattern and predict performances. Obviously the more analytical we are becoming in our approach, the more data we require to analyze.

Share on Twitter
Share on LinkedIn
Share on facebook