BIG DATA: Making Sense in Real Time

Author: V R Ferose
Managing Director, SAP Labs India
BIG DATA: Making Sense in Real Time -By-V R Ferose
What if we found a ‘needle’ in a hay stack that is ten times the size of planet Jupiter, in less than a second? Think about it. Time is money. Information from various sources is increasing at a mind-boggling rate begging to be analyzed correctly and made sense of. Traditional way of running business is passé. To stay ahead of competition, you require tools that will handle these large volumes of data in real-time helping you take informed business decisions. You need to know the status as-is as of ‘now’. Not as of July 31, 2010.

Big Data. Slice n’ Dice!

‘Big Data’ is the buzz word from the high performance computing niche of the IT market. Big Data is suddenly the focus of most presentation from suppliers of processing virtualization and storage virtualization software. But what is Big Data?
According to a recent report from McKinsey, data are flooding in at rates never seen before—doubling every 18 months— this is all because of greater access to customer data from public, proprietary, and purchased sources, as well as new information gathered from Web communities and newly deployed smart assets. These trends are broadly known as ‘Big Data’. But if I were to explain this more simplistically, then any data is big when one has to really sit and take decisions on how to organize it, manage it and most importantly analyze it to get some desired results. In other words, the phrase refers to the tools, processes and procedures allowing any organization to create, manipulate, and manage very large data of the size of many gigabytes to terabyte, petabyte or even larger collections of data.

Data accumulated and not managed can pose a great problem. There are many reasons for data to grow. From government regulation needs where in data is stored for future reference to data needed for critical research and analysis for example in sectors like health/pharma, energy or weather environment. While on the other hand big data can really provide a huge competitive edge for those companies who can manage it, analyze it and use it for optimizing operations or any other beneficial task. There are examples cited by big companies like Google who clearly demonstrate how one can have an edge over its competitors simply by analyzing the data that gives you information on the ground you are operating.

Today Big Data management stands out as one point challenge for IT companies and increasingly the solution is moving from providing hardware to more manageable software solutions. But before actually talking about the solutions, let’s take a step back and understand where all this extra data is coming from. Well, the sources are many; the web itself is one source which is giving out a lot of valuable and critical information that needs to be analyzed and therefore stored as data. Facebook, in just over two short years, has quintupled in size to a network that touches more than 500 million users. Twitter, since its creation in 2006 has grown to over 100 million users worldwide and is now attracting 190 million visitors per month and generating 65 million tweets a day. Computing and sensor networks itself is becoming more dynamic and needs extra data. People are using more than before data to predict behavioral pattern and predict performances. Obviously the more analytical we are becoming in our approach, the more data we require to analyze.

Terabyte Trends in Technology
There is an exponential increase in the raw computing power that is available at our disposal today. The last decade especially has seen a drastic increase in the processing power which hit a road block about seven years back when the clock speed touched 3 gigahertz and could not run any faster. This brought out the need for multi core processing which is not only faster but highly reliable too. As a result, today, you can find a 32 core processor with half terabyte of main memory and 2 terabytes of solid state disk at our disposal at a highly affordable price. This kind of computing power was unthinkable in the 90’s.
With this kind of computing power at our disposal, software product development companies are now looking at way to harness this power and rewrite software that can process data tens of thousands of times faster.
We at SAP Labs have been quick to take on this challenge and our engineers are already involved in rewriting codes that will revolutionize the way big data is analyzed and processed and all this in real time.

A disruption called ‘In-Memory’
As mentioned earlier, one of the highest priorities for organizations of any size and across any industry is managing and analyzing the soaring quantity of data, and harnessing that information to improve their business. The world’s major oil companies, major governments, educational institutions, pharma companies, banks, internet portals deal with millions of transactions daily and are struggling to analyze and use this data which normally takes weeks and months. SAP has always understood this and has addressed this challenge and is currently developing in-memory solutions which will allow our customers (to cite an example, large enterprises in the FMCG space who have data of more than 5 - 6 terabytes) to explore business data at the speed of thought.
In-memory computing is considered as one of the largest disruptions of the 21st century that will enable business users to instantaneously access, explore, model and analyze transactional, analytical and Web-based data in real-time in a single environment, without impacting the data warehouse or other systems.

For example, a utility company employee could look at usage data to identify patterns over a particular region or time period, and then analyze that data comparatively enabling real-time planning based on immediate access to usage data. Employees at a manufacturing company could analyze asset utilization in real-time on transactional data, while a financial services firm could perform real-time risk management and measure market exposure by combining structured credit scoring data with unstructured data, including information from the Internet.

According to Information Technology Research major, Gartner, in-memory analytics is an emerging technology that will drive mainstream business intelligence, making it optimal and that will change the scenario of how people make more informed decisions while interacting with data.

To power the next generation of business intelligence, business planning and business analytic applications, SAP is now working with leading hardware partners to deliver an in-memory software and hardware appliance optimized for real-time analytics using data from operational systems, data warehouses, real-time events and
the Web.

Concluding the future
The dynamics of handling data is only getting more complex. The appetite for getting real-time information is increasing at an unimaginable rate and catering this need is going to be exciting and challenging. I believe this is a great opportunity for all of us to rethink our approach to making sense of the ever-increasing amount of information available, no matter where it comes from. The world is waiting for the next ‘iPod moment’.
It’s time we gave the real world a real deal. Real-time products and applications. The only thing that will make good business sense. And this requires a real-time mindset shift. It’s possible!
Next article
Write your comment now

Email    Password: 
Don't have SiliconIndia account? Sign up    Forgot your password? Reset
Reader's comments(5)
1: From: Mrs. Mary David

This mail may be a surprise to you because you did not give me the permission to do so and neither do you know me but before I tell you about myself I want you to please forgive me for sending this mail without your permission. I am writing this letter in confidence believing that if it is the will of God for you to help me and my family, God almighty will bless and reward you abundantly. I need an honest and trust worthy person like you to entrust this huge transfer project unto.

My name is Mrs. Mary David, The Branch Manager of a Financial Institution. I am a Ghanaian married with 3 kids. I am writing to solicit your assistance in the transfer of US$7,500,000.00 Dollars. This fund is the excess of what my branch in which I am the manager made as profit last year (i.e. 2010 financial year). I have already submitted an annual report for that year to my head office in Accra-Ghana as I have watched with keen interest as they will never know of this excess. I have since, placed this amount of US$7,500,000.00 Dollars on an Escrow Coded account without a beneficiary (Anonymous) to avoid trace.

As an officer of the bank, I cannot be directly connected to this money thus I am impelled to request for your assistance to receive this money into your bank account on my behalf. I agree that 40% of this money will be for you as a foreign partner, in respect to the provision of a foreign account, and 60% would be for me. I do need to stress that there are practically no risk involved in this. It's going to be a bank-to-bank transfer. All I need from you is to stand as the original depositor of this fund so that the fund can be transferred to your account.

If you accept this offer, I will appreciate your timely response to me. This is why and only reason why I contacted you, I am willing to go into partnership investment with you owing to your wealth of experience, So please if you are interested to assist on this venture kindly contact me back for a brief discussion on how to proceed.

All correspondence must be via my private E-mail ( for obvious security reasons.

Best regards,
Mrs. Mary David.
Posted by: mary lovely david - Monday 26th, September 2011
2: Hi my dear,
My name is Mounace, i would like to establish a true relationship with you in one love. please send email to me at ( i will reply to you with my picture and tell you more about myself. thanks and remain blessed for me,
Your new friend Mounace
Posted by: mounace love love - Thursday 09th, June 2011
3: Big Data as described by Ferose is definitely the BIG THING of coming times, there is no 2 thoughts about it.

But. I feel in todays world distractions are inevitable. And when you select any technology as a future technology for your businesss, you should also take care of 2 things i.e. whether your technology is compatible with future distractions or not ? and whether you have a back-up plan in-place or not ?

Posted by: Sagar Bhavsar - Wednesday 30th, March 2011
4: Nice blog...
Posted by: Sharad Chaturvedi - Thursday 30th, December 2010
5: Hi,
I agree to most part of the article. However, apart from In-memory systems, there are other disruptive aproaches to Big Data problem. One of my favourites is FPGA based appliances from Netezza. We at Zinnia Systems are also working on creating next generation Business Analysis systems which uses BASE properties instead of traditional ACID properties. This paradigm shift results in performance gains in the magnitude of thousands. With this we have created world's fastest BSS/OSS applications certified both by IBM and SUN on their comodity servers.

One problem we see in only In-Memory products is the management of consistency and how to handle system restart.There is typically a delay until the in-memory data are re-populated which may be substantial for multi terabyte systems. With FPGA based systems, the full table scans are immediate and provide results at fraction of time required by traditional systems.

In summary, In-memory systems are a great leap forward ut like any other technology suffer from drawbacks.I will suggest to evaluate system capabilities before chosing any new technology.

Akhilesh Singh
CTO, Zinnia Systems
Posted by: Akhilesh Singh - Friday 29th, October 2010
More articles
The retail industry is witnessing an increased migration of customers ... -By-Kaushal Mehta
by Kaushal Mehta - Founder & CEO, Motif Inc..
The retail industry is witnessing an increased migration of customers from traditional brick and mortar retail to E-commerce (online retail)...more>>
Its 1 AM. Do You Know What Your Offshore Team Is Doing? -By-Samir Shah
by Samir Shah - CEO, Zephyr .
You probably do because you are on the phone with them! For all of you working in some technical management capacity here in Silicon Valley,...more>>
Disconnect: The Root Of All Execution Evils -By-Raj Karamchedu
by Raj Karamchedu - Chief Operating Officer, Legend Silicon .
These days are a mixed bag for me. Of late I have been considering "doing something bigger and better," in my life, perhaps seriously though...more>>
IT Services Rise Of Tier II Companies -By-Madhavi Vuppalapati
by Madhavi Vuppalapati - CEO of Prithvi Information Solutions .
IT Services Rise of Tier II companies The Indian IT outsourcing industry is going through very exciting phase in its business life...more>>
DLP, Prevention Is Better Than A Cure -By-Bhaskar Bakthavatsalu
by Bhaskar Bakthavatsalu- Country Manager, India and SAARC of Check Point Software Technologies.
Data loss occurs every day through corporate email. In fact, given the sheer number of emails an organization sends every day, data loss inc...more>>