An Architecture for the Future - Convergence of Cloud, IoT & Big Data

Date:   Monday , December 26, 2016

Headquartered in California, Brillio is a global technology services company with proven excellence in developing and deploying cutting-edge solutions that enable businesses to tackle the competition better and capture business value quickly and efficiently.

Having completed my walk the other day, I turned to check the number of steps I had taken. By then, this information on my smart watch synced up with my phone, and then on to some servers of the fitness app, where information from thousands of people who go for walks around the world is crunched and in a blink of an eye, I am informed by the app on where I stand, vis-a-vis the average person of my age. I would have spent the next hour strategizing how to catch up with others. Were it not for a message on my phone that let me know that my daughter had got off her school bus and entered the building. Another message told me that a package I was expecting had left the logistics provider\'s delivery center. That is when rumination caught hold of me, and got me thinking about how life has changed. Imagine this being just the beginning! As one of my friends, incidentally a futurist, puts it - we shall soon be living the life H. G. Wells and Arthur Clarke dreamt of.

Gartner predicts that by 2020, there will be over 26 billion units of machinery installed worldwide, which will form the IoT network. That is four times the number of humans on this planet. Most of these machines will generate data 24*7, leading to enormous amounts of data being available to us to in order to assist in making critical decisions. What this means is everything we do today can be tracked, monitored and improved. Efficiencies will no longer be derived from statistical samples, but large volumes of data leading to more accurate decision making.

Machines fitted with sensors have been around for some time now. What has changed is the ability to connect geographically spread machines and manage them remotely. These probabilities have become possibilities by dint of the perfect storm that has arisen from the convergence of disruptive technologies of the past few years namely Big Data, IoT and Cloud.

The benefit that we have come to expect of cloud is that it offers elasticity, scalability and flexibility. Cloud also offers redundancy, reliability and easy accessibility that were until now difficult to achieve with enterprise IT data centers. Cloud also brings in the flexibility of starting small and not having to invest in large capital upfront.

In conjunction with big data, cloud offers the ability to analyze massive amounts of IoT data which can be streamed easily, without the worry of building a private network of devices. What also helps is that with cloud, one can start with a smaller set of devices, and then scale up seamlessly as the quantum of data grows.

It is however the ability to manage large volumes of data while simultaneously deriving real-time insights using big data analytics, that has really opened up to mankind a world of opportunities. With every passing day, newer tools and techniques emerge that improve the ability to handle large volumes of high-velocity data and churn out real-time insights. Data management options too are aplenty, ranging from Cassandra to HBase to Mongo db, with the choice being based on the need.

In the last three years, Cloud providers like Azure, Amazon, Google and IBM have significantly added new data related services to quickly provision and build large scale big data solutions. We have also seen increased acceptance and adoption of cloud by enterprises who had questions in the past about security, accessibility and adherence to compliance laws.

This shift in the way enterprises today use data has propelled big data to the core of the business ecosystem. Enterprises are able to gain insights from this data, thereby making more well-informed decisions and therefore thriving, while others lacking these options are lagging. A sales decision is done much quicker but with larger amount of data, Inventories are better planned with more sophisticated demand sensing systems, logistics are more streamlined with every minute detail being now captured. Discovering correlations between two unrelated variables have helped companies make some strategic decisions.

Amazon for instance leverages Big data analytics for product recommendations and has seen a lot of improvement in cross-sell and up-sell opportunities. South-west airlines has been able to use customers travel data and online behaviors to come up with new offers and experiences. Of late big data analytics has seen increased adoption in the world of politics. Elections are won and lost based on strategies derived using big data analytics.

Manufacturers are today able to achieve better productivity by managing their machines better, utility companies are able to reduce transmission losses basis data from remote sensors, logistics companies are able to improve operational efficiencies through better fleet management, and large industries are able to control asset thefts; all of these eventually leading to improved bottom-lines.

There are however a few pitfalls that companies should look out for while setting up these solutions like managing data security, adhering to industry standard protocols and data exchange formats - else integrating with other products will be a challenge, ensuring enough fail-over is maintained and training the models well before productionizing them. These are only a few of the many things to be kept in mind but none of this will slow the rate adoption of technology that we have been witnessing in the past few years.

Governments and civic authorities too, are today leveraging the value of this convergence. Smart meters, energy efficient homes, traffic management solutions and even health monitoring solutions are commonplace today, to the extent where we take them for granted sometimes forgetting the technology behind them. What we are seeing now is the beginning of a revolution, driven by the dual powerhouses of technology and data, which will change our lives for the better.

The real impact will be the one that we will see in daily lives. Our home appliances will generate data by the second that would get stored in the cloud. This then gets aggregated with data from many such appliances from millions of homes nationwide. Analytic models are then run on this data to come up with smart insights on electricity usage, potential break-down and also energy efficiency.