The Era of Big Data Applications

Date:   Tuesday , April 01, 2014

Continuuity is a cloud-based Big Data application platform for developers. Based in Palo Alto, CA, the company is backed by leading investors including Battery Ventures, Andreessen Horowitz and Ignition Partners.

We are in the midst of a major shift in technology, and it has incredible potential to change the way we work, live, and play. Big data is no longer just an experimental framework for efficient data processing at scale but is also becoming the central repository for companies who want to take full advantage of their data. Some have called this phenomenon the \"data lake,\" a place where data with dissimilar volume and variety seamlessly comes together to enable unprecedented insights.

Until recently, this concept used to be limited solely to Silicon Valley giants like Facebook, Yahoo, Google, and LinkedIn. While these web companies pioneered the use of data as a competitive advantage, even companies outside of the technology industry-particularly in financial services, retail, and telecommunications-can use data for competitive differentiation thanks to continued advances in technology.

Some experts claim that data assets will become more valuable to businesses than financials. But the truth is that most companies still face tremendous barriers in their efforts to use data for better customer experiences, more empowered employees, and higher revenue. Chief among these challenges are the inherent complexity of the technology and lack of big data talent.

How to Address the Data Skills Gap

There has been much consternation over the rise of the demand for data scientists and the shortage of Hadoop specialists who can master the complexity of big data but also extract practical business insights from the volumes of disparate and often convoluted data. Recent research from Accenture contends that there is not enough PhD talent to fill the jobs, particularly in the United States, where 80 percent of new data scientist jobs created between 2010 and 2011 remain unfilled. On top of this, even the number of graduates with the required technicals skills for basic data analysis can\'t keep up with rising demand.

Analytical skills will certainly remain important, but focusing too narrowly on analytics makes it unlikely that we will realize the full potential of data. Companies treating Hadoop and related ecosystem components like a traditional database for only data analysts make it much harder to put their data to work in a meaningful and measurable way. What we need is a true platform that allows developers--who vastly outnumber scarce data scientists--to build big data applications.

Big Data Applications are the Future

While analytics may allow data analysts to identify patterns and make data-driven decisions, the technology has two considerable deficiencies-it is accessible to a narrow group of workers and it cannot be automated. In contrast, big data applications do not require the same level of human involvement; these closed-loop applications have powerful machine learning and predictive analytics capabilities built in without needing a whole team of analysts to oversee them. At the same time, big data applications can deliver a superior outcome because they are developed for specific business needs rather than forcing companies to rely on \"one size fits all\" tools that may not be optimal for generating the actual result that is needed.

To understand the idea behind big data applications, you need look no further than your own web browser or smartphone. Yahoo\'s front page was one of the first big data applications-unstructured data turned into an exploration and discovery platform that anyone can search. If you shop online at Amazon ot Gilt Groupe or read the news on Yahoo or LinkedIn\'s Pulse, you are already enjoying the benefits of big data applications including personalized recommendations based on your own preferences and online behavior. In the future, your car may use a big data application to drive itself or to warn you of potential mechanical issues before they arise.

The Big Data (application) Opportunity

Big data has been described as the \"new gold\" and \"the most valuable currency,\" but we cannot take complete advantage of its value by relying on antiquated tools and traditional approaches to tomorrow\'s workforce. As with consumer technology, what matters and impacts our lives the most are applications, not the underlying infrastructure.

Similar to the apps that consumers already use every day, these applications will hide the complexity of big data by making it useful and practical for specific tasks. This is where the full opportunity of data resides-creating the next generation of big data applications.