The Big Data Challenge Date:   Tuesday , February 03, 2009 Wired Magazine has called our era “The PetaByte Age” – an age of infinite storage, clouds of processors, and most importantly, exponential amounts of data generation and capture. Added to this is the relentless increase in the speed of competitor response. Companies must implement strategies to digitize and monetize a diverse set of business data – about purchasing patterns, customer behavior, potential fraud patterns, and so on at scales and speeds never before contemplated. Businesses have a strong desire to use data as a competitive frontline weapon rather than post-fact on. Companies that can extract value from their data can reshape their offerings to be in line with customer needs and discover new opportunities to outpace the competition. Harnessing this new competitive weapon is the biggest infrastructural investment every organization will undertake. The most competitive companies, like Google, Netflix, Amazon.com, and Walmart have leveraged their data assets and successfully dominated their respective markets. The new processes required to bring analytics to the business frontline are fundamentally about efficient data management based on the need for fast and low-latency access to data. Legacy data warehouses typically take a day or more to provide new business data to analysts and applications. For the business frontline, 24 hours is now unacceptable latency. The new infrastructure will require frontline data warehouses. The current decade is the best time for software entrepreneurs to build lasting companies. Only a handful of new enterprise offerings (e.g., VMWare, Data Domain) have emerged in the last decade. This has left in place a stale IT infrastructure. The infrastructure is breaking, creating consistent and systematic opportunities for a new infrastructure base. The dominant challenge for entrepreneurs will be raising and managing capital: the art of “doing more with less” will skew the market in favor of entrepreneurs rather than larger fossilized organizations. The drought in capital results in a systematic ripple through execution: markets will be less tolerant of execution failures, requiring entrepreneurs to plan and execute carefully and make investments based on data rather than fashion. But these same factors will work in favor of a well-run entrepreneurial company: the competition will be mowed down leaving a bigger market for you! The author is CEO, Aster Data Systems