Sophisticated Rule Engine to Manage Robust Data
Date: Tuesday , October 20, 2015
I have been in the enterprise technology business over 30 years and believe we are on the cusp of a second Database revolution. It\'s hard to remember Oracle as a $250M company, but it was when I was there in 1988 and we were kicking and scratching to get people to get their heads around the benefits of moving off of hierarchal IBM databases and into the relational world. We are early days in another major technology shift catalyzed by the reality that the cost of enterprise compute continues to plummet. The two main drivers of this price deflation are major improvements in hardware efficiency, and the proliferation of the open source movement. These two realities have removed the once nearly impenetrable barrier of capital from the process of innovation, and the result is the rapid increase of two kinds of companies:
1. Companies that sell technology (Aerospike, MongoDB and Cloudera)
2. Companies that exist solely because of technology (Uber and airbnb)
In the hardware world, Amazon Web Services has come out of nowhere to become a $9B business in about five years. I haven\'t had a conversation with a customer in years that doesn\'t have AWS as a major component of its compute paradigm. In the software world, the open source movement has mutated from a small group of likeminded hackers writing code to upset the establishment, to a large group of developers taking code for great software that they can use for free. These deflationary dynamics are a reaction to decades of pricing abuse and a lack of innovation from the large vendors that have abused duopolies in nearly every major category of technology.
The Deflationary Impact on the Database World
I believe a brief discussion of how these dynamics are affecting my company Aerospike, the High Performance NoSQL Database, are an analog for the changes happening in every category of technology today. From my perspective, most of the changes in the database space are transitional; there\'s a new crop of vendors with a better way to accomplish existing tasks. The database is no longer a product, it\'s a category, and IT departments now have the luxury of selecting the right tool for the right job, at the right price. The growth of well-known start ups like Cloudera and MongoDB is predicated on a better/more cost effective way to accomplish the same task historically run on a legacy database. The vast majority of deployments are replacements or extensions of existing commodity workloads, like data warehouses, content management platforms, web catalogs etc.
These workloads are deflationary in that users are able to accomplish the same task for a fraction of the price because they are running them in AWS on community editions of open source software. These projects are quickly moving from the early adopter phase to the mainstream, and the best evidence of this is the organic growth rates of the largest enterprise technology vendors like IBM, Oracle, Teradata and Dell which are all in decline.
Building New New Things and New Old Things
However, at Aerospike, our product is transcendent, not transitional, because the speed and scale of our database enables our customers to do things that have never been possible. We have removed some of the most critical technical limitations around speed and scale that have made traditional databases an inhibitor to innovation, and we have repositioned the database as an enabling technology. Two use cases illustrate this point.
A New Thing
AdTech as we know it today didn\'t exist a decade ago because the technology needed to create it was exclusively the domain of large, profitable tech companies that could afford to spend 10s of millions of dollars on in house engineers with the mandate to create the technology to support the project. When we created a commercial version of this kind of technology, the space took off. There aren\'t many household names in the AdTech space that resonate with engineers, but we are very proud of the fact that 10 of the top 20 AdTech vendors are customers and over $10B in ad spending runs through our database.
A simplified explanation of the use case goes like this: Every person on the Internet has a profile that\'s been created with cookies aggregated from years of Internet browsing. The AdTech vendors take your profile, and match it with your real time web activity. The nexus of past (your profile) and present (what you\'re doing on line right now) creates the opportunity to serve up an ad that is relevant and thus most likely to create a transaction. The better the AdTech vendors are at serving up an ad that creates a transaction, the more they can charge for their services. Aerospike offers AdTech companies the scale to ingest an enormous amount of data, and the speed to calculate complex algorithms that produce superlative results. Our secret sauce, a key value store written explicitly for solid-state drives, enables us to deliver this functionality with very reasonable TCO.
A New Old Thing
The domain of managing credit card risk and fraud is as old the credit card itself. However, many of the processes that manage risk/fraud are also that old because they have been limited by the capabilities of legacy technology. Our most forward thinking customers are recreating these processes by writing much more sophisticated rules engines that allow them to manage much more robust data sets. Success in this endeavor will be a significant competitive differentiator.
Historically, risk/fraud is measured on a rather narrow set of criteria because legacy databases couldn\'t handle large data sets or run complicated rules engines in the short amount of time they had to return an answer. By rewriting risk/fraud apps on Aerospike, credit card companies now approve transactions in atypical situations like traveling abroad. The data set is large enough and the rules engine is fast enough that the credit card vendors know that you purchased a plane ticket and a hotel room two weeks ago, so when you land in London tomorrow, you can charge that cup of coffee at Heathrow. What new thing can you create?