Cyber Security in the Era of Big Data Date:   Thursday , February 05, 2015 Cyber security is everywhere. You can\'t pick up a the Wall Street Journal, New York Times, or any online business publication without reading a story about data breaches at both corporations and government organizations. A CIO has said, a company either knows it\'s been breached or it doesn\'t know it\'s been breached. But it was not always like this. What has changed? Why are we seeing cyber security and data breaches in the new paradigm so often now? Longing for the good old days of certain security Today\'s adversaries, commonly called hackers although that term seems a little obsolete given the gravity of their attacks, have evolved from hobbyists to professionals. They are well trained and well-funded, and run the gamut from social activists and state-sponsored operators to criminal syndicate members. Just as they have become more sophisticated, so have their tools, techniques and procedures. Attacks used to be indiscriminant, like viruses in the wild, spreading and replicating on unprotected systems. Now they\'re targeted to specific firms with the objective of stealing, encrypting or destroying data. Security professionals used to be confident they could lock down and secure their networks to prevent incursions. Now, the mindset is that incursions are inevitable and the burden is on them to figure out how to detect and remediate an attack before the data is compromised. Another problem facing Chief Information Security Officers (CISO) is the shortage of skilled network defenders. Now that the cyber security problem has gotten worse, the demand for network defenders is skyrocketing against a limited pool of qualified candidates. Enter the Chief Information Security Officer CISO\'s have an incredibly tough job, and at the end of the day they know there are systems are going to be breached. The traditional approach is to use a layered defense strategy. These strategies, first adopted for the battlefield or physical security, are important to operate and maintain, but have limitations. For example, intrusion detection and prevention systems (IDS/IPS) userules so they can only protect organizations from previously identified threats. Security Incident Event Management Systems (SIEMS), rely on system logs, second-hand representations of what happened on the network, to track incursions. Skilled adversaries can delete or change logs and make the data worthless for cyber security. Network Data Capture Tools can see all but they can also be too slow. It is tempting for the CISO to add more cyber security instrumentation to the network or to increase the sensitivity of existing security tools. Unfortunately, this results in more alerts to be processed by an already overworked security staff. The CISO must evaluate new tools, techniques, and procedures to stay ahead of the adversaries. Harnessing Big Data for Cyber Security is the Path Forward In 2013, the Ponemon Institute conducted the Big Data Analytics in Cyber Defense* study to learn about organizations\' cyber security defenses and the use of big data analytics to recognize the patterns that represent network threats. As a result, some surprising statistics surfaced, along with powerful insights that are helping to forge more productive conversations on cyber security in organizations of all sizes. The study showed there is an alarming new reality: cyber security challenges continue to grow, with new threats expanding exponentially and with greater sophistication. However, what will shock any professional is how organizations perceive their understanding of readiness and vulnerabilities as well as their views on big data cyber security analytics versus reality. For example, 61 percent of respondents believe big data analytics can solve pressing security issues faced by companies and government; though only 35 percent say they have solutions in place that are the same or comparable to big data analytics for cyber defense. Sixty percent of respondents agree that launching a strong defense against hostile actors and other cyber criminals require their organization to see and quickly contain anomalous and potentially malicious traffic in networks. However, only about half say that what is hindering their efforts to do so effectively is the scarcity of in-house personnel or expertise. When asked specifically where their organizations are most deficient in their ability to become more proactive in their approach to cyber threats, more than one-third say it is enabling security technologies and 35 percent say professional expertise. Using Big Data for Cyber Security Rules-based IDS/IPS, SIEMS that analyze log data and network capture tools all have a role to play. Big data can be used to augment these defense strategies by providing fast and actionable information to the network defenders. Integrating network operations data and security product data from a layered defense strategy will provide an integrated data source that can capture event correlations or relationships where individually the risk appears low, but when analyzed in aggregate paint a clearer picture of cyber risk. Once integrated, security professionals can apply analytics using MapReduce functions for discovery of new anomalous network traffic or behavior and statistical functions to calibrate rules to detect know anomalous network traffic. Additionally, the integrated security data is ideal for BI reporting. Another example of using big data for cyber security is network data packet capture and analysis. Live network data, or big data in this case, is captured from taps on the network, written to disc, enriched with external data, and then made available for analysts to query. This network data is considered to be \"Ground Truth\" and inherently more reliable and valuable than data from traditional approaches. The benefits of using big data for cyber security include: - Increased events-responded to by the security staff. - Substantially reduced or eliminated damage from breaches. - Create a dramatically more effective and efficient security team. - Maximized security infrastructure investment. - Enhanced confidence that the network is actually secure. The world has changed for security analysts, network defenders and hunters, the CISO and the rest of the C-Suite. Harnessing big data for more effective cyber security is the path forward.