Test or Not to Test Date:   Wednesday , October 02, 2013 Coverity is a development testing company based out of California. Founded in 2003, the company has raised $22 million from Foundation Capital and Benchmark Capital. The world is changing. It’s speeding up. It’s software led. There are millions of lines of code written every day. Coverity Mars Rover alone has more than two million lines of code pre-programming its every move. There is more room for error yet zero tolerance for it. Releasing faulty software is something that no business can afford. All of this makes development testing critical to any development process. But is that the reality? The Issue As with any new thing, when software development was in its infancy, developers were not really aware of the necessity to test it, which led to a flood of faulty software. Essentially, the only form of testing was basic debugging as hackers were yet to pose a real threat and resource management was a much bigger concern. For a long time after that, development testing was still pushed on the periphery of the development cycle. The reason for that was simple, developers were forced to produce more lines of code in a shorter amount of time to meet the demands of accelerating software market. The time dedicated to testing was reduced to bare minimum. The result? Faulty software resulting in data leakage, unexpected errors and serving as a gateway for hacking attempts. Once the importance of the testing process was acknowledged, developers thought it was enough to test without putting importance on the quality of testing. This era saw developers testing their software after it was written, which meant another stage to the development cycle was added and often thousands of additional lines of code would be written to attempt to 'cover up' errors. Creating this extra code increased the overall size of the software, made it less transparent and slower to react. Think again. Forrester Consulting wrote a Software Security Risk Report in 2012 that found that many organisations today still struggle with the most basic security flaws and do not have a holistic or strategic approach to application security. Staggeringly, only 17 percent of companies surveyed actually test during the development cycle and more than half do not audit their code before integration testing. This resulted in 51 percent of respondents having had at least one web application security incident since the beginning of 2011. Plus, 18 percent of those respondents experienced losses of at least $500,000. There were other problems that developers struggled with - such as the need for too much security expertise - that slowed down the whole testing process, highlighting a skills gap of security in development testing and development testing in security professionals. The Holy Grail of Development Testing Traditional unit testing, while valuable, has been inefficient, failing to reliably identify critical parts of the code and lacking insight into change impact. Similar to matching medicine to each strain of a disease, some defects are architectural in nature, and it's best to find these during threat modelling and design review. Other defects are code-level problems, best found during development. Very few defects should be left for verification after development, and by the time you reach release date, the cat is out of the bag. If most of the security issues being fixed are in the verification or release phase, that's a red flag that upstream practices are not working effectively. However, there is not always the time to test everything. As mentioned earlier, there is a limited amount of time that can be allocated to development testing. This highlights the need for efficient testing of the most critical parts of the code. It's a risk mitigation exercise that aims to focus the limited test cycles available on the riskiest issues. The question to ask is no longer to test or not to test. Instead, it is what code to test, and when to test it.