Making Automated Functional Testing Work in SaaS and Agile Environments Date:   Tuesday , November 01, 2011 One of the major barriers to test automation is the volatility of the application to be tested. Even benign changes such as moving a button to a different part of the screen or changing a label from 'Next' to 'Continue' can cause test scripts to fail. This problem is especially acute when testing an application's functionally at the GUI level because this tends to be the area in which changes are most frequent. For this reason, test teams have historically tended to avoid functional test automation until an application has become stable. However, this approach does not work when building SaaS products or developing products in an Agile environment where change is a constant. In this situation, a more sophisticated test approach is required to avoid incurring 'technical debt' (i.e., quality issues or tasks that are deferred to be addressed later or not at all). Develop a Reusable Subroutine Library The first level of sophistication in test automation is to develop a library of reusable subroutines that will encapsulate and hide an application's implementation details from the test scripts themselves. For example, you might implement a test subroutine called 'login_to_my_app()' that accepts two parameters: 'UserName' and 'Password'. At runtime, this test subroutine will first find the appropriate text fields in the application GUI and then execute the keystrokes and mouse movements required to fill in the given user name and password, thus completing the login operation. If the application's login GUI changes (e.g., if the on-screen label for 'User Name' is changed to 'Login Name'), then the 'login_to_my_app()' test function might also need to be updated. However, the scripts that called this subroutine would not need to change. The reason this type of abstraction is a win for some projects is that, in this scenario, a change to the application now only requires you to maintain a single subroutine rather than potentially hundreds or thousands of test scripts. Since complex applications frequently have thousands of test scripts, the savings in effort and time is clear. The trade-off is that the level of skill required to maintain a subroutine library is generally higher and of a different type than that needed to maintain test scripts. Obviously this requirement will change your staffing profile and cost structure. However, the benefit of this approach is that you can accommodate change much more readily. More importantly, it will allow you to safely write test cases before the application has been fully implemented. Test script writers only need to know (a) the software's functionality and requirements and (b) the 'stubs' of the test subroutines that will be implemented. You can significantly compress your implementation schedule by enabling your development and test automation processes to function in parallel. Use a Domain-Specific Language Approach The next level of sophistication in scripted test development is referred to as 'keyword' or 'abstract' test automation. In this approach, words from the problem domain, not the implementation, are used to describe test scenarios in what is sometimes referred to as a 'domain-specific language' (i.e., DSL). Translator or interpreter software is then used to map the abstract keywords that make up the script into the keystrokes and mouse movements that are required to drive the application under testing. Sometimes this process utilizes verb and noun analysis of the use cases or user stories to produce a suitable set of keywords or abstractions to describe test scenarios - even before any code is written. This functional test automation approach is the most Agile because test cases can be written first and thus enable true test-driven development, even at the functional level. Let us look at a test script for a (simplified) ATM / Cash Machine as an example. Using a simple self-explanatory syntax for this illustration, a fragment of one test script might look something like this: VERIFY ACCOUNT_BALANCE = 10000 INR MAKE_WITHDRAWAL 1000 INR VERIFY ACCOUNT_BALANCE = 9000 INR In this example, 'VERIFY' and 'MAKE_WITHDRAWAL' would be verbs in our scripting language, and 'ACCOUNT_BALANCE' would be a noun. Note that the terms used to describe the test scenario are taken from the problem domain and make no reference whatsoever to the implementation. All knowledge of the implementation is completely hidden in the translator software. In many cases, this means that the translator must be quite sophisticated. For example, it may need to know the page structure of the application and then navigate from page to page in order to execute the required functionality. Developing or customizing a complex translator can be a larger investment up-front, but the long-term benefits are significant. And although translator software is typically custom-built for a specific application, general-purpose programmable translators are beginning to appear more frequently. Regardless of its origin, the advantages of a sophisticated translator include the following: 1. Test scripts that are written entirely in the problem domain are robust in the face of nearly any change to the implementation since all possible implementations must solve the same domain-specific problem. When you have thousands or even tens-of-thousands of scripts accumulated over the course of years, the scripts themselves become your major investment. Developing the translator is a relatively minor cost compared to this investment. 2. As mentioned earlier, you can easily write scripts before implementing the software, thus enabling a true test-driven development approach. Although new features will sometimes require you to extend the scripting language, this process is generally easy to perform since it is taken directly from the domain. 3. Non-technical domain experts can easily author test scripts because the concepts come directly from the problem domain. With a good architecture, the translator itself can be made robust in the face of changes to the application under testing. This can be done by (a) using a state table to describe page navigation, (b) taking advantage of the object structure of the UI, or (c) using a data-driven approach. Using XML as a scripting language enhances the language's extensibility and can provide a guided script-driven approach with syntax driven by the XML Schema itself. XML is also easy to parse using off-the-shelf tools. Conclusion Without a doubt, developing a library of reusable subroutines and a domain-specific language to test your system requires an up-front investment. However, in situations where you (a) need to build or customize a SaaS product, (b) wish to truly gain the benefits of an Agile, test-driven development methodology; (c) need to compress your test and development schedule as much as possible while still ensuring high quality; or (d) must cope with constant change during pre- or post-deployment, taking a more sophisticated approach to test automation can be a lifesaver for your project.