Future of Test Automation Tools & Infrastructure Date:   Thursday , January 06, 2011 There are some specific trends noticeable in the way we do UI-based test automation. Technology has advanced, new interfaces have been created, and, as a result, new tools have been developed that changed our way of doing test automation. Evolution of Test Automation Frameworks Let us revisit how the test automation tools and frameworks have evolved. l The crux of any automation framework is its core engine. l The traditional record-and-playback set of tools sit on top of this core framework. l The rigidity and difficulty (amongst other factors) in customizing the standard record and playback scripts resulted in the new layer being added - that of the Custom Frameworks. Custom Frameworks are our customized scripts that do optimal record and playback. These frameworks can be of any type as depicted in the picture on the side. More often than not, when one starts to develop a Custom Framework using either of the 4 mentioned types, you eventually end up with a Hybrid solution - which is a combination of the different frameworks. There has been a need for writing tests in a new lingo. Something that will be easier for non-coders (example: Business Analysts) to read, understand, and maybe also contribute to. Thus arose a new type of methodology and framework for building our Automated Tests - BDD - Behavior Driven Development. There are many tools in the market that allow BDD, namely, Cucumber, JBehave, RSpec, Twist, etc. The BDD frameworks have also been around for some time now. When thinking about this pattern, the question that comes in my mind is - What is next? What is next? To answer the question - “What is next?" we need to understand the current UI advancements. Most major portals / websites now allow the user to create their own customized pages, based on the data they want to see. And all this is done not by major configuration changes, or talking to a sys-admin. Example: In the below image, the Yahoo page has widgets which you can configure and arrange in the order of your preference, so that you are able to see what you want to see. This is done simply by doing some mouse gestures and actions (drag and drop). The whole world appears to be moving towards providing content based on preferences or doing actions based on direct “interactions“. New generation mobile phones have touch screens that allow doing various different activities using touch-based gestures, or orientation of the device itself. If you recall the movie, “The Minority Report”, the technology depicted there is a step ahead of all this. The movie shows the actors interacting with images, videos, voices, all using gestures made by the sensors in his hand. This technology is already advanced enough and is being used / tried in a few specialized fields - like Image editing, high-end store catalogs, etc. “Gesture Based Technology“ is the future that we have already stepped into! How does this affect the software test automation? Well, this affects us in a major way. * We eventually will be developing software to support and build on top of this technology. * If we are developing software, that means we need to test it. * This also means that we need to do automation for it. It is imperative for us to start thinking about how will we, as testers, test in this new environment? What tool support do we need to test this effectively? Lastly, let’s think BIG - why can’t we create / write our automation tests using similar interactions? UDD - UI Driven Development If a user can interact with a system using gestures, why can’t we testers change the way we write automated tests? Why do we have to rely on coding, or writing tests in BDD format? Why can’t we raise the bar and write tests using a different, interactive mechanism to represent our tests? UDD provides us the ability to develop our test automation suite and framework using a visual (non-code, non-text) medium. Since a picture speaks a thousand words, seeing the test components as UI objects will provide quicker visibility into the functionality being tested. The UDD framework will have the following components: Discovery This is a very essential and critical piece of functionality. We want to make sure we do not need to reinvent the wheel. We would also like to reuse our existing frameworks as much as possible and make the transition to UDD as seamless as possible. This component will be able to reverse engineer the existing code base, and create an UI object hierarchy available in a palette / repository. Example: After running the discovery tool against the existing source repository, the UI objects will be created like this: Author To create new objects / tests scripts, the test author will use the UI objects from the palette / repository, and, ‘simply’ drag-&-drop various UI objects to create new objects / test scripts. All the ‘intelligent’ code refactoring and restructuring will happen automatically in the backend. Refer to the picture below for reference. Note: Currently, using reverse engineering tools, we can create class diagrams / UML diagrams from existing code base. In the context of UDD, these are at present dummy objects. The UDD objects, which when moved across, would result in the framework making appropriate modifications in the underlying code-base, without the user having to manually intervene. Example: To create a new test, simply drag & drop various existing UI objects in proper sequence to create new test scenarios as shown below. This provides a pictorial view for the people looking at these tests. When new functionality needs to be added in the code base, the test author can use the Record & Playback Engine functionality in combination with writing specific code for the same if so required, and the UDD framework will automatically create appropriate UI objects for it. There is also an option to publish it to the repository for everyone’s use. Execution Engine The execution engine provides a lot of flexibility in terms of how the tests should be run. There are various options: * Specify Critical Test Failure (CTF) dependencies. * Run the tests within UDD framework * Generate a command for the set of tests the user wants to run, which the user can simply copy and paste at the command prompt and execute the tests directly without having to create the command manually. * Execute the tests on the same machine, remote machines or combinations so desired. * Launch tests via CI (Continuous Integration). Reporting Engine & Trend Analyzer We are used to seeing the default, yet quite comprehensive reports generated by the various unit testing frameworks (jUnit, nUnit, TestNG, etc.). However, what is lacking in this is the ability to automatically consolidate reports from various different runs and archive them, create trend analysis and charts of various types which may be required and interesting to track the health of the system. The Reporting Engine & Trend Analyzer will provide this functionality out of the box. Plugin Manager This complete framework will be built on plugins. A set of core plugins enables this framework. The community can contribute various other types of plugins based on their need, requirement and vision. A ‘hot-deployment’ mechanism will enable additions of the new plugins into the framework without needing a restart. Example: Some plugins include: * Record & Playback Engine - to record user actions as UI objects * Custom Reporting / Trend Analyzer - to allow creation of customized reports * Scheduler - to provide test scheduling * Languages / Frameworks - It should be easy to change the underlying framework at the click of a button. This way the admin user can choose to change from say using Selenium to Sahi just by choosing which UI framework is to be used. Similarly, it should be possible to select which language (C#, Java, Ruby, etc.) should be used for the code generation. * Integration with External Tools and Repositories - example: file diff / compare tools, source code repositories, integration with CI (Continuous Integration) systems, etc. How do we get ready for the future? By following a few practices in existing test automation, we can be in a good state to adopt what the future would have to offer. The Author is a Lead Consultant, QA, ThoughtWorks