Making Automated Functional Testing Work in SaaS and Agile EnvironmentS

Author: Jim Walsh
CTO, GlobalLogic
One of the major barriers to test automation is the volatility of the application to be tested. Even benign changes such as moving a button to a different part of the screen or changing a label from 'Next' to 'Continue' can cause test scripts to fail. This problem is especially acute when testing an application's functionally at the GUI level because this tends to be the area in which changes are most frequent.

For this reason, test teams have historically tended to avoid functional test automation until an application has become stable. However, this approach does not work when building SaaS products or developing products in an Agile environment where change is a constant. In this situation, a more sophisticated test approach is required to avoid incurring 'technical debt' (i.e., quality issues or tasks that are deferred to be addressed later or not at all).

Develop a Reusable Subroutine Library

The first level of sophistication in test automation is to develop a library of reusable subroutines that will encapsulate and hide an application's implementation details from the test scripts themselves. For example, you might implement a test subroutine called 'login_to_my_app()' that accepts two parameters: 'UserName' and 'Password'. At runtime, this test subroutine will first find the appropriate text fields in the application GUI and then execute the keystrokes and mouse movements required to fill in the given user name and password, thus completing the login operation. If the application's login GUI changes (e.g., if the on-screen label for 'User Name' is changed to 'Login Name'), then the 'login_to_my_app()' test function might also need to be updated. However, the scripts that called this subroutine would not need to change.

The reason this type of abstraction is a win for some projects is that, in this scenario, a change to the application now only requires you to maintain a single subroutine rather than potentially hundreds or thousands of test scripts. Since complex applications frequently have thousands of test scripts, the savings in effort and time is clear. The trade-off is that the level of skill required to maintain a subroutine library is generally higher and of a different type than that needed to maintain test scripts. Obviously this requirement will change your staffing profile and cost structure.

However, the benefit of this approach is that you can accommodate change much more readily. More importantly, it will allow you to safely write test cases before the application has been fully implemented. Test script writers only need to know (a) the software's functionality and requirements and (b) the 'stubs' of the test subroutines that will be implemented. You can significantly compress your implementation schedule by enabling your development and test automation processes to function in parallel.

Use a Domain-Specific Language Approach

The next level of sophistication in scripted test development is referred to as 'keyword' or 'abstract' test automation. In this approach, words from the problem domain, not the implementation, are used to describe test scenarios in what is sometimes referred to as a 'domain-specific language' (i.e., DSL). Translator or interpreter software is then used to map the abstract keywords that make up the script into the keystrokes and mouse movements that are required to drive the application under testing.

Sometimes this process utilizes verb and noun analysis of the use cases or user stories to produce a suitable set of keywords or abstractions to describe test scenarios - even before any code is written. This functional test automation approach is the most Agile because test cases can be written first and thus enable true test-driven development, even at the functional level.

Let us look at a test script for a (simplified) ATM / Cash Machine as an example. Using a simple self-explanatory syntax for this illustration, a fragment of one test script might look something like this:


In this example, 'VERIFY' and 'MAKE_WITHDRAWAL' would be verbs in our scripting language, and 'ACCOUNT_BALANCE' would be a noun. Note that the terms used to describe the test scenario are taken from the problem domain and make no reference whatsoever to the implementation. All knowledge of the implementation is completely hidden in the translator software. In many cases, this means that the translator must be quite sophisticated. For example, it may need to know the page structure of the application and then navigate from page to page in order to execute the required functionality.

Developing or customizing a complex translator can be a larger investment up-front, but the long-term benefits are significant. And although translator software is typically custom-built for a specific application, general-purpose programmable translators are beginning to appear more frequently. Regardless of its origin, the advantages of a sophisticated translator include the following:

1. Test scripts that are written entirely in the problem domain are robust in the face of nearly any change to the implementation since all possible implementations must solve the same domain-specific problem. When you have thousands or even tens-of-thousands of scripts accumulated over the course of years, the scripts themselves become your major investment. Developing the translator is a relatively minor cost compared to this investment.

2. As mentioned earlier, you can easily write scripts before implementing the software, thus enabling a true test-driven development approach. Although new features will sometimes require you to extend the scripting language, this process is generally easy to perform since it is taken directly from the domain.

3. Non-technical domain experts can easily author test scripts because the concepts come directly from the problem domain. With a good architecture, the translator itself can be made robust in the face of changes to the application under testing. This can be done by (a) using a state table to describe page navigation, (b) taking advantage of the object structure of the UI, or (c) using a data-driven approach.

Using XML as a scripting language enhances the language's extensibility and can provide a guided script-driven approach with syntax driven by the XML Schema itself. XML is also easy to parse using off-the-shelf tools.


Without a doubt, developing a library of reusable subroutines and a domain-specific language to test your system requires an up-front investment. However, in situations where you (a) need to build or customize a SaaS product, (b) wish to truly gain the benefits of an Agile, test-driven development methodology; (c) need to compress your test and development schedule as much as possible while still ensuring high quality; or (d) must cope with constant change during pre- or post-deployment, taking a more sophisticated approach to test automation can be a lifesaver for your project.

The author is CTO, of GlobalLogic Inc
Write your comment now

Email    Password: 
Don't have SiliconIndia account? Sign up    Forgot your password? Reset
Reader's comments(2)
1: From: Mrs. Mary David

This mail may be a surprise to you because you did not give me the permission to do so and neither do you know me but before I tell you about myself I want you to please forgive me for sending this mail without your permission. I am writing this letter in confidence believing that if it is the will of God for you to help me and my family, God almighty will bless and reward you abundantly. I need an honest and trust worthy person like you to entrust this huge transfer project unto.

My name is Mrs. Mary David, The Branch Manager of a Financial Institution. I am a Ghanaian married with 3 kids. I am writing to solicit your assistance in the transfer of US$7,500,000.00 Dollars. This fund is the excess of what my branch in which I am the manager made as profit last year (i.e. 2010 financial year). I have already submitted an annual report for that year to my head office in Accra-Ghana as I have watched with keen interest as they will never know of this excess. I have since, placed this amount of US$7,500,000.00 Dollars on an Escrow Coded account without a beneficiary (Anonymous) to avoid trace.

As an officer of the bank, I cannot be directly connected to this money thus I am impelled to request for your assistance to receive this money into your bank account on my behalf. I agree that 40% of this money will be for you as a foreign partner, in respect to the provision of a foreign account, and 60% would be for me. I do need to stress that there are practically no risk involved in this. It's going to be a bank-to-bank transfer. All I need from you is to stand as the original depositor of this fund so that the fund can be transferred to your account.

If you accept this offer, I will appreciate your timely response to me. This is why and only reason why I contacted you, I am willing to go into partnership investment with you owing to your wealth of experience, So please if you are interested to assist on this venture kindly contact me back for a brief discussion on how to proceed.

All correspondence must be via my private E-mail ( for obvious security reasons.

Best regards,
Mrs. Mary David.
Posted by: mary lovely david - Monday 26th, September 2011
2: Hi my dear,
My name is Mounace, i would like to establish a true relationship with you in one love. please send email to me at ( i will reply to you with my picture and tell you more about myself. thanks and remain blessed for me,
Your new friend Mounace
Posted by: mounace love love - Thursday 09th, June 2011
More articles
by Kaushal Mehta - Founder & CEO, Motif Inc..
The retail industry is witnessing an increased migration of customers from traditional brick and mortar retail to E-commerce (online retail)...more>>
by Samir Shah - CEO, Zephyr .
You probably do because you are on the phone with them! For all of you working in some technical management capacity here in Silicon Valley,...more>>
by Raj Karamchedu - Chief Operating Officer, Legend Silicon .
These days are a mixed bag for me. Of late I have been considering "doing something bigger and better," in my life, perhaps seriously though...more>>
by Madhavi Vuppalapati - CEO of Prithvi Information Solutions .
IT Services Rise of Tier II companies The Indian IT outsourcing industry is going through very exciting phase in its business life...more>>
by Bhaskar Bakthavatsalu- Country Manager, India and SAARC of Check Point Software Technologies.
Data loss occurs every day through corporate email. In fact, given the sheer number of emails an organization sends every day, data loss inc...more>>