siliconindia

Test Automation Framework – Challenges In The Ever Changing Technology Scenario

Author: Ramesh Krish
Chief Technology Officer, Indium Software India
The Background

Test automation has matured over a period of time; from the crude form of automated tests that were set up just to run specific or stop gap runs, we today have much refined and well defined automation tests. According to a study, in the year 2004 the number of automated tests that were conducted globally as against manual tests were a mere 5 percent. However, by 2006 this increased to 20 percent. Going by this rate, we may be around 50 percent today. According to a study conducted by the US NIST, software producers lose 21.2 billion dollars annually because of inadequate testing. Based on market estimates, software companies worldwide invested 931 million dollars in automated software testing tools in 1999, with an estimate of at least 2.6 billion dollars in 2004. This certainly tells us one thing: Customers are seeing the real benefits of test automation. The last 5-6 years has seen a considerable run in commercial testing tools and we now are seeing the trend of customers requesting for ‘opensource’ tools, thanks to changing economic situations.

One of the key factors in choosing a test automation tool is the technology that is used to build the application. Not all tools support all technologies and that throws up the challenge of maximizing test automation returns with limited capabilities of tools. For example, an application developed in .NET or Java may require test automation tools to support such technology. Some of the tools come with specific plugins to address such challenges. So, choosing the right tool goes a long way in getting the expected RoI.

Technology, a Show-Stopper?
One would certainly run out of business if technology becomes a show-stopper. While that could be a very generic statement specific to test automation, technology has been one of the biggest challenges. With every new patch or release of an OS, browser, languages, or for that matter even test automation tools, the challenges to maintain automation scripts are undeniably compounded. By the time you start a test automation project and complete it, you may find changes in the environment it is set to work. The script that you have developed, while the customer agreed for a specific version, will not be of any use to him if the customer is going to upgrade their systems to the new version of the browser, for instance. Here is when a robust framework and good coding practice facilitate a smooth transition.
Develop framework that is independent of test automation tools. Since every tool has some limitation, it is best to use a combination of tools to take the maximum returns out of test automation investment. A combination of commercial and opensource tools is a good mix. Having a framework that best supports such a mix will give the best advantage over other frameworks.

Never Ending CRs
CRs or change requests have become the norm of the day. Today, CRs not necessarily include change in functionality and features due to changing business requirements but also the impact the application goes through because of the technological changes. Let us consider a situation like this: an application that is stable has about 1,250 automated test scripts that are run every other day and being used by 100s of customers worldwide. Due to business reasons the company decides to change the technology or architecture of the application. Now this is going to have an impact on the automation scripts as well. Either the scripts will have to be discarded completely, which is money into the drain, or evaluate the possibility to modify the scripts to suit the new architecture or technology. In case the latter being a possibility, an evaluation of time and effort required for such modification and associated costs need to be considered. Since users are realizing the benefits of automation, this aspect should not be missed when someone is considering changes to the application. Thus, it is important to integrate test automation in the purview of CRs, i.e., whenever a CR is raised, it is important to evaluate its impact on the automation scripts. A robust test automation framework will make such impacts minimal.

Build Framework that Works
Test automation framework is a set of rules, procedures, and tools that drive automated testing. One of the main advantages of using a framework is the low cost of maintenance. For any changes to the application, only the concerned test case(s) need to be updated while the rest of the script remains the same.
Some of the most common frameworks are data driven, keyword driven, modular, and hybrid. There are again custom built frameworks that fit to specific needs of the application. Spending enough time on choosing the right framework and designing it is the key to test automation success. A good and robust framework should consider as many assumptions as possible. This will eventually make the framework quite flexible. While it will be difficult to prove due to lack of data but the general belief is that most of the test automation projects fail in the long run as the framework is not flexible enough to adapt to the changes that the application or the environment has undergone.

Let us take an example to explain the above situation. Consider a Web application that is developed in .NET technology compatible to WinXP, Vista, and Win7 and browsers such as IE and Firefox. A version 5.0 of a test automation tool is used to automate a set of sanity test cases. A test automation framework is developed to ensure that as the application undergoes changes, there are re-usable functions that are developed so that with less effort you can address those changes. Most of the frameworks today address this basic need. However, consider a situation where a new version of OS, browser, and the test automation tool need to be addressed in the framework itself. It is not an easy situation or assumption to imagine or incorporate within a framework.

The next generation frameworks should address the best of the following:
* Coverage
* Business Process
* Extended Environment
* Multi OS
* Multi Test Automation Tools
* Best of Reporting and Analysis
* Defect Analysis
* Metric
* Compliance such as HIPPA or SOX
* Recovery Scenario
* Addressing Technology Challenges

Conclusion
Building a test automation framework needs skills such as designing, analytical, and a lot of foresight. While testers bring in a lot of value and insight into developing the framework, it is the designers and developers who eventually build it. Considering the development of framework as a typical development project goes a long way in getting the required RoI. Build the best software engineering practices while developing a framework. Some of the critical aspects, as the fig 1.0 depicts, make the framework dependable and robust in the long run.
Previous  article
Next article
 
Write your comment now

Email    Password: 
Don't have SiliconIndia account? Sign up    Forgot your password? Reset
  Cancel
Reader's comments(4)
1: From: Mrs. Mary David

This mail may be a surprise to you because you did not give me the permission to do so and neither do you know me but before I tell you about myself I want you to please forgive me for sending this mail without your permission. I am writing this letter in confidence believing that if it is the will of God for you to help me and my family, God almighty will bless and reward you abundantly. I need an honest and trust worthy person like you to entrust this huge transfer project unto.

My name is Mrs. Mary David, The Branch Manager of a Financial Institution. I am a Ghanaian married with 3 kids. I am writing to solicit your assistance in the transfer of US$7,500,000.00 Dollars. This fund is the excess of what my branch in which I am the manager made as profit last year (i.e. 2010 financial year). I have already submitted an annual report for that year to my head office in Accra-Ghana as I have watched with keen interest as they will never know of this excess. I have since, placed this amount of US$7,500,000.00 Dollars on an Escrow Coded account without a beneficiary (Anonymous) to avoid trace.

As an officer of the bank, I cannot be directly connected to this money thus I am impelled to request for your assistance to receive this money into your bank account on my behalf. I agree that 40% of this money will be for you as a foreign partner, in respect to the provision of a foreign account, and 60% would be for me. I do need to stress that there are practically no risk involved in this. It's going to be a bank-to-bank transfer. All I need from you is to stand as the original depositor of this fund so that the fund can be transferred to your account.

If you accept this offer, I will appreciate your timely response to me. This is why and only reason why I contacted you, I am willing to go into partnership investment with you owing to your wealth of experience, So please if you are interested to assist on this venture kindly contact me back for a brief discussion on how to proceed.

All correspondence must be via my private E-mail (dmary4love1@yahoo.fr) for obvious security reasons.

Best regards,
Mrs. Mary David.
Posted by: mary lovely david - Monday 26th, September 2011
2: Hi my dear,
My name is Mounace, i would like to establish a true relationship with you in one love. please send email to me at (mounace43@yahoo.com) i will reply to you with my picture and tell you more about myself. thanks and remain blessed for me,
Your new friend Mounace
Posted by: mounace love love - Thursday 09th, June 2011
3: HI CAN WE BE FRIENDS?

I am pamela by name,am friendly and good looking young girl , I want us to
be friends, for friendship can not be seen or even touched because true
friendship does not matter with anything but it must be felt within the
heart.
Wow, friends are like clothes, without them we feel naked! I guess I''m
right? Looking forward to a wonderful response from you.i saw your profle here (www.siliconindia.com

Contact me (singlepamela1@yahoo.in)so that i will send some of my pictures
to you so that we shall continue from there to know eachother.

I will be waiting to hear from you soonest.

pamela CoolSmile.
Posted by: pamela teamah littlepame - Wednesday 18th, May 2011
4: Hi Ramesh,

GREETINGS!

I am Vasumathy, who had a discussion with you on 9th November evening. It was a very nice discussion and I felt the comfort of discussing openly with you.

I got an insight after reading your article and become very much interested to know more on automation and hope to contribute from my part to build the next generation frameworks which you have addressed.

Looking forward to hear from you soon.

With Best Regards,
Vasumathy.M
Posted by: Vasumathy Meenakshi Sundaram - Thursday 11th, November 2010
More articles
by Kaushal Mehta - Founder & CEO, Motif Inc..
The retail industry is witnessing an increased migration of customers from traditional brick and mortar retail to E-commerce (online retail)...more>>
by Samir Shah - CEO, Zephyr .
You probably do because you are on the phone with them! For all of you working in some technical management capacity here in Silicon Valley,...more>>
by Raj Karamchedu - Chief Operating Officer, Legend Silicon .
These days are a mixed bag for me. Of late I have been considering "doing something bigger and better," in my life, perhaps seriously though...more>>
by Madhavi Vuppalapati - CEO of Prithvi Information Solutions .
IT Services Rise of Tier II companies The Indian IT outsourcing industry is going through very exciting phase in its business life...more>>
by Bhaskar Bakthavatsalu- Country Manager, India and SAARC of Check Point Software Technologies.
Data loss occurs every day through corporate email. In fact, given the sheer number of emails an organization sends every day, data loss inc...more>>