Real Software Architectures, not the Fanciful ones Work

To get success in a software project, Software requirements analysis is critical.

The new system must be as good in performance as the legacy system

Software requirements analysis is critical to the success of a software project.

Inappropriate gathering of software requirements is a major reason for projects not achieving the desired results. A META group research indicates that approximately 60-70 percent of IT project failures occur directly as a result of poor gathering of requirements, analysis, and management. Unclear requirements often increase the tendency amongst software designers and architects to over-engineer the solution. Here we will take a look at the ‘software over-engineering trap’ and the ways to avoid it.

In a recent recovery project, we were called to ‘fix’ the software developed by the customer’s own team. The testing team found that the software, though claimed to be highly scalable, was too slow and unresponsive, especially on smaller number of users. On investigating, we found that though the software architecture used the most fanciful technologies, implemented nifty design principles, and seemed 'perfect', it did not work when deployed. This is a classic example of what I call the 'software over-engineering trap'. Let us take a closer look at why we fall into this trap.

Top Reasons for Getting into the ‘Over-engineering' Trap

1. Lust for using latest technologies:

Being abreast of all latest technologies available in the market, it is natural for software architects to have an inclination to use them. But in the eagerness to use the latest technologies, one should not lose sight of whether it is actually required for the project at hand.

2. Lack of clarity on non-functional requirements:
Often the customer is not able to convey the requirements in the best possible way and may also be not sure on the levels of performance and scalability required. In such situations, the software architects may tend to take the safest route: create a software architecture that is straight from the textbook.

3. Limited understanding of technology at client-side:
Customers may not always have a tech-savvy member working directly with them to objectively review what the architects produce.

4. Background of the architects or designers:
Many times, architects play on their strength. They propose what they know the best.

All problems have a solution. We can avoid the trap, by enforcing some simple and effective steps. I call them 'powers' and I believe in them as if they were God sent. Nothing fanciful, but they really work.

The Power of Non-functional Requirements

In contrast to the functional requirements that determine how the system would behave, non-functional requirements (NFRs - that specify things like performance, scalability, and security requirements) are more intangible and ambiguity prone. But they are critical to the overall success of the software project and can dominate the design and architectural decision. Unfortunately, we tend not to give them their due importance because of the reasons discussed earlier. Both customers and analysts are very good at capturing functional requirements, but capturing non-functional requirements is not as easy. In many projects, NFRs are either not defined or defined very vaguely with fancy sounding open statements or defined very late in the game, almost after the software has been developed. Thanks to Mr. Murphy, the process of capturing non-functional requirements is paved with unusually high number of hurdles and speed-breakers. The end result is that we ignore the NFRs and move ahead in order to avoid delays in the project.

For example, if the customer has minimal scalability requirements, then one must carefully evaluate the need of service-orientedness (assuming no other pressing need for service orientation exists). In our recovery project example, we found that the scalability requirements were not captured properly. In reality, the customer had very minimal scalability requirements, but due to unclear requirements the designers ended up designing a highly scalable multi-layered service oriented system. As a result the system became too slow when deployed in production environments. Most architects know that each design decision comes with a trade-off and clarity in non-functional requirements help us make that decision meaningful to the customer situation.

Here are some guidelines that, when followed, can help us exploit the power of NFRs and make our architecture much closer to the real customer requirements:

1.Make NFRs non-negotiable:
Irrespective of how many hurdles you may face in gathering accurate NFRs, one rule I always follow is to define a clear ‘must-achieve’ milestone by which the customer must give you all non-functional and operational requirements before the team moves to the coding phase. No sign off on NFRs, no movement in the project.

2. Document NFRs in the form of data:

Document NFRs in the form of real numbers and not as open statements. For example, rather than saying, "The new system must be as good in performance as the legacy system," one must get customers to give data in the forms such as - 'the new system must load in less than 1 second' or 'the search page must provide all matching employee records in no more than 3 seconds'. Adopt this data-centric approach for scalability, installation, performance, security, and other non-functional requirements.

3.Focus on deployment:
Look at the way the customer is going to deploy the software after you develop. Would it be deployed on one machine, or multiple servers, or a cluster of machines? Get an agreement on the physical deployment of the software early in the game. I particularly recommend that each team member prints the physical deployment diagram and pins it up on his or her desk.

4. Data flow diagrams and threat modeling for security:
Look at the security aspect very carefully. What level of security is needed? Which portions of the system should be more secure than the others? What access levels and permissions you must provide to various roles that exist in the application? I found preparing a basic ‘data flow diagram’ very helpful in determining what type of security is needed at each level of the software. Using this simple data flow diagram, it’s very easy to figure out where a customer is exchanging personal information that requires higher security versus normal information. Many architects also perform what is known as 'threat modeling' based on such diagrams and other inputs before proceeding ahead with design.

The Power of Traceability Matrix

The capability to map the requirements increases the likelihood of delivering a software product that fulfills the customer’s needs. Using a 'traceability matrix' for this purpose is highly recommended. Most likely, you may have heard about this many times before. However, the simple submission here is that most companies either do a mere lip service to it or prepare this matrix in later phases of the lifecycle.

When used right from the start, this matrix helps us trace back the architectural considerations to real customer needs. Whether the customer asked you to prepare a traceability matrix or not, you should do it in your own interest. In this matrix, one would write down specific aspects of the architecture like componentization, service-oriented-ness, configurability, and dynamic discovery and map them to the customer requirement, especially the NFRs captured earlier. In the next column, one would also write what the drawback of using each of these architectural considerations is.

In the end, you would write down what type of hardware and deployment environment would be needed for this design to deliver results. Once complete, this helps us to re-consider our design in the event it is not mapping to any customer requirement, or is having more negatives than positives. Or, if the hardware needed is much more demanding than what the customer is going to use.

The Power of Demonstrations

As they say, ‘the proof of the pudding is in eating’. And the proof of your software architecture is after you see it running live to your total satisfaction! In addition to delivering software in optimally frequent intervals, you will particularly find the concept of creating ‘min-deployment lab’ very useful. This lab mimics the real deployment environment that customer has signed up for. It has the same firewall, DMZ, domain, and security settings as the real environment would. This environment is first used by developers and testers to deploy the system and perform the testing before releasing the software. Later, the same deployment lab can be used by the customer’s teams for phase-end demonstrations. A small investment in such labs gives great ROI, as compared with the cost of fixing failed projects. Your development teams would be surprised to see your own system not behaving as intended or even crashing when really deployed in this mini-lab. This would help them solve some critical deployment level defects early in the game. If setting up a lab at the development location is not always feasible, then the project teams must look at using the customer staging and testing environments during the development cycles.

In today’s environment, with the advancements in development, deployment, and virtualization tools, setting up this lab has become easier than ever before. Teams that follow agile software development methodologies can conduct phase-end demonstrations of functional and non-functional requirements in this mini-lab to avoid unpleasant surprises in the end.
So here’s to real IT that works for the customer, and for you!

Implementing these methodological and effective approaches to software requirements gathering helps the architects collect, segregate, prioritize, analyze, and document all the relevant informational and process needs for the application under design. This understanding of software requirements gathering will help give you a competitive edge over others.