Agile Methodology When Processes Clash with Principles
Date: Sunday , September 12, 2010
In about a decade since Agile methodology’s introduction, it has gained huge acceptance amongst software developers and IT managers. According to a recent Forrester survey, roughly 35 percent of its respondents used Agile methodology1. There is absolutely no doubt that Agile methodology has crossed the chasm.
While Agile brings in significant benefits and cost-savings to the consumer, it also comes with significant challenges. Agile methodology is built on several key principles that are fundamental to the methodology3. A lot of companies rush to adopt Agile methodology ignoring these key principles as well as the cultural and technical nuances associated with it. What results is not controlled chaos, but utter chaos and stress.
Cultural Challenge: One of the fundamental principles of Agile is the concept of self-organizing teams. Teams of software engineers are empowered in unprecedented ways, reducing the role of project managers (e.g. scrum master) to one of facilitator4. Motivation and team work are central tenets to the process.
When project managers that are used to command and control the style of management are entrusted an Agile project, they continue to push their heavy handed management style, which flies in the face of Agile’s self-organizing principle.
Similarly, software engineers who are used to strong oversight and management are tested by the sudden empowerment and freedom that comes with Agile. Lack of documentation, lack of fine-grained planning, and once a day scrum calls work when there is enough team-work and motivation. Agile is a mind-set and every team-member has an equal role to fulfill.
Contracts and Outsourced Software Development: Several outsourcing vendors enter into legal contracts with the customers where the payments are tied to traditional waterfall model. A sample contract could read, “25 percent payment for requirements completion, 25 percent for architecture completion, and so on.”
If typical Agile methodology were to be followed, then the above contract does not make any sense. For example, requirements gathering will be completed only during the last sprint by which time typically 70-80 percent of the entire project would have been completed. Thus, the delivery teams are caught in a bind to interpret the above payment clauses and simultaneously execute projects in pure Agile sense. This often times leads to chaos.
Collocation, Pair Programming and the Importance of Teamwork: It is no secret that Agile promotes team work and collocation5. Based on this tenacious assumption is the notion that working software is preferred over reams of documents. The implicit assumption made here is the fact that collocated engineers will talk to each other as and when required. Informal communication is the underlying principle, than formal meetings, specification documents, and SLAs.
Many companies overlook the importance of small teams and collocation and continue to plan large projects with diverse locations. The overhead of ‘educating’ team members in diverse locations on new changes (especially when there is sparse documentation) is a daunting task. Underestimating this aspect results in chaotic product development and causes a lot of stress.
Staffing and Agile Methodology: Agile is a front-loading process and mandates that all key contributers are assembled up-front. This is in sharp contrast to a waterfall model where staffing follows a normal curve. The following diagram illustrates the difference in staffing across Agile and traditional waterfall methodology.
Several project managers who come from traditional methodologies underestimate the significance of this and try to staff the team in a pure waterfall style. This results in project delays and sprints not meeting the goals.
The Definition of Shippable Product: The definition of task completion is clear in Agile6. Agile mandates that across every sprint, the team completes a 'shippable' product. Thus, from the very first sprint, customers can get a deployed version that they can actually make use of.
This is another area where there is continued misunderstanding. Engineers continue to claim features to be 'complete', whose meaning is not clearly established (that is, in true Agile sense). Project managers (read scrum masters) fall pray to this age-old conundrum and they continue to track feature completion without passing the key litmus test, namely shippability. ‘We can demonstrate feature X’ is very different from feature X ‘working’ at customer deployment and the customer is ‘educated’ to work with that feature.
Change Management and Sprint: It is true that Agile methodology can adapt to changes more easily than other methodologies. The cost of change curve diagram below actually indicates that Agile methodology has a flatter cost for changes compared to traditional waterfall methodology. But, it should be noted that the cost of change is not zero.
Several customers engage with vendors with the assertion, "We are still gathering requirements with our end-customers and thus we anticipate a lot of changes along the way. We prefer that you execute the project in Agile to incorporate these changes." While there is nothing wrong with this statement and this is perhaps a way of life in modern software development, the problem comes when the stakeholders and project managers push the change management to the extreme.
Agile recommends that within a sprint changes are well understood and nailed down7. It does not say that you can change requirements within a sprint. Agile recommends that if there are major revisions in requirements within a sprint, the sprint should be abandoned and a new one constituted.
A second pertinent point is that Agile promotes the notion of shippable product. So, every sprint is supposed to move the product towards the shippability continuum. Thus, changes to earlier sprints are perhaps as expensive in Agile as it is in the waterfall model (perhaps even more). Not understanding this crucial difference can make change management equally hard in Agile.
Conclusion: Agile methodology is here to stay, no doubt. However, as more companies rush to adopt Agile, they ignore the key principles and tenets that are behind the institutionalization of Agile methodology. This article highlights some of the common pitfalls that companies have to be aware of as they roll out Agile methodology within the organization. By far, the biggest challenge seems to be cultural, where the incumbent development methodologies clash with the Agile mindset. This problem is exacerbated when roll-outs happen without adequate training and when there is a lack of emphasis on shift in mindset.
The author is Associate Vice President, Photon