Enterprise software, as designed today, is like a one-legged robot. Customers have to spend a lot of money, time and energy building crutches to help the robot limp along. The missing leg is functionality within the enterprise software to enable its own efficient and effective 24X7 operations, especially for large-scale deployments.

While features and functionality are important determinants of a customer’s selection of enterprise software, the operational budget needed to deploy, customize, and manage the software is fast becoming a key decision criterion. CIOs understand that the cost of purchasing enterprise software is a small part of the IT budget. They allocate a much larger portion of their budgets to ongoing operations—monitoring, problem management, change management, security management, configuration, patch management, and so on. This article can be applied to any enterprise software that has the potential for large-scale deployment and has the need for ongoing operational management—for example, Infrastructure applications (application servers, databases), CRM and ERP applications, security applications, etc.

Enterprise software has to be designed with functionality to simplify the management of the software by operations teams.

Enterprise software should be designed to not only service a customer’s customers, but also the “software operators” (IT departments or managed service providers) working for the customer. Having been responsible for development of software products at companies such as Oracle, and now being part of a managed services business responsible for 24X7 operations of some of the largest eCommerce sites in the U.S., I have come to the conclusion that enterprise software development teams also need:

To spend time on the phone with their support organizations to understand the challenges being experienced by their customers

To spend time implementing, customizing and upgrading their software at customer locations with their consulting organizations to understand how their software is being used and customized

Have members in their team who have operated and managed enterprise software 24X7 in large-scale deployments.

Consider an example of one of the largest online retailers in the US. The application infrastructure supporting its online presence consists of 200+ servers (webservers, application servers and databases) and 60+ network, storage and backup devices in production alone across two data centers. In addition, significant infrastructure exists for its staging and integration environments. This company decided to buy enterprise software from both mature and young companies. Technologies leveraged include (but are not limited to) Oracle database, ATG application server, Apache webserver, Veritas NetBackup and Cluster Server, Verity Search, Yantra’s Order Management, Solaris, HP/UX, Linux and EMC.

The software operators of this company have to address the following operational challenges:

Predicting the stability of their twice a week content changes developed by the marketing and pricing organizations, and bi-weekly code/customization releases developed by the their development team.

Ability to deploy changes such as updated pricing, promotions and even new code into production with tremendous flexibility in real-time (within a matter of hours) while minimizing the risk associated with these changes.

Identifying the source of problems in the production environment—is it a bug within the enterprise software or is it in the customer’s application specific code?

Ability to proactively identify problem trends in available capacity, scalability and stability, especially in the months leading into the Thanksgiving and Christmas.

Ability to understand the role of one enterprise software element within the context of other software elements that form the application infrastructure.

Visibility into the performance (speed and completion rate) of adding to shopping cart and checkout with a credit card transactions through the various layers of the application infrastructure—network, webservers, app servers, databases and hardware. Their business executives did not understand (or care) about their network having 5 9’s of availability, or databases being able to fail over, or application servers being clustered, when the above business transactions failed.

In the absence of each enterprise software element providing operational visibility into its own performance characteristics and manageability, and into its interactions with the other enterprise software elements, the customer’s development team blames any instability, outages and scalability issues upon an enterprise software element of choice. The perception of a problematic software purchase, negatively impacts the enterprise software’s post deployment success, ability to establish a referencable customer base and support costs—eventually hindering its success in the market.

Software operators focus on processes such as problem management, change management, and configuration management among others to enhance their own operational maturity that results in more efficient operations, greater customer satisfaction and reduced costs. The operational paradigms most important for software operators are:

Availability Management or Monitoring
Availability Management is the process of identifying any event that is not part of the standard operation of a service and that causes, or may cause, an interruption to, or a reduction in, the quality of that service.

Enterprise software should enable clear decision making with data and trends:

Easily plug into centralized monitoring infrastructures such as HP OpenView or BMC Patrol

Enable Service Level Agreements (SLAs) to be monitored, measured and reported upon

Problem Management
Problem Management identifies and resolves unplanned incidents that negatively affect a client system. The process begins with incident identification and ends with final resolution of the incident. The objectives of the problem management process are to prevent or minimize client site outages, resolve problems as quickly as possible, and identify and correct negative trends.

Enterprise software should:

Facilitate feedback loops between customer development teams (CDTs) and software operators by providing the latter with the ability to identify and address application infrastructure related problems, and call upon CDTs only for custom code-related issues. In addition, software operators should also have the ability to provide CDTs insight on what portions of code might be causing the problem. This will help CDTs to continually eliminate bugs within their code/customizations thus enhancing stability and quality, while reducing work effort associated with maintaining the application infrastructure.

Proactively identify potential problems to enable IT operations to resolve problems before they impact the business.

Visibility into the growth impact of the business—predict capacity and utilization trends.

Optimizing costs with the help of sizing tools that use business requirements as inputs.

Change Management
Change Management is an ongoing process of planning, communicating, coordinating, scheduling, monitoring, and controlling all related changes to the production environment. It is the process that protects the production environment from changes that are potentially disruptive or have unacceptable risk (business, stability, performance, security, etc.) associated with them.

Enterprise software should create the ability to change, including process change, with ease:

CDTs can define success criteria that can be leveraged by IT operations to assess the appropriate functioning of the software

Ensure that software has the ability to roll-back the last change by saving the previous state

Emergency Changes such as bug fixes, patches and changes driven due to business needs can be deployed in as flexible a manner as possible

Upgrades/patches can be deployed easily across a large installed base

The Impact of Operational Visibility
On being able to establish a feedback loop between development and operations, enabling operational visibility across the layers of enterprise software and focusing on the success of business transactions, one of the largest electronics manufacturers in the world experienced the following benefits:

A reduction in the number of problems in production by 45 percent

An associated reduction by 50 percent in the effort/hours to resolve problems

17 percent reduction in the average mean time to repair of problems.

25 percent reduction in the number of of changes being deployed into production

Increase in the availability of business transactions from 99 to 99.97 percent.

30-40 percent reduction in hiring, training, and process development costs

These benefits for the customer map to enhanced customer satisfaction, creation of a referencable customer base and up-sell opportunities for enterprise software companies.
Enterprise software companies should design their software to enhance the operational maturity of IT operations teams and increase operational visibility. Including functionality for the above operational paradigms will help enterprise software companies make their products stand out within a selection process, especially with C level decision makers. Let’s complete the robot with a second leg.

Siddhartha Agarwal is a Vice President at Totality Corporation, an MSP focused on remotely managing e-business applications and infrastructure for companies such as American Airlines, Best Buy and Sharper Image among others. Prior to this, he was co-founder and CTO of Contractor Online. He also helped found Teamscape, which was aquired by PeopleSoft. Prior to that, he was Director of Development at Oracle for Oracle's Internet Bill Presentment and Payment offering. Agarwal has an MS in Computer Science from Stanford and a BE from California Institute of Technology in Computer Science and Economics.