To date, the key tenet of data center architectures has been “design around failure.” Technological limitations on network systems forced data center managers to protect against failure by deploying redundant systems. If a single system failed, the redundant or stand-by system would ensure continued network operation. Creating this redundancy results in managers endinged up with multi-layer architectures in their data center, ensuing escalating capital and operational expenditures as well as management headaches.

The recent emergence of new technology that breaks through the performance and resiliency limitations of the past decade is now challenging the traditional architecture. This presents managers with innovative options for building data centers.

Data Centers at the Turn of the Century
Data center networks are comprised of servers or mainframes and storage devices, as well as switches and routers that provide connectivity within and between data centers or the WAN. It is the switches and routers that are at the heart of the data center. When one fails, the entire data center fails. At the turn of the century, the hardware and software instability of switches and routers forced data center managers to adopt multi-layer or redundant network architecture.

Earlier generations of switches and routers were not designed to provide the performance and resiliency features required to support large amounts of traffic or users, while maintaining high availability or uptime. At the system-level there were many single points of failure, such as non-redundant switch fabrics, active backplanes and insufficient power and cooling.

Critical switching and forwarding actions were performed by general purpose microprocessors under software control, which prevented scalability without compromising performance. The data path hardware, even under typical conditions, could not perform at the line rate and in some cases performance dropped by 50 percent. In this scenario, packets were forwarded to the control module for processing—an error-prone solution that frequently compromised network performance.

Beyond the hardware issues, software also presented an obstacle because of regular crashing and whole system being shutdown. Software typically lacked a number of protection features, like memory management and task separation while including many extraneous features intended for campus edge applications, not data centers. Including such unnecessary features led to corner cases and software instability. Additionally, data centers often utilized many non-standard features and protocols that burdened software images with error-prone code.

These limitations forced data center designers to adopt various strategies to ensure their networks were highly available for mission critical applications. System-level redundancy was widely employed and other connectivity limitations were enforced. While the cost and operational implications of such architectures were enormous, available technology limited the data center manager’s options.

Evolving to the Next-Generation Data Center
Today, the data center is transitioning through an accelerated evolution, enabled by advanced network components and driven by the growing demand for more bandwidth and computing resources. Data centers are increasingly deploying clusters of commodity blade servers to perform computing tasks and increase network utilization, creating more end points for network devices to interconnect. Compliments of advances in silicon technology and microprocessor capabilities, blade servers are cheap, extremely powerful and easy to maintain. More impressively, server clusters typically rely on familiar, inexpensive Ethernet technology for interconnection and use industry standard protocols for application layer connectivity.

As more servers are being deployed in the data center to accommodate increasingly complex computing needs, server capacity is also escalating. While just a few years ago only a limited number of servers could drive a full Gigabit of capacity, new servers have breached that barrier and some are now shipping 10 Gigabit Ethernet adapters. The traditional throughput bottlenecks within servers have been circumvented with new bus architectures and better driver architectures.

At the same time, application traffic is becoming less deterministic as web services as well as grid or cluster computing gain popularity. These service-oriented applications require more interprocess communication between discrete systems and each customer may choose to allocate bandwidth differently.

These changes in servers and applications accompany a parallel change in Ethernet and interconnection technology. As the amount of information being passed across the data center network increases, network managers are migrating from Fast to Gigabit and even 10 Gigabit Ethernet networks.

Building the Next-Generation Data Center
With increasingly powerful services and changing traffic patterns, the development of a scalable and flexible architecture, as illustrated in Figure 2, is critical for the next-generation data center. The most efficient and cost-effective next-generation data center architecture will provide high availability and resiliency while simultaneously delivering the high bandwidth and high density required for cluster computing.

Beyond the performance characteristics of the next-generation data center, cost and manageability are crucial factors. Architectures that require redundant systems or multiple switching layers to ensure non-blocking, line-rate connectivity will significantly increase capital costs of the network infrastructure. The cost of maintaining and manag- ing the network also increases exponentially as the number of switches and routers required in the network expands.

With these performance metrics in mind, managers that are building a next-generation data center need to closely examine the switches and routers that will interconnect all data center components. Issues to be considered include the following:
Port density and switching capacity: The port density and switching capacity of a system impacts not only the number of chassis required in the network but also how much traffic a single system can process. Higher switching capacities and a commensurate high port density enables data center managers to activate required protocols and features without impacting packet forwarding, ultimately translating into lower capital costs as well as significantly lower operational and management costs.

System Architecture: The system architecture determines how many ports can be active at full speed and how traffic is impacted when features are turned on. A genuine non-blocking architecture, required in high performance data centers, does not impede packet throughput, enabling data to flow at line rate. To achieve full non-blocking packet processing, a system must have a scalable backplane and switching fabric as well as non-blocking line cards.

Additionally, multiple CPUs further increase system performance by providing bulletproof task separation to keep the system operational under conditions where a subset of tasks or protocols misbehaves.

Built-in Redundancy: Next-generation data centers require a high performance, high availability architecture. To achieve that means eliminating all single points of failure. Any component that falters and brings down the system, such as control modules, power supplies, fans and switch fabrics, must be redundant. Additionally, failover to the standby module must occur with zero packets lost in the process.

Solid, Standards-based Software: A solid software implementation is as critical to system performance as good architecture is. Advanced memory management, task separation and fault isolation are integral to maintain high performance and availability. Additionally, standards-based software designed for specific applications such as the data center, is preferable to a software image burdened with unnecessary and proprietary protocols and features.

Scalability: As higher capacity servers become the status quo and traffic across data centers increases, managers will migrate to Gigabit and even 10 Gigabit Ethernet to accommodate traffic flow and ensure network availability.

In a switch or router, the port density and switching capacity, architecture, built-in redundancy and solid software implementations combine to ensure high availability and resiliency–the cornerstones of the next-generation data center network.

With technological advances that enable reliable processing of increased traffic on a single system, managers can evolve from the expensive multi-layer architecture that has dominated data centers since the 1990s. As is typical of survival of the fittest, the significant operational and management benefits of a simplified architecture will ensure that it thrives and comes to dominate data centers a great deal as the multi-layer architecture dominated the 90s. With the right equipment to implement this architecture, data center managers have the ability to build the next-generation data center today.

Sachi Sambandan is Vice President of Engineering at Force10 Networks. He holds 14 patents, an MSEE from the University of Maryland and an MBA from Santa Clara University.