Virtual Facility Simulation Cuts Data Center Cooling Costs

19 May, 2010 By: Rob Aldrich

Cisco Systems saves an estimated $120,000 per year by redesigning exhaust circulation using Future Facilities' 3D modeling technology.

Cisco Systems, an Internet networking company that employs close to 70,000 people, is taking steps to reduce its resource usage. The company is currently aiming for a 2012 carbon footprint that will be 25% smaller than it was in 2007. Given that almost 80 percent of Cisco's footprint is generated by labs and data centers, the solutions developed to support an in-house reduction will also have relevance to the company's customers in support of a green agenda.

Cisco has several teams focusing on external energy-efficient solutions development. The Data Center Advanced Services group helps users determine the capacity, density, and efficiency requirements of improvements to the data center architecture. This group utilizes a standard methodology focused on quantifying power and cooling usage; establishing an efficiency benchmark for the current data center power and cooling design; establishing operative efficiency benchmarks across compute, network, and storage systems; and designing and implementing more efficient data center architectures.

Setting a Green Example

We demonstrated this method on one of our own data centers, located in our San Jose campus. Our plan for this data center was to replace volume servers with blade servers, providing roughly 23 times the computing capacity in the same space. Blade servers, however, require much more power — typically 20 kilowatts per rack, compared with 5 kilowatts per rack for volume servers.

The data center has been in operation since 1999, with limited considerations for efficient operations. It occupies approximately 7,000 square feet and is filled with 3,202 units of IT equipment, drawing 770 kW. There is 1 MW of total power available and 820 kW of cooling capacity. The energy bill for the facility was originally $1.4 million per year, comprising $660,000 per year in cooling energy costs and $707,000 per year in IT equipment energy costs. Controlling temperature and airflow in such a facility is essential because tightly packed servers generate a great deal of heat, which can impair their reliability and durability.

A team led by Cisco DCSTG (Data Center Switching Technology Group) Engineering Manager Chris Noland employed two techniques to improve energy efficiency in the data center. This first was a commonly used set of best practices that included blanking panels (which fill unused positions in the server rack) and plastic curtains to prevent mixing of the cooled air supply and the warmed air exiting the servers. Noland's team noted two major limitations of this approach: Firstly, best practices offer no prediction of outcome, and Noland wanted a return on investment estimate in advance to justify the required expenditure. Secondly, best practices are designed to address room-level efficiency problems. In most data centers, efficiency problems are as likely to be caused by thermal incompatibilities between IT equipment and cabinets as they are by flawed room designs.

The second technique the team used was the Virtual Facility (VF) simulation-based approach. The VF, a detailed 3D model created with Future Facilities' 6SigmaDC software tools, can provide data to manage the resiliency and efficiency of a facility throughout its life cycle, from initial design to daily operation. The VF simulates the power usage and cooling behavior of the data center, including the thermal interactions among the room infrastructure, cooling system, cabinets, and individual units of IT equipment.

Building a 3D Model

Noland's team gathered all of the available information on the data center, including floor plans and asset inventory lists for the IT infrastructure, and entered it into 6SigmaDC. The software provides libraries that include the most commonly used data center infrastructure components and computing equipment. Users can simply drag relevant objects from a library into the VF.

An example of a simple Virtual Facility (VF) created with Future Facilities' 6SigmaDC software.

In comparison with general-purpose computational fluid dynamics modeling, the VF simplifies the modeling process, while at the same time increasing the level of detail at which the data center can be represented. This makes it possible to provide a modeling capability tailored to the needs of owner-operators such as IT managers and facilities managers without a mechanical systems background. The computational fluid dynamics solver is designed to handle thousands of individual objects, such as server blades, each with their own individual thermal and airflow characteristics.

Visualizing a Better Design

The first benefit of the VF is simple visualization — providing everything needed by mechanical engineers, electrical engineers, and facility managers in a single view that they could access from anywhere in the world. This helped project participants understand each other's constraints, and it also saved money by eliminating the need to visit the site in person.

An isometric view of the Cisco data center.

The VF enabled us to evaluate a range of options to determine what the final design should look like. We studied three areas: the thermal removal capabilities of the computer room air-handler (CRAH) equipment, the supply and distribution airflow, and the return airflow. The VF allowed us to examine all three elements while modeling changes to both the facilities and IT infrastructure.

Each segment of the cooling path has a specific design goal and an associated set of design options. The design objective for the CRAH equipment was to meet the temperature and airflow requirement for each floor grille by finding the right combination of floor height, air handler selection, and placement of air handlers, floor grilles, and under-floor objects.

The design objectives for the supply and distribution airflow and the return airflow were to meet the intake temperature and airflow requirements for each unit of IT equipment. Additional requirements to minimize bypass and recirculating airflow were imposed to increase cooling system efficiency. These requirements were met by finding a combination of cabinet design, cabinet placement, cable routing, and slot location that was compatible with the thermal characteristics of the IT equipment to be installed.

Performing Analysis with the Virtual Facility

The VF simulation predicted the airflow and thermal environment, enabling optimization of the floor void depth; pressure distribution within the void, including cabling and obstructions; flow rates through the grilles and choice of grille damping; power load distribution throughout the facility; and other aspects of the data center.

The VF analysis clearly showed that exhaust recirculation within cabinets was the pressing problem, which was causing high IT equipment inlet temperatures and the need to overcool the chilled water system. Noland found that blanking and containment curtains actually increased inlet temperatures for some units of equipment in the data center. The VF was used to guide the tactical placement of floor grilles and blanking panels that eliminated the worst of the cabinet and room-level hot spots.

This image shows hot exhaust air flowing directly into the intake of a unit in the adjacent cabinet.


This image shows a reduced inlet temperature for the overheating unit due to moving the floor grille under the adjacent cabinet.

The resulting 8ºF increase in chilled water set point provided a 30% reduction in power required for cooling, resulting in a savings of $120,000 per year. Because there was no increase in inlet air temperature, there was no corresponding decrease in equipment resilience. Based on the success of this application, Cisco Systems has adopted the VF approach to maximize server resilience and efficiency over time, in both its own data centers and those of its customers.


About the Author: Rob Aldrich

Add comment