Open Menu Close Menu

University of Utah: C-SAFE Uses Linux HPCC in Fire Research

A high performance computing cluster (HPCC) is an increasingly popular method for delivering computational power to CPU-hungry scientific applications. A cluster consists of several commodity personal computer systems (desktop or rackmount) connected with a commodity or special-purpose network. The low cost of these systems makes an attractive option for a wide range of scientific applications.

In 1997, the University of Utah created an alliance with the Department of Energy (D'E) Accelerated Simulation and Computing Program (ASCP), formerly ASCI, to form the Center for the Simulation of Accidental Fires and Explosions (C-SAFE, C-SAFE focuses specifically on providing state-of-the-art, science-based tools for the numerical simulation of accidental fires and explosions, especially within the context of the handling and storage of highly flammable materials. The primary objective of C-SAFE is to provide a software system comprising a Problem Solving Environment in which fundamental chemistry and engineering physics are fully coupled with non-linear solvers, optimization, computational steering, visualization, and experimental data verification. The availability of simulations using this system will help to better evaluate the risks and safety issues associated with fires and explosions. Our goal is to integrate and deliver a system that is validated and documented for practical application to accidents involving both hydrocarbon and energetic materials. Efforts of this nature require expertise from a wide variety of academic disciplines.

Additional Resources
Simulation of an explosive in a fire involves computing thermodynamics, fluid mechanics, chemical reactions, and other physical processes at each of millions to billions of points in the simulation domain. These scenarios require significant computing resources to perform accurately. Simulations performed to date employed hundreds to thousands of processors for days at a time to simulate portions of this problem. Fortunately, we were also granted time on the D'E ASCP computers at Los Alamos, Lawrence Livermore and Sandia National Laboratories. These machines are some of the fastest in the world, and we received considerable allocations of CPU time as a part of the C-SAFE grant. However, sometimes these resources are not enough.

Recently, the University of Utah purchased a Linux cluster that will be used to augment the resources provided to us through the D'E. The cluster will be used for development and debugging of the simulation, parameter studies, and small- to medium-scale simulations. Large-scale computations will still require use of the larger D'E ASCP computing resources. Unlike the shared D'E resources, the cluster will be largely devoted to C-SAFE research 365 days a year.

This cluster consists of 128 Dell PowerEdge 2550 servers, each containing two Pentium 4 Xeon processors. The entire cluster contains 256 Gigabytes of main memory and over nine Terabytes of disk space. Each server (node) runs the Linux operating system, and is networked with a Gigabit Ethernet (currently being installed). This is a class of system often referred to as a Beowulf cluster, named after one of the first systems to demonstrate this concept. The cluster is listed on the November 2002 Top 500 list ( as the 89th fastest computer in the world.

Linux Advantage
C-SAFE chose to implement a Linux cluster for one primary reason: We could not afford a traditional supercomputer of the class and scale of those purchased by the D'E for the ASCP program. The cluster solution is based on commodity PC technology; each node is very similar to a personal computer. This leverages the mass-production of personal computers to provide a price/performance ratio unattainable by most other technologies.

Other advantages of this approach include leveraging a growing familiarity with the Linux operating system, and of course the fact that it is considered an “in vogue” thing to do in Computer Science circles. We also intend to utilize the cluster for real-time interaction with these long-running simulations, something that would be difficult to do on a larger shared resource.

However, this power d'es not come without caveat. First, the application must be able to perform well in this environment. A Linux cluster has communication latencies (both hardware and software) that are quite large relative to a traditional supercomputer. The C-SAFE software employs a novel approach to parallelism that helps mitigate the effects of these latencies, and we performed preliminary benchmarks that gave us confidence that we could achieve good performance on a machine of this size. Second, the installation and setup of a cluster can be quite complex and time-consuming.

Fortunately, Dell provided the physical installation of our cluster, which saved considerable time. Installation of operating systems on each of the 128 nodes, as well as installation and setup of additional HPC software, consumed considerable additional time.

Numerous companies, both large and small, offer computational clusters of this type. The University of Utah chose Dell to implement our cluster because of a competitive price/performance ratio, company reputation, and features of the PowerEdge 2550. Dell provided physical installation and setup of the cluster, which immensely reduced the work required to deploy this machine. The cluster fills seven standard racks, stands about six feet tall, four feet deep and 14 feet long. It produces a considerable amount of heat and fan noise.

Lessons Learned
Those that are considering a large compute cluster should keep in mind several lessons that we have learned along the way (either from others or from the school of hard knocks).

First, start small; we leveraged expertise from building smaller clusters before we considered a project of this size. Second, do not underestimate the time involved in physical installation and software setup of a machine of this nature. Depending on the complexity of the required environment, this can take weeks to months. Third, planning is critical. Considerations must be taken for the power consumption, heat production and physical footprint of the machine.

Finally, the biggest lesson is that it can actually work; we are performing simulations that required computers that just five years ago cost nearly a factor of 100 more. This gives us hope that our largest simulations may be achievable on a relatively common platform in the not-too-distant future. Early access to this magnitude of resources from the D'E was critical to the development of these simulations. Once the hurdles are cleared, a high-performance cluster can provide a dramatic quantity of compute cycles for complex scientific computations such as those performed for C-SAFE research.

For information contact Steven Parker, Research Assistant Professor, University of Utah, at [email protected].

comments powered by Disqus