DOE takes another step toward exascale computing

The US Department of Energy has awarded a $325 million contract to IBM to build two high-performance computers that are likely to be the world’s most powerful when they are completed in 2017. Both machines will have a peak processing speed of at least 150 petaflops—five to seven times faster than the current top US computers. (A petaflop is 10¹⁵ floating point calculations per second.) They will be installed at national laboratories, one at Oak Ridge and one at Lawrence Livermore.

A third contract, for a supercomputer with comparable performance, is expected to be awarded within months to a different vendor. That machine will be installed at Argonne National Laboratory. In addition to the IBM award, DOE has announced a $100 million contract to five US companies to develop technologies that will be needed for affordable and energy-efficient machines as DOE moves toward its goal of exascale (1000 petaflops) computing within the next decade.

The Oak Ridge system, named Summit, will be five times as powerful as the lab’s Titan machine, currently the second fastest in the world with a peak performance of 27 petaflops. The ratings are based on the Linpack performance benchmark, which measures how quickly the machines solve a dense set of linear equations. China’s Tianhe-2, at 55 petaflops, is the world’s fastest computer. “Once again the world’s fastest computer will be in the United States and once again it will be at Oak Ridge,” said Senator Lamar Alexander (R-TN) at a 14 November press conference.

The Livermore computer, dubbed Sierra, will offer seven times the performance of that lab’s Sequoia, the world’s third most powerful, rated at 20 petaflops. Although Sierra will be used mainly for classified nuclear weapons simulations, it will also carry out some unclassified research, says Livermore director William Goldstein. Los Alamos National Laboratory, the other nuclear weapons design lab, will begin installing its new high-performance Cray computer, Trinity, this year.

More realistic simulations

The new Oak Ridge and Argonne supercomputers will be open to all users, including industry, says Barbara Helland, director of facilities utilization in the DOE Office of Science’s advanced scientific computing research division. Titan and Argonne’s Mira, the current leading computers, are oversubscribed by a factor of three to four, and Helland expects that will also be the case with the new Oak Ridge and Argonne systems. The number of users has steadily increased over the past five years as industry looks to replace expensive experiments and prototyping with less costly simulations, Helland says.

Summit will offer the ability to run simulations with greater detail, says Helland. “If you want to do groundwater or carbon sequestration, you are talking about going through a material that is very heterogeneous, and you can put more realism into what that rock does. More power makes [the simulation] more realistic,” she says. “In many cases you try to make it more realistic by picking the right parameters. We can do some uncertainty quantification to adjust the parameters during the calculations on the new machine” and improve confidence in the simulation.

Research performed on Titan has included simulations of the reactions inside fusion tokamaks and the design of advanced nuclear reactors. One climate model used 22 000 years of geological data from after the last glaciation to gain understanding of the role of carbon in glaciation. Other DOE-sponsored research included modeling more efficient photovoltaics and more efficient use of LEDs, Helland says.

Industrial users of the Argonne and Oak Ridge facilities include Procter and Gamble, which modeled the ways lotions interact with skin, and Pratt and Whitney, which used the machines in the development of more energy efficient jet engines.

New algorithms needed

Titan, in addition to modeling combustion and turbulence, is being used by GE to develop anti-icing surfaces. Masako Yamada, a GE researcher, says that wind turbine blades are one potential application for those surfaces. Her research using molecular dynamics seeks to model the behavior of each individual molecule in a water droplet. Yamada says Titan is capable of modeling about 1 million molecules, still a far cry from what’s needed.

“We are many, many orders of magnitude away from representing a true physical science system of the type we’d like to do,” Yamada says. She notes that there are 10²³ molecules in a mole of water. “However, we are far, far ahead of where we used to be even 10 years ago. And to be able to observe the freezing of water in a million molecules is a great advance that was not accessible to us 10 years ago.” Even exascale computers won’t be powerful enough to model the required number of molecules, she says, so new algorithms and scientific methods also are needed.

Thomas Schulthess, director of the Swiss National Supercomputing Center and a computational physics professor at ETH Zürich’s Institute for Theoretical Physics, has a part-time appointment at Oak Ridge and is using Titan to run quantum Monte Carlo simulations of strongly correlated systems. Recent advances in algorithms have made it possible to study the superconducting transition in cuprate high-temperature superconductors. “We are now working on Titan to extract the phase diagram of this model,” he says.

Summit “will have a lot more memory so we can think outside the box on how differently to use the machine. We will gain a lot more flexibility. I think the things we do today on Titan will transfer relatively seamlessly to [Summit],” Schulthess says. “I would say it’s a profound step forward towards an exascale machine.”

Oak Ridge has the existing infrastructure to support the substantial electrical and cooling requirements of supercomputers, says lab director Thom Mason. The IBM machines will draw up to 13 MW, only about 10% more power than the current leaders. That makes the new machines five times as energy efficient as the current top performers.

High-performance machines have limited lifetimes. Fully commissioned in June 2013, “Titan will be on its last legs” by the time Summit is up and running, and it will likely be closed within a year, says Helland. Processors become unstable and unreliable, she explains, and the maintenance required to keep them going becomes cost ineffective.