215. Scientific computation in the cloud

This is a guest blog post by Lukas Svec who is working on high performance computing related problems in the University of Washington together with F. Vila, J. Gardner and J. Rehr. I thought that their work is rather cool and decided to learn more about it (whether we want it or not, cloud computing is the way of the future as Google proves very effectively so far). Please also consider this post as a kind of press release for the code that the group of John Rehr has developed. Dmitry.

Over the past several months, our group in the Department of Physics at the University of Washington has begun looking at ways to implement “cloud computation” schemes through Amazon Web Services (AWS). Our group specializes in scientific computation and has been developing the widely used FEFF codes for calculations of x-ray spectroscopy and electronic structure in molecules and solids. Our aim is to evaluate the usefulness of AWS elastic compute cloud, aka EC2, not only as a means to make our codes more accessible, but also to assess the viability of cloud computation for scientific calculation in general.

Cloud computation has become a hot computer science topic over the last few years, but it’s application to scientific computing has been relatively unexplored. However, our preliminary results demonstrate that the cloud-computing paradigm can indeed be extended to high performance scientific computation. Our findings raise the possibility of affordable, user-friendly access to high-end, remotely accessible, virtualized pools of computer clusters and storage. This means that computationally intensive calculations no longer require a group to purchase in-house computer banks with a high front-end cost, not to mention the overhead cost of maintaining the hardware. AWS also provides users with tools that remotely manage and create machine images. These machine images are customizable with virtually any Linux or Windows operating system. For the developer, this means the software can be configured and optimized on a machine image designed for a user to run. As an example, one machine image we made has FEFF compiled and optimized on a Fedora 8 32-bit image. Thus when users access this image, they can immediately start running an optimized version of the code on cloud computer clusters.

Our preliminary findings indicate that the AWS tools Amazon provides are relatively straightforward to use and can be programmed in most commonly used scripting languages. So, how far can we take these tools? After getting the command line FEFF8 code to run on our image, we then added JFEFF as well. JFEFF is a Java based graphical user interface to FEFF. However, running JFEFF remotely through X11 proved to be unsatisfactory. On standard network connections the lag times were significant enough to be annoying. Even on high-speed connections lag-times were noticeable. Thus the cloud is currently not well suited to remotely run graphical programs. On the other hand, the cloud is well suited for client-server model with either a graphical or command line program running on a local machine. Thus currently we are supplementing JFEFF with code to interface with FEFF located on the cloud.

Apart from developing ways to interact with the cloud, we are also exploring ways to make the cloud more attractive to the scientific community by parallelizing our images. Our current batch of scripts include seven simple-to-use commands that launch, connect to, transfer files between, list, and terminate virtual clusters. They are coded in Bash, but they can be implemented in other languages as well. Currently, the user follows a simple procedure for launching and connecting to the cluster, running FEFFMPI, and terminating the clusters after completion. All detailed control of the clusters is handled behind-the-scenes in the scripts. Such simplicity in using FEFFMPI was surprising. Indeed, a user does not need to know anything about MPI to use it effectively. At the moment not many FEFF users make use of the parallelized versions because it is both more difficult to use and requires more sophisticated hardware. Based on these findings, we believe that running on the cloud should make FEFFMPI more popular and accessible to the scientific community. A further implication is that many other scientific applications can also be ported to the cloud.