This blog post is one in the series of the use case descriptions. This use case runs usually in an HPC environment and complex enough to demonstrate, how to adapt an HPC application to the MIKELANGELO architecture and software stack.
The purpose of the Cancellous bone simulation is to understand the structure of the cancellous, or spongy, parts of the human bone, in order to efficiently and safely attach the required implants to it for the medical treatment of certain pathologies. Using this simulation, implants can be produced in a more accurate fashion, which increases production yield and at the same time, improves the healing time for bone implant surgeries and improve the lifespan of the implant as well. All these advances will improve at the end the patient’s quality of life. Cancellous bone simulations require a lot of computational resources (processors, memory, storage, network, etc.), however, typically hospitals cannot afford HPC systems for specific simulations. Hospitals run their simulations on HPC systems from third parties. This use case simulations run in the Supercomputing Center of Stuttgart University HLRS.
A small look inside the Cancellous Bone Simulation
The input to a Cancellous bone simulation is produced by computed tomography (CT) scanners which give a raw 3D output of the structure of the bones inside the human body. To actually use this data to help scientists and hospital staff, it is important to understand how the bones are structured inside. In particular, to understand the cancellous or spongy parts of the bones. These cancellous parts of the bones are where implants are positioned. However, attaching or inserting a new component inside (or outside) of the existing structure, it is possible to stress the bone in a way that it amplifies its weakness. This can lead to further damages, to the bone as well as the implant itself.
To prevent damage to the bone through implants, the structure and the density of the bone have to be determined. Through a finite element method other physical material properties like local elastic behavior, stiffness or material symmetries are calculated. These properties help with designing better implants and their fit to the patients. Most importantly, these measurements and simulation prevent bone damage from implants.
The figure on the left shows the typical progress of this simulation. First, we select a small part of the bone on its critical part, do the direct discretisation, transfer it to the continuum space and finally, perform the simulation in a continuum material model space. Even without going into specific details of the process, it is evident that this method gets extremely complex. A fine granularity is needed to actually help the medical personnel or the implants designer.
To illustrate the complexity of the problem, the resolution of the CT has to be 0.02 mm or even higher and therefore, even a small part of a single bone will result in a big data set. The simulation of this size is simply too complex to be computed on a single machine – it has to be performed in an HPC cluster. An illustration of the actual steps that need to be taken is provided in the figure below.
The role of HLRS
The Supercomputing Center of Stuttgart University, also called HLRS, is in charge of adapting the simulation program itself to the MIKELANGELO’s technology. Such technology will provide a faster execution and greater flexibility.
Flexibility is achieved through virtualization. Virtualization is a technique used in Cloud environments where a program, in this case a cancellous bone simulation, is executed in several virtual machines. These virtual machines abstract the program from the physical machines. These virtual machines are created using just a single image with the necessary software inside (operating system, required software packages, configuration files, etc.). Such image can be easily copied to any cluster, but opposite to the Cloud environment, the creation of virtual machines inside a HPC environment requires some adaptations. At the HLRS, developers are currently working to adapt Torque, an open-source job scheduler and resource manager, to support simulation executions on virtual machines inside an HPC cluster.
Virtualization, however, has a drawback that is the negative impact in the I/O performance. To overcome such issue in MIKELANGELO, an I/O optimized hypervisor, images for virtual machines based on the OSv operating system, created specifically for virtualization, and techniques like virtual RDMA, are used together to achieve the I/O performance that corresponds to a HPC environment.
Academic institutions put more focus on the research opportunities and the (academic) quality of the results than the actual return of investment. With the MIKELANGELO stack it is possible to use HPC in a “cloud-like” way, or use cloud environments with more performance (the cloud moves closer to traditional HPC). A pure HPC application will be adapted in order to support distributed execution in state-of-the-art clouds. This adaptation will increase the flexibility of the application drastically, as HPC and Clouds can be supported, which results in a huge variety of infrastructure providers that can be used to execute the simulation. In addition, with the help of MIKELANGELO, information on application execution will be captured in order to analyse the application and optimize it for further usage in clouds and cluster systems. In particular, different kinds of executions are expected, making use of virtual machines and different resource combinations, reaching from large VMs with 16 cores and 32GB RAM to small VMs that provide only 1 core and 1GB RAM. As a consequence, a deep understanding of the involved components will be generated that can be transferred in optimised implementations like the Cancellous bone simulation.
Benefits for Industry
In contrast to academia, the robustness, fast execution times, reliability of the environment and short waiting times for simulation starts are more important than optimizing the simulation to the last bit. Therefore, a distributed application, including the execution environment is highly beneficial. This increases amount of possible providers, where the simulation can run. As a consequence the waiting time decreases significantly and the results are delivered faster.
For example, hospitals can use that application in order to simulate cancellous bones for improving surgeries as well as healing. The simulation can be used by implant manufacturers that create the bone implants for future implantation as well.
With the help of the MIKELANGELO stack it is possible to run a clearly HPC-like application, the Cancellous Bones Simulation, in an HPC-Cloud. This will help to run this application on different hardware without adjustments to the software. MIKELANGELO will help to deploy applications (inside VM) faster and to almost any infrastructure. In addition this stack will help to minimize the overhead of virtualization that prevented VMs on HPC infrastructures.