Mikelangelo_2500x400 Releases

The MIKELANGELO project proudly presents the overall MIKELANGELO technology stack.

We plan to release technology updates in six month intervals – this means you know when to expect the new flavours of the MIKELANGELO components and when to schedule internal testing of our releases. As of today, two releases have been published:

You can find all sources at Github: https://github.com/mikelangelo-project. A list of all current repositories can be found at the bottom of this page.

MIKELANGELO Stack 2.0:

Over the last six months, MIKELANGELO has improved a large number of components to deliver the MIKELANGELO Stack 2.0. Please find all detailed improvements below.

  • sKVM has been improved in particular through IOcm, vRDMA, and SCAM.
    • IOcm has been extended by a dynamic IO core manager. The dynamic IO core manager samples I/O statistics, estimates the I/O pressure, and allocates the right amount of I/O cores.
    • A second prototype of vDRMA has been implemented. Compared to first prototype, the front-end driver has been improved. It now allows the use of a verbs-based API for improved performance.
    • SCAM has been extended by monitoring, profiling, and initial mitigation of attacks.
  • In the guest OS layer, OSv and Seastar have been extended.
    • OSv has been improved for better support of applications including additional system call and C-lib functions, new build environments, a better NFS client, isolated threads, partial NUMA support, cloud-init support, improved DHCP, file system performance, and assorted bug fixes.
    • Seastar improvements cover improved RPC, I/O scheduling, IOtune, CPU scheduling, monitoring, ext4 support, a log-structured memory allocator, and DPDK support.
  • The middleware extensions cover the MIKELANGELO Package Manager (MPM), Snap for monitoring, Scotty for system testing, vTorque for virtual HPC, and the MIKELANGELO Cloud Manager (MCM).
    • The MPM has been extended with a Docker container to sever the dependency to KVM during VM composition. Furthermore, MPM has been integrated with UniK for a better cloud integration.
    • Snap has been extended by ten new collector plugins and two new processor plugins. Most of those plugins relate directly to MIKELANGELO-specific components and some of them have already been further enhanced by the snap community.
    • Scotty has been developed from scratch to allow automated experimentation with virtual infrastructures (cloud and HPC). The initial version of Scotty is shipped with Puppet-based deployment scripts.
    • For HPC, vTorque has been developed as an extension to Torque to allow the deployment of virtual machines. The current release allows the transparent provisioning of Linux and OSv instances for virtual workloads.
    • For the cloud, MCM has been developed as an extension to OpenStack. MCM allows live-scheduling of resources in OpenStack. MCM integrates with OpenStack’s APIs and InfluxDB to collect monitoring data. The main goal of the MCM development is to foster the development of cloud resource management algorithms.

The individual components have been tested in two integrated testbeds. The first testbed uses OpenStack for a cloud deployment. This testbed integrates OSv, IOcm, Snap, MCM, Scotty, and MPM. The second testbed targets HPC and integrates OSv, IOcm, Snap, vRDMA, vTorque, and Scotty. The project’s use cases (link) have used the integrated testbeds to validate the components.

MIKELANGELO Stack 1.0:

This is the first release of the MIKELANGELO project and presents the first step towards the fully integrated and demonstrated MIKELANGELO technology stack. This release is the result of the ramping up period, followed up by intensive development in the first 18 project months. You can see the release as an indication of the directions the project is about to take (as we prepare the ground to introduce new packages, techniques, and more).

We prepared three extensive documents for everyone who wishes to fully check and understand the work on different components, how they fit together and how well they perform.

The first document is about the overall architecture of the MIKELANGELO technology stack and the individual components – The intermediate MIKELANGELO architecture. In short, skimming through the document will show you the components developed, the outlook for new features and also provide you with information on how to run them and what to expect.

Understanding the technology is a good way to start thinking if it is a good fit for your particular use case. In MIKELANGELO, we have four use-cases, ranging from HPC to Cloud – these are all described in the Use Cases Implementation Strategy. You will see what our use-cases are and how we tackled them.

Finally, as MIKELANGELO is about improving the performance in the virtual technology stack, increasing flexibility, unifying the HPC and Cloud software stack, we put these to the test – The Architecture and Implementation Evaluation. This document shows where we are at M18.

Repositories

Capstan is a tool for composing application packages into runnable OSv virtual machines. Capstan supports execution on top of local KVM and VirtualBoX, as well as OpenStack and Google Compute Engine clouds.

OpenStack Infrastructure is puppet-based automation of the OpenStack infrastructure roll-out. It allows you to seamlessly configure and deploy OpenStack on the fly.

vHPC Integration is a set of extensions to Torque open-source resource management system for deploying virtualised jobs on top of your HPC infrastructure. Upon resource reservation, done by Torque, requested jobs are deployed in dedicated virtual machines and monitored centrally.

Side Channel Attack Mitigation is a tool designed to prevent malicious virtual machine trying to exploit shared CPU cache to extract sensitive information from another virtual machine from the same physical host. It intends to provide a general-purpose solution preventing side channel attacks on the hypervisor layer as opposed to application layer, typically used.

IOcm Kernel is a set of patches on top of the Linux kernel which improves the I/O performance and scalability of virtual machines running on top of KVM.

Snap Collector Plugins have been designed and implemented to collect data from three different layers of the MIKELANGELO architecture: virtualisation (libvirt), guest operating system (OSv unikernel) and application layer (OpenFOAM).

Snap Tag Plugin allows tagging of data collected with any of the collector plugins with arbitrary user data set by the snap task itself.

Snap Anomaly Detection Plugin automatically reduces the resolution of telemetry that is statistically stable. When an anomaly is detected, the resolution is automatically increased.

Snap PostgreSQL Publisher Plugin is a demo plugin supporting PostgreSQL database as the backend for collected and processed data.

OpenFOAM Cloud is a demo application showcasing the MIKELANGELO technology stack using the Aerodynamics use case. It integrates Capstan to dynamically compose OSv-based virtual machines and deploy them onto OpenStack cloud infrastructure. Snap is used to collect data from different layers. The user interface is provided as an OpenStack Horizon dashboard allowing users to control and monitor OpenFOAM scientific experiments. This video shows the application in action.

MIKELANGELO Applications is a set of applications suitable for compilation and execution on OSv unikernel. These applications are mostly driven by use cases, however they serve as an invaluable resource for learning how to migrate applications to OSv.

Open MPI repositories provide an Open MPI Process Launch Module (PLM) extension to support parallelisation of OSv-based workers. Because OSv does not support processes this PLM exploits the capabilities of the OSv to run workers as threads. It furthermore replaces SSH with OSv’s REST API to configure individual workers. An example Open MPI application is also provided as a showcase.

OSv is a lightweight operating system designed specifically for virtualised environments. OSv is one of the most Linux-compliant unikernels supporting execution of mostly unmodified Linux applications.