This document briefly presents the capabilities of the Capstan tool. This tool, originally introduced by Cloudius Systems, is used for building of the virtual machine images with the OSv unikernel.
Capstan has been significantly updated within the MIKELANGELO project. The introduced updates provide extended functionality and vastly simplify the composition of the OSv virtual machine images.
To install Capstan, first create
$HOME/bin folder and add it to your
PATH environment variable. Download Capstan binary (Linux, Mac OS X) and place it in
$HOME/bin. Alternatively, you can use the following command to automatically select appropriate version and place it into your
curl https://raw.githubusercontent.com/mikelangelo-project/capstan/develop/scripts/download | bash
Before We Begin
This guide assumes that MIKELANGELO package repository is used. The original Capstan repository does not contain required application packages. Setting the correct package repository can be done in three different ways, depending on how you are going to use it:
- Capstan flag
-u. Every time capstan is invoked, you can pass the repository URL using the
-uflag, for example
capstan -u ‘https://mikelangelo-capstan.s3.amazonaws.com/ info. Use this option if you plan to switch between different repositories frequently.
CAPSTAN_REPO_URLenvironment variable. By setting this value to the MIKELANGELO package repository, capstan will always use it the command does not specify the
-uflag. Use this if you will normally use a single repository.
- Config file in your Capstan environment. Config file should be placed in
~/.capstan/config.yaml. Currently, the config file supports only the
repo_urloption. Setting this will ensure that a corresponding repository is used. To override it, use one of the aforementioned two options have been set as well.
Whichever option you choose, make sure to include the trailing slash. Future versions of Capstan are going to fix this, but until then, it won’t work if the slash is omitted.
This version of Capstan still relies on the base image. Contrary to the original version of Capstan, the base image now only contains the OSv loader, so it is only few megabyte image.
To import this base image, use the following command:
$ capstan pull mike/osv-loader
This will import the image into your local Capstan repository
In case you’d like to use a different base image, refer to the section at the end of this document.
Capstan tool provides a built-in help for all the available commands and subcommands. Simply add
-h and the tool will show the subcommands and configuration options available.
To get all top-level commands, run
$ capstan -h NAME: capstan - pack, ship, and run applications in light-weight VMs USAGE: capstan [global options] command [command options] [arguments...] VERSION: 'v0.1.8-52-g6282532' COMMANDS: config Capstan configuration info show disk image information import import an image to the local repository pull pull an image from a repository rmi delete an image from a repository run launch a VM. You may pass the image name as the first argument. build build an image compose compose the image from a folder or a file images, i list images search search a remote images instances, I list instances stop stop an instance delete delete an instance package package manipulation tools stack, openstack OpenStack manipulation tools GLOBAL OPTIONS: -u value remote repository URL (default: "https://s3.amazonaws.com/osv.capstan/") --help, -h show help --version, -v print the version
To show help only for the
package subcommand, use
capstan package -h:
$ capstan package -h NAME: capstan package - package manipulation tools USAGE: capstan package command [command options] [arguments...] COMMANDS: init initialise package structure build builds the package into a compressed file compose composes the package and all its dependencies into OSv image collect collects contents of this package and all required packages list lists the available packages import builds the package at the given path and imports it into a chosen repository search searches for packages in the remote repository (partial name matches are also supported) pull pulls the package from remote repository and imports it into local package storage OPTIONS: --help, -h show help
The rest of this document focuses only on the application package management that is conveniently provided by the <
capstan package command.
A package is any directory in your file system that contains a special package manifest file. The file should be formatted in YAML and stored in
meta/package.yaml file and folder, relative to the package root.
There are two ways to initialise a package:
- Manually edit the file in
- Let Capstan initialise the package.
Manual Initialisation of a Package
Since package manifest is a YAML file, it is fairly trivial to define basic information about the package. Using your favourite editor, create
meta/package.yaml in your package root directory and add the following information:
name: eu.mikelangelo-project.app.demo title: DEMO App author: lemmy
This is the smallest possible package manifest file. It defines an unique name of the application, the title of this application and the author. The suggested convention for naming packages and applications is the reversed domain notation as in the example above. The title and the author are currently only used in package listing.
Automatic Initialisation using Capstan
To initialise a package using Capstan, provide all the necessary information in a single command, like:
$ capstan package init --name "eu.mikelangelo-project.app.demo" --title "DEMO App" --author "lemmy"
This will create a meta subdirectory and
meta/package.yaml file with the given content.
Capstan package initialisation command allows you to optionally specify one or more required packages. These package will be included when the application will be composed into a VM. For example, to include the CLI (Command Line Interface) tool in our own application, initialise a package using
$ capstan package init --name "eu.mikelangelo-project.app.demo" --title "DEMO App" --author "lemmy" --require eu.mikelangelo-project.osv.cli
The same can be achieved by manually adding one or more required packages in
name: eu.mikelangelo-project.app.demo title: DEMO App author: lemmy require: - eu.mikelangelo-project.osv.cli
When applications are composed, the required packages are recursively inspected and the content of all of them is added to the application. For example, the
eu.mikelangelo-project.osv.cli package requires
eu.mikelangelo-project.osv.httpserver which consequently gets added to the target VM image as well.
A package may override the content of any of its required packages which allows users to customise or to reconfigure one of the base packages.
To list all packages available in your local repository, use
capstan package list command. For each of the packages, its name, description (title), version and time of creation will be displayed. Use this command to find out how to refer to required packages.
$ capstan package list Name Description Version Created eu.mikelangelo-project.app.hadoop-hdfs Hadoop HDFS 2.7.2 2016-05-30T14:35:23+02:00 eu.mikelangelo-project.app.node-4.4.5 NodeJS-4.4.5 4.4.5 2016-07-15T13:58:16+02:00 eu.mikelangelo-project.eu.test-pkg Test package 1.0 2016-01-19T20:46:41+01:00 eu.mikelangelo-project.ompi Open MPI 1.10 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.openfoam.core OpenFOAM Core 2.4.0 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.openfoam.simplefoam OpenFOAM simpleFoam 2.4.0 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.osv.bootstrap OSv Bootstrap v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.osv.cli OSv Command Line Interface v0.24-116-g73b38d8 2016-07-13T07:07:40+02:00 eu.mikelangelo-project.osv.cloud-init cloud-init v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.osv.httpserver OSv HTTP REST Server v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00 eu.mikelangelo-project.osv.java Java JRE 1.7.0 v0.24-116-g73b38d8 2016-07-08T10:04:48+02:00 eu.mikelangelo-project.osv.nfs OSv NFS Client Tools v0.24-116-g73b38d8 2016-07-08T10:03:21+02:00
Using Remote Package Repository
Initially, your package repository will be empty. To search for packages available in the specified remote repository use
capstan package search command. If executed without additional argument, it will list all available packages. Alternatively, the command accepts one additional argument that can be used to filter listed packages.
The following will show available packages that contain
node in their names:
$ capstan package search node Name Description Version Created eu.mikelangelo-project.app.hello-node NodeJS-4.4.5 4.4.5 0001-01-01T00:00:00Z eu.mikelangelo-project.app.node-4.4.5 NodeJS-4.4.5 4.4.5 0001-01-01T00:00:00Z
To import a package from a remote repository into your local one from where it can be used, use
capstan package pull command, for example:
$ capstan package pull eu.mikelangelo-project.app.node-4.4.5 Downloading eu.mikelangelo-project.app.node-4.4.5.yaml... 122 B / 122 B [============================================================================================================] 100.00 % 0 Downloading eu.mikelangelo-project.app.node-4.4.5.mpm... 19.15 MB / 19.15 MB [====================================================================================================] 100.00 % 15s
You can verify that the package has been successfully imported into your local repository by listing available packages again (
capstan package list).
Collecting Package Content
Collecting package content allows you to inspect the content of the application package exactly as it will be uploaded into target VM without actually uploading it. This can be useful for debugging application packages in cases when they are built one on the other and may alter each other’s (configuration) files.
The content is collected into a subdirectory named
mpm-pkg. It is not necessary to delete this directory as it is ignored by all package related commands.
To collect a package using capstan, simply execute the following command at the root of package:
$ capstan package collect $ tree mpm-pkg
Package composition takes the content of the package and all of its required packages and creates a new QCOW2 virtual machine image. Current version of the command supports the following additional configuration options:
--size, -s: specify the size of the target VM. Human readable representation can be used, for example 1G or 512M to request a 1 GB or 512 MB image.
--update: request an update of an existing VM. See below for more details
--run: specify the default run command to be used when starting a VM. It will be read by the OSv loader and executed immediately after the kernel is booted
--verbose: get detailed information about the files that are being uploaded onto the VM
To compose a VM image, simply execute:
$ capstan package compose [image-name]
image-name can be arbitrary name of the target image, for example
mike/hello. When the image is composed it is stored inside Capstan’s image repository, namely
Updating Virtual Machine
When making small changes to the application content, it is inefficient to compose a VM from scratch every time. Thus, the
--update option to the
capstan package compose allows you to request composition by uploading only the files that have been modified since the last run of this command. If the target image (
image-name) does not exist, it will be created and all files will be uploaded. However, if it already exists, only those files that have changed since the last virtual machine composition will be uploaded.
IMPORTANT: current version does not support removal of files or directories. If such an operation is necessary, –update should not be used. Furthermore, modifications are determined only an SHA1 hash of the files on the host composing the VM images. If any of the files have been changed on the VM itself, this will not be detected with this mechanism.
Running Applications with Capstan
Once we have a full VM stored in our local repository, we can launch it using
capstan run command. If we have composed an application with name
mike/hello, we can launch it with:
$ capstan run mike/hello
This will execute whatever the previous command was set to. In case you have not specified the
--run command when composing the image, this is not going to be what you wanted to do, as it will actually format VM’s entire root disk.
Instead, you should specify the run command either during image composition :
$ capstan package compose --run /usr/bin/myapp mike/hello $ capstan run mike/hello
$ capstan package compose mike/hello $ capstan run -e /usr/bin/myapp mike/hello
If you have included CLI into your application, you may launch it right away:
$ capstan run -e /cli/cli.so Updating image /home/lemmy/.capstan/repository/eu.mikelangelo-project.app.demo/eu.mikelangelo-project.app.demo.qemu... Setting cmdline: /tools/cpiod.so --prefix / Uploading files 287 / 287 [====================================================] 100.00 % 0 All files uploaded Created instance: eu.mikelangelo-project.app.demo Setting cmdline: /cli/cli.so OSv v0.24-78-g69bd35e eth0: 192.168.122.15 /# exit Goodbye
Capstan provides initial support for composing and running Java-based applications. It exploits the mechanism of the OSv build scripts and python modules that set a specific command line launching main class that serves as a proxy to the actual Java commands.
To enable Java application, you must add a dependency to
eu.mikelangelo-project.osv.java, for example:
name: eu.mikelangelo-project.app.hellojava title: Hello Java author: lemmy require: - eu.mikelangelo-project.osv.java
Additionally, you have to provide another manifest file configuring the Java application. This manifest file consists of the following options:
- main: fully classified name of the main class
- args: a list of command line args used by the application
- classpath: a list of paths where classes and other resources should be found
- vmargs: a list of JVM args (for example Xmx, Xms, …)
This manifest must be stored in
meta/java.yaml file. An example of a simple Java manifest is:
main: main.Hello classpath: - /
This will start class main.Hello. Classpath is set to the root because the main class is located in
A slightly more complex example of Java manifest (taken from our Hadoop HDFS application; note that classpath is trimmed).
main: org.apache.hadoop.hdfs.server.datanode.DataNode classpath: - /hdfs/etc/hadoop - /hdfs/share/hadoop/common/lib/commons-logging-1.1.3.jar - /hdfs/share/hadoop/common/lib/jersey-json-1.9.jar - ... vmargs: - Dproc_datanode - Xmx1000m - Djava.net.preferIPv4Stack=true - Dhadoop.log.dir=/hdfs/logs - Dhadoop.log.file=hadoop.log - Dhadoop.home.dir=/hdfs - Dhadoop.id.str=xlab - Dhadoop.root.logger=INFO,console - Djava.library.path=/hdfs/lib/native - Dhadoop.policy.file=hadoop-policy.xml - Djava.net.preferIPv4Stack=true - Dhadoop.security.logger=ERROR,RFAS - Dhadoop.security.logger=INFO,NullAppender
Replacing the OSv Base Image
Occasionally, it is required that a different base OSv loader image is used for composing virtual machine images. In this case, follow the instructions on how to build the OSv kernel. Once the build is finished, look for a file called
build/last/loader.img. You may copy this image into your Capstan repository for Capstan to use it as the base image for all your future
capstan package compose calls:
$ cp build/last/loader.img ~/.capstan/repository/mike/osv-loader/osv-loader.qemu
>Mind the target file name as it has to be exactly the same. Otherwise, the old base image will still be used.