This document briefly presents the capabilities of the Capstan tool. This tool, originally introduced by Cloudius Systems, is used for building of the virtual machine images with the OSv unikernel.
Capstan has been significantly updated within the MIKELANGELO project. The introduced updates provide extended functionality and vastly simplify the composition of the OSv virtual machine images.

For further inquiries, use the contact form or create Github Issue.

Installation

To install Capstan, first create $HOME/bin folder and add it to your PATH environment variable. Download Capstan binary (Linux, Mac OS X) and place it in $HOME/bin. Alternatively, you can use the following command to automatically select appropriate version and place it into your $HOME/bin:

curl https://raw.githubusercontent.com/mikelangelo-project/capstan/develop/scripts/download | bash

 

Before We Begin

This guide assumes that MIKELANGELO package repository is used. The original Capstan repository does not contain required application packages. Setting the correct package repository can be done in three different ways, depending on how you are going to use it:

  • Capstan flag -u. Every time capstan is invoked, you can pass the repository URL using the -u flag, for example capstan -u ‘https://mikelangelo-capstan.s3.amazonaws.com/ info. Use this option if you plan to switch between different repositories frequently.
  • CAPSTAN_REPO_URL environment variable. By setting this value to the MIKELANGELO package repository, capstan will always use it the command does not specify the -u flag. Use this if you will normally use a single repository.
  • Config file in your Capstan environment. Config file should be placed in ~/.capstan/config.yaml. Currently, the config file supports only the repo_url option. Setting this will ensure that a corresponding repository is used. To override it, use one of the aforementioned two options have been set as well.

Whichever option you choose, make sure to include the trailing slash. Future versions of Capstan are going to fix this, but until then, it won’t work if the slash is omitted.

Base Images

This version of Capstan still relies on the base image. Contrary to the original version of Capstan, the base image now only contains the OSv loader, so it is only few megabyte image.

To import this base image, use the following command:

$ capstan pull mike/osv-loader

This will import the image into your local Capstan repository ~/.capstan/repository/mike/osv-loader.

In case you’d like to use a different base image, refer to the section at the end of this document.

Getting Help

Capstan tool provides a built-in help for all the available commands and subcommands. Simply add -h and the tool will show the subcommands and configuration options available.

To get all top-level commands, run capstan -h:

$ capstan -h
NAME:
   capstan - pack, ship, and run applications in light-weight VMs
USAGE:
   capstan [global options] command [command options] [arguments...]
VERSION:
   'v0.1.8-52-g6282532'
COMMANDS:
     config            Capstan configuration
     info              show disk image information
     import            import an image to the local repository
     pull              pull an image from a repository
     rmi               delete an image from a repository
     run               launch a VM. You may pass the image name as the first argument.
     build             build an image
     compose           compose the image from a folder or a file
     images, i         list images
     search            search a remote images
     instances, I      list instances
     stop              stop an instance
     delete            delete an instance
     package           package manipulation tools
     stack, openstack  OpenStack manipulation tools
GLOBAL OPTIONS:
   -u value       remote repository URL (default: "https://s3.amazonaws.com/osv.capstan/")
   --help, -h     show help
   --version, -v  print the version

To show help only for the package subcommand, use capstan package -h:

$ capstan package -h
NAME:
   capstan package - package manipulation tools
USAGE:
   capstan package command [command options] [arguments...]
COMMANDS:
     init     initialise package structure
     build    builds the package into a compressed file
     compose  composes the package and all its dependencies into OSv image
     collect  collects contents of this package and all required packages
     list     lists the available packages
     import   builds the package at the given path and imports it into a chosen repository
     search   searches for packages in the remote repository (partial name matches are also supported)
     pull     pulls the package from remote repository and imports it into local package storage
OPTIONS:
   --help, -h  show help

The rest of this document focuses only on the application package management that is conveniently provided by the <capstan package command.

Package Initialisation

A package is any directory in your file system that contains a special package manifest file. The file should be formatted in YAML and stored in meta/package.yaml file and folder, relative to the package root.
There are two ways to initialise a package:

  • Manually edit the file in meta/package.yaml.
  • Let Capstan initialise the package.

Manual Initialisation of a Package

Since package manifest is a YAML file, it is fairly trivial to define basic information about the package. Using your favourite editor, create meta/package.yaml in your package root directory and add the following information:

This is the smallest possible package manifest file. It defines an unique name of the application, the title of this application and the author. The suggested convention for naming packages and applications is the reversed domain notation as in the example above. The title and the author are currently only used in package listing.

Automatic Initialisation using Capstan

To initialise a package using Capstan, provide all the necessary information in a single command, like:

$ capstan package init --name "eu.mikelangelo-project.app.demo" --title "DEMO App" --author "lemmy"

This will create a meta subdirectory and meta/package.yaml file with the given content.

Dependencies

Capstan package initialisation command allows you to optionally specify one or more required packages. These package will be included when the application will be composed into a VM. For example, to include the CLI (Command Line Interface) tool in our own application, initialise a package using --require option:

$ capstan package init --name "eu.mikelangelo-project.app.demo" --title "DEMO App" --author "lemmy" --require eu.mikelangelo-project.osv.cli

The same can be achieved by manually adding one or more required packages in meta/package.yaml:

When applications are composed, the required packages are recursively inspected and the content of all of them is added to the application. For example, the eu.mikelangelo-project.osv.cli package requires eu.mikelangelo-project.osv.httpserver which consequently gets added to the target VM image as well.
A package may override the content of any of its required packages which allows users to customise or to reconfigure one of the base packages.

Listing Package

To list all packages available in your local repository, use capstan package list command. For each of the packages, its name, description (title), version and time of creation will be displayed. Use this command to find out how to refer to required packages.

$ capstan package list
Name Description Version Created
eu.mikelangelo-project.app.hadoop-hdfs Hadoop HDFS 2.7.2 2016-05-30T14:35:23+02:00
eu.mikelangelo-project.app.node-4.4.5 NodeJS-4.4.5 4.4.5 2016-07-15T13:58:16+02:00
eu.mikelangelo-project.eu.test-pkg Test package 1.0 2016-01-19T20:46:41+01:00
eu.mikelangelo-project.ompi Open MPI 1.10 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.openfoam.core OpenFOAM Core 2.4.0 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.openfoam.simplefoam OpenFOAM simpleFoam 2.4.0 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.osv.bootstrap OSv Bootstrap v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.osv.cli OSv Command Line Interface v0.24-116-g73b38d8 2016-07-13T07:07:40+02:00
eu.mikelangelo-project.osv.cloud-init cloud-init v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.osv.httpserver OSv HTTP REST Server v0.24-116-g73b38d8 2016-06-15T11:45:30+02:00
eu.mikelangelo-project.osv.java Java JRE 1.7.0 v0.24-116-g73b38d8 2016-07-08T10:04:48+02:00
eu.mikelangelo-project.osv.nfs OSv NFS Client Tools v0.24-116-g73b38d8 2016-07-08T10:03:21+02:00

 

Using Remote Package Repository

Initially, your package repository will be empty. To search for packages available in the specified remote repository use capstan package search command. If executed without additional argument, it will list all available packages. Alternatively, the command accepts one additional argument that can be used to filter listed packages.
The following will show available packages that contain node in their names:

$ capstan package search node
Name                                               Description                                        Version                   Created
eu.mikelangelo-project.app.hello-node              NodeJS-4.4.5                                       4.4.5                     0001-01-01T00:00:00Z
eu.mikelangelo-project.app.node-4.4.5              NodeJS-4.4.5                                       4.4.5                     0001-01-01T00:00:00Z

To import a package from a remote repository into your local one from where it can be used, use capstan package pull command, for example:

$ capstan package pull eu.mikelangelo-project.app.node-4.4.5
Downloading eu.mikelangelo-project.app.node-4.4.5.yaml...
122 B / 122 B [============================================================================================================] 100.00 % 0
Downloading eu.mikelangelo-project.app.node-4.4.5.mpm...
19.15 MB / 19.15 MB [====================================================================================================] 100.00 % 15s

You can verify that the package has been successfully imported into your local repository by listing available packages again (capstan package list).

Collecting Package Content

Collecting package content allows you to inspect the content of the application package exactly as it will be uploaded into target VM without actually uploading it. This can be useful for debugging application packages in cases when they are built one on the other and may alter each other’s (configuration) files.

The content is collected into a subdirectory named mpm-pkg. It is not necessary to delete this directory as it is ignored by all package related commands.
To collect a package using capstan, simply execute the following command at the root of package:

$ capstan package collect
$ tree mpm-pkg

 

Package Composition

Package composition takes the content of the package and all of its required packages and creates a new QCOW2 virtual machine image. Current version of the command supports the following additional configuration options:

  • --size, -s: specify the size of the target VM. Human readable representation can be used, for example 1G or 512M to request a 1 GB or 512 MB image.
  • --update: request an update of an existing VM. See below for more details
  • --run: specify the default run command to be used when starting a VM. It will be read by the OSv loader and executed immediately after the kernel is booted
  • --verbose: get detailed information about the files that are being uploaded onto the VM

To compose a VM image, simply execute:

$ capstan package compose [image-name]

The image-name can be arbitrary name of the target image, for example mike/hello. When the image is composed it is stored inside Capstan’s image repository, namely ~/.capstan/repository/[image-name].

Updating Virtual Machine

When making small changes to the application content, it is inefficient to compose a VM from scratch every time. Thus, the --update option to the capstan package compose allows you to request composition by uploading only the files that have been modified since the last run of this command. If the target image (image-name) does not exist, it will be created and all files will be uploaded. However, if it already exists, only those files that have changed since the last virtual machine composition will be uploaded.

IMPORTANT: current version does not support removal of files or directories. If such an operation is necessary, -update should not be used. Furthermore, modifications are determined only an SHA1 hash of the files on the host composing the VM images. If any of the files have been changed on the VM itself, this will not be detected with this mechanism.

Running Applications with Capstan

Once we have a full VM stored in our local repository, we can launch it using capstan run command. If we have composed an application with name mike/hello, we can launch it with:

$ capstan run mike/hello

This will execute whatever the previous command was set to. In case you have not specified the --run command when composing the image, this is not going to be what you wanted to do, as it will actually format VM’s entire root disk.
Instead, you should specify the run command either during image composition :

$ capstan package compose --run /usr/bin/myapp mike/hello
$ capstan run mike/hello

or runtime:

$ capstan package compose mike/hello
$ capstan run -e /usr/bin/myapp mike/hello

If you have included CLI into your application, you may launch it right away:

$ capstan run -e /cli/cli.so
Updating image /home/lemmy/.capstan/repository/eu.mikelangelo-project.app.demo/eu.mikelangelo-project.app.demo.qemu...
Setting cmdline: /tools/cpiod.so --prefix /
Uploading files 287 / 287 [====================================================] 100.00 % 0
All files uploaded
Created instance: eu.mikelangelo-project.app.demo
Setting cmdline: /cli/cli.so
OSv v0.24-78-g69bd35e
eth0: 192.168.122.15
/# exit
Goodbye

 

Java Applications

Capstan provides initial support for composing and running Java-based applications. It exploits the mechanism of the OSv build scripts and python modules that set a specific command line launching main class that serves as a proxy to the actual Java commands.
To enable Java application, you must add a dependency to eu.mikelangelo-project.osv.java, for example:

Additionally, you have to provide another manifest file configuring the Java application. This manifest file consists of the following options:

  • main: fully classified name of the main class
  • args: a list of command line args used by the application
  • classpath: a list of paths where classes and other resources should be found
  • vmargs: a list of JVM args (for example Xmx, Xms, …)

This manifest must be stored in meta/java.yaml file. An example of a simple Java manifest is:

main: main.Hello
classpath:
    - /

This will start class main.Hello. Classpath is set to the root because the main class is located in /main/Hello.class file.
A slightly more complex example of Java manifest (taken from our Hadoop HDFS application; note that classpath is trimmed).

main: org.apache.hadoop.hdfs.server.datanode.DataNode
classpath: 
    - /hdfs/etc/hadoop
    - /hdfs/share/hadoop/common/lib/commons-logging-1.1.3.jar
    - /hdfs/share/hadoop/common/lib/jersey-json-1.9.jar
    - ...
vmargs: 
    - Dproc_datanode 
    - Xmx1000m 
    - Djava.net.preferIPv4Stack=true 
    - Dhadoop.log.dir=/hdfs/logs 
    - Dhadoop.log.file=hadoop.log 
    - Dhadoop.home.dir=/hdfs 
    - Dhadoop.id.str=xlab 
    - Dhadoop.root.logger=INFO,console 
    - Djava.library.path=/hdfs/lib/native 
    - Dhadoop.policy.file=hadoop-policy.xml 
    - Djava.net.preferIPv4Stack=true 
    - Dhadoop.security.logger=ERROR,RFAS 
    - Dhadoop.security.logger=INFO,NullAppender

 

Replacing the OSv Base Image

Occasionally, it is required that a different base OSv loader image is used for composing virtual machine images. In this case, follow the instructions on how to build the OSv kernel. Once the build is finished, look for a file called build/last/loader.img. You may copy this image into your Capstan repository for Capstan to use it as the base image for all your future capstan package compose calls:

$ cp build/last/loader.img ~/.capstan/repository/mike/osv-loader/osv-loader.qemu

 

>Mind the target file name as it has to be exactly the same. Otherwise, the old base image will still be used.