In 2021 we in Robur have been working towards easing deployment of reproducible mirage applications. The work has been funded by the European Union under the Next Generation Internet (NGI Pointer) initiative. The result is online.

The overall goal is to push MirageOS into production in a trustworthy way. We worked on reproducible builds for Opam packages and MirageOS - with the infrastructure being reproducible itself. Reproducible builds are crucial for supply chain security - everyone can reproduce the exact same binary (by using the same sources and environment), without reproducible builds we would not publish binaries.

Reproducible builds are also great for fleet management: by inspecting the hash of the binary that is executed, we can figure out which versions of which libraries are in the unikernel - and suggest updates if newer builds are available or if a used library has a security flaw -- albatross-client update my-unikernel is everything needed for an update.

Several ready-to-use MirageOS unikernels are built on a daily basis - ranging from authoritative DNS servers (secondary, let's encrypt DNS solver), DNS-and-DHCP service (similar to dnsmasq), TLS reverse proxy, Unipi - a web server that delivers content from a git repository, DNS resolver, CalDAV server, and of course your own MirageOS unikernel.

Brief robur and MirageOS introduction

MirageOS is an operating system, developed in OCaml, which produces unikernels. A unikernel serves a single purpose and is a single process, i.e. only has the really needed dependencies. For example, an OpenVPN endpoint does neither include persistent storage (block device, file system) nor user management. MirageOS unikernels are developed in OCaml, a statically typed and type-safe programming language - which avoids common pitfalls from the grounds up (spatial and temporal memory safety issues).

Robur is a collective that develops MirageOS and OCaml software with open source license. It was started in 2017, and is part of the non-profit company Änderwerk gGmbH. We received funding from several projects (prototypefund, NGI pointer), donations, and commercial contracts.

Deploying MirageOS unikernel

To run a MirageOS unikernel on your laptop or computer with virtualization extensions (VT-x - KVM/BHyve), you first have to install the solo5 and albatross packages. Afterwards you need to setup a virtual network switch (a bridge interface) where your unikernels will communicate, and forwarding.

Host system package installation

For Debian and Ubuntu systems, we provide package repositories. Browse the dists folder for one matching your distribution, and add it to /etc/apt/sources.list:

$ curl -fsSL https://apt.robur.coop/gpg.pub | gpg --dearmor > /usr/share/keyrings/apt.robur.coop.gpg
$ echo "deb [signed-by=/usr/share/keyrings/apt.robur.coop.gpg] https://apt.robur.coop ubuntu-20.04 main" > /etc/apt/sources.list.d/robur.list # replace ubuntu-20.04 with e.g. debian-11 on a debian buster machine
$ apt update
$ apt install solo5 albatross

On FreeBSD:

$ fetch -o /usr/local/etc/pkg/robur.pub https://pkg.robur.coop/repo.pub # download RSA public key
$ echo 'robur: {
  url: "https://pkg.robur.coop/${ABI}",
  mirror_type: "srv",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/pkg/robur.pub",
  enabled: yes
}' > /usr/local/etc/pkg/repos/robur.conf # Check https://pkg.robur.coop which ABI are available
$ pkg update
$ pkg install solo5 albatross

For other distributions and systems we do not (yet?) provide binary packages. You can compile and install them using opam (opam install solo5 albatross). Get in touch if you're keen on adding some other distribution to our reproducible build infrastructure.

There is no configuration needed. Start the albatross_console and the albatross_daemon service (via systemctl daemon-reload ; systemctl start albatross_daemon on Linux or service albatross_daemon start on FreeBSD). Executing albatross-client info should return success (exit code 0) and no running unikernel. You may need to be in the albatross group, or change the permissions of the Unix domain socket (/run/albatross/util/vmmd.sock on Linux, /var/run/albatross/util/vmmd.sock on FreeBSD).

To check that albatross works, get the latest hello world unikernel and run it:

$ wget https://builds.robur.coop/job/hello/build/latest/bin/hello-key.hvt
$ albatross-client console my-hello-unikernel & # this is sent to the background since it waits and displays the console of the unikernel named "my-hello-unikernel"
$ albatross-client create my-hello-unikernel hello-key.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
$ albatross-client create --arg='--hello="Hello,\ my\ unikernel" my-hello-unikernel hello-key.hvt # executes the same unikernel, but passes the boot parameter "--hello"
$ fg # back to albatross-client console
$ Ctrl-C # kill that process

Voila, we have a working albatross installation. Albatross also supports a remote client (using a TLS handshake) albatross-client --ca <ca.pem> --ca-key <ca.key> --server-ca <cacert.pem> --destination <myhost>, monitoring of unikernels (albatross_stats and albatross_influx services), and a TLS endpoint (inetd/systemd: albatross-tls-endpoint).

Please ensure to have albatross in version of at least 2.0.0 to follow this page.

Network for your unikernel

Next we want to setup networking for our unikernels. We use a so-called "bridge" interface for this, which is a virtual network switch where you connect "tap" interfaces (layer 2 ethernet devices). A MirageOS unikernel uses tap interfaces for communication. We give our bridge the name "service" (and for example for monitoring and management you may want to setup another bridge "management").

If you're using a network manager that is capable of setting up bridge interfaces, use that interface.

If not, on Linux you can add the following to /etc/network/interfaces (the reason for adding a dummy interface to the bridge is that otherwise Linux uses the mac address of the first connected tap interface, and there'll be rather confusing issues):

auto service
# Host-only bridge
iface service inet manual
    up ip link add service-master address 02:00:00:00:00:01 type dummy
    up ip link set dev service-master up
    up ip link add service type bridge
    up ip link set dev service-master master service
    up ip addr add 10.0.42.1/24 dev service
    up ip link set dev service up
    down ip link del service
    down ip link del service-master

On FreeBSD, add the following to /etc/rc.conf:

cloned_interfaces="bridge0"
ifconfig_bridge0_name="service"
ifconfig_service="inet 10.0.42.1/24"

Afterwards either restart your system or re-run the service scripts to have the bridge setup in your running system.

To check that the networking works, get the latest static website unikernel and run it:

$ wget https://builds.robur.coop/job/static-website/build/latest/bin/https.hvt
$ albatross-client console my-website & # this is sent to the background since it waits and displays the console of the unikernel named "my-website"
$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' my-website https.hvt # this returns once the unikernel image has been transmitted to the albatross daemon
$ ping 10.0.42.2 # should receive answers
$ open http://10.0.42.2 # in your browser - also https://10.0.42.2 (you'll get a certificate warning)
$ wget http://10.0.42.2/ # should download the Hello Mirage world!
$ wget --no-check-certificate https://10.0.42.2/ # should also download the Hello Mirage world!
$ fg # back to albatross-client console
$ Ctrl-C # kill that process
$ albatross-client destroy my-website # kills the unikernel

When you reached this point, you have successfully launched a MirageOS unikernel, and are able to communicate from your computer with it. This uses the OCaml networking stack, and the host bridge interface.

Routing and Internet

Your unikernel may want to communicate not only with your host, but also with the Internet. The other way around is also important (the Internet wants to talk with your unikernel).

There are several options, depending on your setup:

• Your unikernel will be masqueraded (using NAT) - some ports may be forwarded to the unikernel,
• Your computer has several public IP addresses (and put the ethernet device with the ethernet cable on the bridge) and there is an external router,
• Your computer acts as a router for a subnet.

NAT

This won't allow your unikernel to be reachable from the outside.You'll need to:

• enable IPv4 forwarding
• add a firewall rule

On Linux:

$ echo "1" > /proc/sys/net/ipv4/ip_forward # enables IP forwarding
$ iptables -t nat -A POSTROUTING -o enp0s20f0 -j MASQUERADE # replace enp0s20f0 with your network interface

On FreeBSD:

$ echo 'gateway_enable="YES"' >> /etc/rc.conf # enable IP forwarding
$ echo 'pf_enable="YES"' >> /etc/rc.conf # enables the packet filter
$ echo "nat pass on em0 inet from 10.0.42.0/24 to any -> (em0)" >> /etc/pf.conf # replace em0 with your ethernet interface)

Public IP addresses

To put your unikernels on the same network as your host system, add that external network interface to the bridge:

On Linux, add up ip link set dev enp0s20f0 master service in /etc/network/interfaces (replace enp0s20f0 with your ethernet interface). On FreeBSD, add ifconfig_service="addm em0" to /etc/rc.conf (replace em0 with your ethernet interface).

Router

Enable IPv4 forwarding, and setup one IP address on the bridge (replacing the 10.0.42.1/24 above).

Unikernel execution

Let's test that your unikernels have access to the Internet by using the traceroute unikernel:

$ wget https://builds.robur.coop/job/traceroute/build/latest/bin/traceroute.hvt
$ albatross-client console my-traceroute & # this is sent to the background since it waits and displays the console of the unikernel named "my-traceroute"
$ albatross-client create --net=service --arg='--ipv4=10.0.42.2/24' --arg='--ipv4-gateway=10.0.42.1' my-traceroute traceroute.hvt # the IP configuration depends on your setup, use your public IP address and actual router IP if you've set it up.
$ fg # back to albatross-client console
$ Ctrl-C # kill that process

That's it. Albatross has more features, such as block devices, multiple bridges (for management, private networks, ...), restart if the unikernel exited with specific exit code, assignment of a unikernel to a specific CPU. It also has remote command execution and resource limits (you can allow your friends to execute a number of unikernels with limited memory and block storage accessing only some of your bridges). There is a daemon to collect metrics and report them to Grafana (via Telegraf and Influx). MirageOS unikernels also support IPv6, you're not limited to legacy IP.