Why I like Unikernels

Unikernels allow us to explore new ideas: to rethink old OS interfaces and to remove unnecessary layers of code accumulated over the last several decades. Modern general-purpose operating systems are fantastic, but they're so huge it's hard to experiment with totally different designs. Before the cloud, a research prototype OS would probably only be able to support one or two types of hardware. Now that we have common interfaces like Xen `netfront` and `blkfront` and `virtio-net` and `virtio-blk` it's feasible to write a new OS that can run on any cloud provider, using the hypervisor as a common substrate.

Unikernels are library OSes

Unikernels are also "library OSes" where network stacks and device drivers are libraries which are linked directly with the custom application logic into a single binary. We based our experiments on Mirage OS, which has many nice libraries in OCaml including

Note that since these are libraries, they can be used in other contexts too (not just unikernels). For example the TCP/IP stack can just as easily run in userspace on a traditional OS. Working at Docker, we were able to use the unikernel TCP/IP stack to solve some tricky networking problems in Docker Desktop . The same vpnkit tool (based on the Mirage OS libraries) can also be used to provide network connectivity to containers running in docker's rootless mode.

Unikernels are a good choice for high-assurance applications

Due to only linking code that is actually needed, unikernels have a naturally reduced attack surface compared to traditional applications. My favourite example is the "xenstore" service of the Xen hypervisor. In Xen, the hypervisor focuses on isolating untrusted VMs from each other, but does not perform any I/O directly. I/O is delegated to special privileged VMs called "driver domains". The connections between the driver domains and the user VMs are configured via a control service called "xenstore".

The xenstore service in Xen configures connections between driver domains and user VMs

The xenstore service is a critical part of a Xen host, so we must be confident that it's rock solid. Normally the xenstore service runs as a regular Linux process in a "domain zero", a trusted VM. Running as a regular process means that it's open to interference from other processes, perhaps via signals or shared files or shared memory. Luckily for us there was already a fast, memory-safe implementation of xenstore written in OCaml, which we could then port to run as an isolated unikernel VM using MirageOS, increasing our confidence in the overall security of the system.

Thanks to the library-based design, we could also increase the fault-tolerance of xenstore by using the OCaml Irmin (git-like) database library to store the xenstore state. This means that if the xenstore VM crashes, it can recover from a snapshot of its state. Here's our prototype "Xenstore TNG" showing a simulation of booting lots of VMs concurrently:

Unikernels have been used to increase the security of other OSes too. For example, QubesOS which describes itself as a "a reasonably secure" operating system (!) uses Xen to isolate apps running in separate VMs from each other and from driver domains (running complex and therefore potentially buggy drivers). Network driver domains are particularly vulnerable to attack, since they are exposed to the hostile Internet. Thomas Leonard used a MirageOS unikernel for the trusted firewall component, which sits between the untrusted network driver VMs and the user application VMs. The MirageOS firewall VM is the last line of defence between the user applications and the hostile Internet.

Unikernels allow experimentation

Traditional OSes have mature but relatively fixed interfaces between kernel and user code. Since unikernels run everything linked together in the same binary, we can experiment with new interfaces more easily. We did a fun experiment where we spawned unikernels in response to network requests, and were able to hide the VM startup latency by accepting the TCP connection in a separate "inetd-like" process, and then when the unikernel was ready, we transmitted the TCP connection state (sequence numbers etc) over shared memory to the unikernel. This was easy to do because the TCP/IP stack was a library which we could easily customize.

Lots of different styles of Unikernels exist

Since we worked on MirageOS, several other interesting unikernel projects have appeared. UniKraft is a "unikernel development kit" that supports running existing applications (not just OCaml) as unikernels. I really appreciate their Docker-inspired UX:

$ kraft run unikraft.org/helloworld:latest

Also worth checking out is Nanos, which I notice also has OCaml support. Nanos has several interesting additional features, including a OpenBSD-style pledge and unveil and support for confidential computing via AMD SEV. I'm also excited to see upstream Linux embrace Unikernel-style approaches with Unikernel Linux (UKL) which treats Linux itself as a library which can be linked directly against a single process.