Cédric Bosdonnat - Virtualization

Virt-bootstrap 1.0.0 released

2017-09-08T09:24:00+02:00

Yesterday, virt-bootstrap came to life. This tool aims at simplifying the creation of root file systems for use with libvirt's LXC container drivers. I started prototyping it a few months ago and Radostin Stoyanov wonderfully took it over during this year's Google Summer of Code.

For most users, this tool will just be used by virt-manager (since version 1.4.2). But it can be used directly from any script or command line.

The nice thing about virt-bootstrap is that will allow you to create a root file system out of existing docker images, tarballs or virt-builder templates. For example, the following command will get and unpack the official openSUSE docker image in /tmp/foo.

$ virt-bootstrap docker://opensuse /tmp/foo

Virt-bootstrap offers options to:

generate qcow2 image with backing chain instead of plain folder
apply user / group ID mapping
set the root password in the container

Enjoy easy containers creation with libvirt ecosystem, and have fun!

System container images

2017-03-08T16:37:00+01:00

As of today creating libvirt lxc system container root file system is a pain. Docker's fun came with its image sharing idea... why couldn't we do the same for libvirt containers? I will expose here is an attempt at this.

To achieve such a goal we need:

container images
something to share them
a tool to pull and use them

Container images

OpenBuildService thanks to kiwi knows how to create images, even container images. There even are openSUSE Docker images. To use them as system container images, some more packages need to be added to those. I thus forked the project on github and branched the OBS projects to get system container images for 42.1, 42.2 and Tumbleweed.

Using them is as simple as downloading them, unpacking them and use them as a container's root file system. However, sharing them would be so fun!

Sharing images

There is no need to reinvent the wheel to share the images. We can just consider them like any docker image. With the following commands we can import the image and push it to a remote registry.

docker import openSUSE-42.2-syscontainer-guest-docker.x86_64.tar.xz system/opensuse-42.2
docker tag system/opensuse-42.2 myregistry:5000/system/opensuse-42.2
docker login myregistry:5000
docker push myregistry:5000/system/opensuse-42.2

The good thing with this is that we can even use the docker build and Dockerfile magic to create customized images and push them to the remote repository.

Instanciating containers

Now we need a tool to get the images from the remote docker registry. Hopefully there is a tool that helps a lot to do this: skopeo. I wrote a small virt-bootstrap tool using it to instanciate the images as root file systems.

Here is how instanciating a container looks like with it:

virt-bootstrap.py --username myuser \
                  --root-password test \
                  docker://myregistry:5000/system/opensuse-42.2 /path/to/my/container

virt-install --connect lxc:/// -n 422 --memory 250 --vcpus 1 \
                --filesystem /path/to/my/container,/ \
                --filesystem /etc/resolv.conf,/etc/resolv.conf \
                --network network=default

And voila! Creating an openSUSE 42.2 system container and running it with libvirt is now super easy!

Easy sandboxed apps

2015-08-25T09:54:00+02:00

These days, I wanted to build some SUSE documentation, but daps drags quite a few dependencies. Thus I decided to use that occasion to move it to a sandbox. The idea is very similar to what was detailled in my post on containers for GUI apps, but will be made much easier thanks to the recent progresses in virt-sandbox.

Note that this is only possible with recent virt-sandbox. To make sure you have virt-sandbox with all features, install it from the OBS Virtualization repository.

Creating the disk image

Most of this part will be simplified, as I reused the test-base.qcow2 image created in the previous post. I only added it a non-root user. Doing this is pretty straight forward thanks to qemu-img's commit feature.

First create a working overlay image based on test-base.qcow2:

qemu-img create -f qcow2 \
                -o backing_file=$PWD/test-base.qcow2 \
                daps.qcow2

As a non-privileged user, boot a sandbox running this disk image:

virt-sandbox -n daps \
             --privileged \
             -m host-image:/=$PWD/daps.qcow2,format=qcow2 \
             -- \
             /bin/sh

Note that the --privileged parameter keeps you as root in the sandbox. Otherwise you would be logged in as a user with the same UID as the one you ran virt-sandbox with. You can do the changes you need in the base image. In our case, I will add an unprivileged user.

useradd -m myuser

As we want this change to propagate to the base image, please refrain from installing daps or doing other things that you don't want to see in the base image. Exit the shell to exit the sandbox and get back to your host command.

In order to have working network later in the sandboxes, we have to install util-linux-systemd and dhcp-client as it wasn't done in when creating the base image. For this we need to switch to root and mount the image with libguestfs tools since zypper can't get any network so far.

sudo guestmount -a $PWD/daps.qcow2 -m /dev/sda:/ /mnt
sudo zypper --root /mnt in dhcp-client util-linux-systemd
sudo guestunmount /mnt

We will now commit all the changes made in our overlay image to the base image:

qemu-img commit $PWD/daps.qcow2

Note that you may have to get permissions to write on test-base.qcow2 as we created it as root in the previous post.

Now, we only have to install daps in our now-empty overlay image. To do so, run the following command:

virt-sandbox -n daps \
             --privileged \
             -m host-image:/=$PWD/daps.qcow2,format=qcow2 \
             -N dhcp \
             -- \
             /bin/sh

Note the -N dhcp argument that will now run dhclient and will provide you network in the sandbox. This network is limited, since it is user networking. For more details on it, report to this page.

In the sandbox, we can now install daps normally:

zypper ar http://download.opensuse.org/repositories/Documentation:/Tools/openSUSE_13.2/Documentation:Tools.repo
zypper in daps

Exit the shell to end the sandbox: your disk image is ready.

Running daps

In order to have a smooth user experience with daps, better create a script to run virt-sandbox for you. Create executable ~/bin/daps with content similar to this one:

#!/bin/sh
virt-sandbox \
    -n daps \
    -m host-image:/=/path/to/daps.qcow2,format=qcow2 \
    -m host-bind:/home/myuser=/home/myuser \
    -m ram:/tmp=100MiB \
    -m ram:/run=100MiB \
    -- \
    /usr/bin/daps $@

You can add options to mount your host folders in your sandbox. For example, this will mount /home/myuser at the same place in the sandbox:

-m host-bind:/home/myuser=/home/myuser

Make sure that your documentation sources will be mounted in the sandbox.

When running the daps command on your machine, you will run the daps command within a super tiny KVM machine running with your UID. Note that I didn't add the -N dhcp option in the script since daps doesn't need it, but you may need it for other applications or to update your packages.

Libvirt container for GUI apps

2015-05-06T10:13:00+02:00

I recently tried to get the openFATE client working in a container. I know it may sound stupid, but I didn't want to pollute my Gnome machine with KDE libraries (end of troll) and wanted to use this to seriously play with application containers. My constraints were:

minimize the duplication of the root file system. I know docker already does this, but I'm a libvirt hacker after all.
get the application show up on the host display for a smooth use.

Creating the root file system

To minimze the file system duplication, I went with a qcow2 disk image with a backing file. The first step is to create the base image with the file system to be reused by other containers.

Create the disk image:

qemu-img create -f qcow2 test-base.qcow2 15G

Create the nbd device from the disk image, format it and mount it. Depending on your linux distribution you may even need to load the nbd kernel module, hence the first line.

modprobe nbd
/usr/bin/qemu-nbd --format qcow2 -n -c /dev/nbd0 $PWD/test-base.qcow2
mkfs.ext3 /dev/nbd0
mount /dev/nbd0 /mnt

Populate the image with the openSUSE 13.2 Minimal_base pattern:

zypper --root /mnt ar http://download.opensuse.org/distribution/13.2/repo/oss/ main
zypper --root /mnt ar http://download.opensuse.org/update/13.2/ updates
zypper --root /mnt in -t pattern Minimal_base

Unmount and clean up the nbd device:

umount /mnt
pkill qemu-nbd

Now, we will create an overlay image on top of the one we just created. In this image, we will only install fate, and create an unprivileged user to run it.

Create the image. Note the backing_file and backing_fmt options as they will actually setup the backing chain of qcow2 images.

qemu-img create -f qcow2 \
                -o backing_fmt=qcow2,backing_file=$PWD/test-base.qcow2 \
                myapp.qcow2

Mount the new image via qemu-nbd again. Note that there is no need to format that new image: it is really only a diff with the test-base.qcow2 file.

/usr/bin/qemu-nbd --format qcow2 -n -c /dev/nbd0 $PWD/myapp.qcow2
mount /dev/nbd0 /mnt

Install the application (fate in this example):

zypper --root /mnt ar http://download.opensuse.org/repositories/FATE/openSUSE_13.2/ FATE
zypper --root /mnt in fate

Add an unprivileged user:

useradd -m myuser

For the application to find the X server display, set the DISPLAY in the user's profile. The localhost:0 value here can be adjusted depending on the network settings of the container. In this case, I will use a container without the netns namespace to simplify, but with the default libvirt network the value would be 192.168.122.1:0.

echo 'export DISPLAY=localhost:0' > ~myuser/.profile

Unmount and clean up the nbd device:

umount /mnt
pkill qemu-nbd

Setting up the container

Sadly, there is no other possible way to create the container than by manually feeding the XML definition to libvirt. Here is a template for the definition:

<domain type='lxc'>
  <name>myapp</name>
  <memory unit='MiB'>256</memory>
  <vcpu placement='static'>1</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64'>exe</type>
    <init>/usr/bin/su</init>
    <initarg>-</initarg>
    <initarg>myuser</initarg>
    <initarg>-c</initarg>
    <initarg>/usr/bin/fate</initarg>
  </os>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <controller type='ide' index='0'/>
    <filesystem type='file'>
      <driver type='nbd' format='qcow2'/>
      <source file='/path/to/myapp.qcow2'/>
      <target dir='/'/>
    </filesystem>
    <filesystem type='mount'>
      <source dir='/home/myuser/.Xauthority'/>
      <target dir='/home/myuser/.Xauthority'/>
    </filesystem>
    <filesystem type='ram'>
      <source usage='10240' units='KiB'/>
      <target dir='/run'/>
    </filesystem>
    <filesystem type='ram'>
      <source usage='102400' units='KiB'/>
      <target dir='/tmp'/>
    </filesystem>
    <console type='pty'/>
  </devices>
  <seclabel type='dynamic' model='apparmor' relabel='yes'/>
</domain>

In this template, the memory, vcpu and paths to images need to be adapted to your setup. Note that the qcow2 image is mounted in the container using the filesystem nbd driver.

The host user's .Xauthority file is mounted in the container's user home. This is needed for the X applications to connect to the host display.

For the application to be automatically launched as the unprivileged user, the definition init command is set to

/usr/bin/su - myuser -c /usr/bin/fate

Running the application

Before being able to connect to the host Xorg server, we need to have it listen for TCP connection. In openSUSE, this can be achieved by changing DISPLAYMANAGER_XSERVER_TCP_PORT_6000_OPEN to yes in /etc/sysconfig/displaymanager.

The application needs to be able to authenticate on the host Xorg server. For my tests I manually crafted the .Xauthority using xauth. Just remember that the cookie will change, so either regenerate it before each start or mount $XAUTHORITY directly to the container's .Xauthority.

xauth extract - $DISPLAY | xauth -f $HOME/.Xauthority merge -

Running the application is as easy as starting the container:

virsh -c lxc:/// start --console myapp

Of course it would be much more convenient to have it wrapped in a script with proper sudo configuration to let normal users run the container without needing to become root.

Limitations

This approach is still pretty complex and not that easy for newcomers.

The image creation process will be simplified as part of Google Summer of Code 2015 with libvirt.
The container definition and start could be done with virt-sandbox, but the host-image mount parameter should first be able to take qcow2 images.
libvirt doesn't stop the qemu-nbd process when the container is stopped, but hey, that's a bug I can work on!

Hackweek 12

2015-04-17T13:50:00+02:00

This week was hackweek 12 at SUSE. I decided to do whatever was needed to get the open build service (aka OBS) able to create virt-builder images repositories. OBS is already able to create VM images using kiwi, but it is not able to generate the signed index file and publish the compressed images.

I owe many thanks to Adrian for setting me on the right tracks to get this done. In short, there are two pieces needed to reach the goal:

a kiwi hook script running after the kiwi build. How to do this is largely described in the containment-rpm README.
a patch for the bs_publisher script to create the index from the parts generated for each image, sign it and publish it together with the compressed images.

So far a project is ready in openSUSE's build service in the home:cbosdonnat:Builder project. It only waits for the PR#909 to land on this instance of OBS.

With this, I hope we will soon be able to provide official openSUSE images for our openSUSE virt-builder users.

Hackweek 11

2014-10-25T18:03:00+02:00

Last week was hackweek 11 at SUSE. I chose to work on starting an Android client application for libvirt. I know there is already something like this on the Google play, but it's closed source and seems unmaintained... and anyway it helped me dive in Android NDK and native code building.

As of today, the application can connect to a remote server using libssh2 and list the domains on it. The good thing is that the hard work to get libvirt.so, JNA and the Java code running together is solved.

The code can be found on github.com/cbosdo/libvirt-droid even if it still needs a huge bunch of love!

Migrating an LXC container to libvirt lxc

2014-07-17T15:41:00+02:00

I am now working on libvirt and the tools gravitating around it for almost a year and still haven't blogged about anything related to it. As a first post on virtualization, I'll tell you how to migrate your LXC container to use the libvirt goodness.

In order to achieve the migration, the host machine needs to be upgraded to openSUSE 13.1 or later. Quite some of the required features are only in Factory, so those of you running 13.1 will need to add the Virtualization repository like this:

host ~# zypper ar -f http://download.opensuse.org/repositories/Virtualization/openSUSE_13.1/Virtualization.repo

The first thing to do is to install libvirt-daemon-lxc, to drag all the needed packages to run containers on libvirt. Of course, installing virt-manager will also provide you a convenient GUI to handle them.

host ~# zypper in libvirt-daemon-lxc

The LXC container files are usually living in the /var/lib/lxc/ folder. Let's assument for this example, that we have an lxc container named mycontainer to migrate.

To avoid the boring task of creating a similar configuration for the libvirt container, use the virt-lxc-convert to generate an equivalent libvirt domain configuration for the container.

host ~# virt-lxc-convert /var/lib/lxc/mycontainer/config >mycontainer.xml

The dumped configuration file, needs to be reviewed before feeding it to libvirt. Most of the configuration will be OK, but the network configuration may need some adjustments. For example, for now, libvirt doesn't have any equivalent to lxc.network.ipv*.

When the configuration is ready, define the libvirt container using the following command:

host ~# virsh -c lxc:/// define mycontainer.xml

To be able to connect as root on the console, /dev/pts/0 needs to be added to the container securetty:

host ~# echo "pts/0" >> /var/lib/lxc/mycontainer/rootfs/etc/securetty

Due to rhbz#966807, the kernel 3.14 or later is required to be able to login on the container.

After this you should be able to start and connect to the container console using virsh or virt-manager.

Updating the openSUSE in the container

This is just normal zypper manipulation, just that the --root parameter needs to be added to tell zypper to work on the container's root file system. In the following command this will just be aliased as zypper-mycont.

The following example commands will just replace the container repositories by openSUSE 13.1 repositories, refresh them and run a dist upgrade.

host ~# alias zypper-mycont="zypper --root=/var/lib/lxc/mycontainer/rootfs"
host ~# zypper-mycont lr
# | Alias    | Name     | Enabled | Refresh
--+----------+----------+---------+--------
1 | repo-oss | repo-oss | Yes     | No
2 | update   | update   | Yes     | No
host ~# zypper-mycont rr repo-oss
host ~# zypper-mycont rr update
host ~# zypper-mycont ar http://download.opensuse.org/distribution/13.1/repo/oss repo-oss
host ~# zypper-mycont ar http://download.opensuse.org/update/13.1/ udpate
host ~# zypper-mycont ref
host ~# zypper-mycont dup