Moving LXD Containers Between ZFS Pools

When migrating ZFS datasets from several smaller ZFS pools to a single larger pool I ran into the issue of also migrating a few LXD containers from one pool to another. A short research showed mainly guides for doing this in LXD 2.x and involved editing the SQLite database of LXD, something I wanted to spare myself.

However, a post on the linux containers forum suggested that a feature like this is going to implemented in LXD 3.0. My instance is running LXD 3.2 and a quick lxc help showed the lxc move command.

Preparation

First, I created the new ZFS dataset on the new ZFS pool by running:

zfs create tank/lxd

To use this dataset I added it as a new storage to LXD by executing:

lxc storage create default_new zfs source=tank/lxd

This makes the storage available to lxd. Running zfs list will show additional sub-datasets like tank/lxd/containers, tank/lxd/custom, tank/lxd/deleted etc. The storages available to LXD can then be seen with lxc storage list. At this point the list should at least show the old default storage and the newly created default_new.

Now that the new dataset is created and made available to LXD, the existing containers can be moved to their new destination.

Moving the Containers

The containers are running services that mostly affect myself, so keeping the services up during the migrations is not a priority for me. Therefore, I stopped any of the running containers before moving them to the new pool by running lxc stop my_container. You might have to do things differently if you are required to keep them up during the process.

For each of the containers I was basically running the following commands:

lxc move my_container my_container_NEW -s default_new
lxc move my_container_NEW my_container

It seems to be necessary to change the name of the container (a change in capitalisation of the first letter is fine as well). Therefore, the second line is just renaming the container to its original name. The -s flag is used to specify the name of the lxd storage as used during the preparation of the lxc storage.

Finally, running lxc storage list show if the old default storage is still used anywhere. This might still indicate that the storage is used by at least one entity. In my case the storage was still used by the default profile, which can easily be verified by running:

lxc profile show default

In the devices -> root -> pool section of the output the used storage is listed. The used storage pool for the profile can then be changed by running:

lxc profile device set default root pool default_new

If the USED BY column of the old storage shows a 0, the storage can finally be removed by running:

lxc storage delete default

And this concludes this guide on moving lxd containers from one ZFS pool to another in LXD 3.2

As an interesting sidenote: Before the migration I had several images listed in the old_tank/lxd/deleted dataset. I was told that they cannot really be deleted as long as any container is referencing them. However, moving the containers from one pool to another seemed to get rid of them.

social