When migrating ZFS datasets from several smaller ZFS pools to a single larger pool I ran into the issue of also migrating a few LXD containers from one pool to another. A short research showed mainly guides for doing this in LXD 2.x and involved editing the SQLite database of LXD, something I wanted to spare myself.
However, a post on the linux containers forum suggested that a feature like this is going to implemented in LXD 3.0. My instance is running LXD 3.2 and a quick
lxc help showed the
lxc move command.
First, I created the new ZFS dataset on the new ZFS pool by running:
zfs create tank/lxd
To use this dataset I added it as a new storage to LXD by executing:
lxc storage create default_new zfs source=tank/lxd
This makes the storage available to lxd. Running
zfs list will show additional sub-datasets like
The storages available to LXD can then be seen with
lxc storage list.
At this point the list should at least show the old
default storage and the newly created
Now that the new dataset is created and made available to LXD, the existing containers can be moved to their new destination.
Moving the Containers
The containers are running services that mostly affect myself, so keeping the services up during the migrations is not a priority for me.
Therefore, I stopped any of the running containers before moving them to the new pool by running
lxc stop my_container.
You might have to do things differently if you are required to keep them up during the process.
For each of the containers I was basically running the following commands:
lxc move my_container my_container_NEW -s default_new lxc move my_container_NEW my_container
It seems to be necessary to change the name of the container (a change in capitalisation of the first letter is fine as well).
Therefore, the second line is just renaming the container to its original name.
-s flag is used to specify the name of the lxd storage as used during the preparation of the lxc storage.
lxc storage list show if the old
default storage is still used anywhere.
This might still indicate that the storage is used by at least one entity.
In my case the storage was still used by the default profile, which can easily be verified by running:
lxc profile show default
devices -> root -> pool section of the output the used storage is listed.
The used storage pool for the profile can then be changed by running:
lxc profile device set default root pool default_new
USED BY column of the old storage shows a 0, the storage can finally be removed by running:
lxc storage delete default
And this concludes this guide on moving lxd containers from one ZFS pool to another in LXD 3.2
As an interesting sidenote: Before the migration I had several images listed in the
old_tank/lxd/deleted dataset. I was told that they cannot really be deleted as long as any container is referencing them. However, moving the containers from one pool to another seemed to get rid of them.