During my latest system maintenance task, I downgraded the Docker version in my system from the latest to the Debian-installed one (latest is currently 28, while Debian uses Version 26). I hope to simplify management with that, and additionally get rid of external dependencies. The last motivation for this change was the latest bug with Docker not running properly in LXC due to some AppArmor problems.

What happened?

During the downgrade, it seems that volumes got deleted. I am not yet sure what actually went wrong: Either I pruned too much and deleted the volumes while pruning, or the volumes just did not survive the Version Downgrade and hence were deleted.

Anyways, after the update, all data of my Mail Server (which is Mailcow, and which stores loads of things in Docker Volumes) was gone. I found out because logging in suddenly was not possible any more.

How did I fix it?

My server backups are stored in Proxmox Backup Server as follows:

  • My datastore is called main.
  • The backup group is called host/myserver.
  • The backup is then called host/myserver/2025-11-12T05:00:02Z.

Now I wanted to restore a set of Docker volumes. Since this is only a small part of the full server, a full restore would have been overkill; I instead wanted to just restore the files inside the containers. I did that as follows:

  • Mount the volume:
    1
2
3
4
5
6

    mkdir /tmp/foo
proxmox-backup-client mount \
  --repository "root@pam@my-pbs-server.tech-tales.blog:8007:main" \
  host/myserver/2025-11-12T05:00:02Z \
  root.pxar.didx \
  /tmp/foo
    Note that I did not mount this on the server, as the affected server and the backup server are in different networks and have a quite low bandwidth between them. Hence all the extraction happened on my local machine (where bandwidth is fine), and then I uploaded the extract to the server. Now what happens in this command:
    • Mount this latest backup into /tmp/foo.
    • Technically, mount root.pxar.didx. If you open the backup in the UI, this is the file of the backup that allows to “browse files” and extract them. Not quite sure if this name could change.
    • /tmp/foo is just some directory on my system.
  • Now open a local shell. You can browse inside /tmp/foo as expected, altough everything is a bit slower.
  • Setup a tar file with the files you want to have (and store this outside the mount). Note to retain permissions, hence the extraction command for me looked something like:
    1

    tar czvpf /tmp/archive.tar.gz /tmp/foo/var/lib/docker/volumes/mailcow*
  • Push this archive to the server.
  • Note: Just extracting in the /var/lib/docker/volumes folder is not sufficient!
    Instead, extract the volume. Again, remember to retain permissions: tar xvpf archive.tar.gz
  • Let Docker setup all the volumes, but stop the containers again.
  • Now I had directories like /var/lib/docker/volumes/mailcowdockerized_mysql-vol-1, and similar. Inside those, I replaced the _data/ directory with the one from the backup.

What did I learn from it?

Proxmox Backup Server is just sitting there and doing their stuff. I don’t really care much about it, but this is one of my most valuable servers. Due to it’s existence, I can essentially mindlessly update things and be sure that breaking something is not too much of a problem. I once again learned how cool that is.

Downloading a directory from the Proxmox Backup Server UI does not work as expected; quite a few files inside this directory were missing. Which was shocking for me at first: Why is the data I expect to be there not there when I download the files?

I typically don’t use Docker Volumes but instead just bind-mounts of local directories. This was a good reason why.