For my Jenkins installation I use a Docker agent inside an LXC container. I want this container to be unprivileged, so that the host is somewhat protected from misconfiguration (not deliberate attacks). The default setup works fine, but after a bit of experimenting, I noticed that I was soon running out of disk-space. The reason for that turned out that Docker had fallen back to using the
vfs storage backend instead of
overlay2, which basically creates a copy for every layer and every running container.
# docker info | grep Storage Storage Driver: vfs
Further investigation showed, that this was due to the fact that the container was unprivileged. Short experiments with making the container privileged also yielded issues with cgroup management of the outer docker container on the host. So what was the reason for the issues? It seems that the ID mapping / shifting of the user IDs prevented the overlay2 driver from working.
Therefore I decided to try to mount a host directory as a “device” into the container’s
/var/lib/docker. But using the
shift=true option, this again fails, since this way the underlying filesystem is
shiftfs and not plain
ext4 (see supported filesystems for various storage drivers). So a solution without “shift” is required.
Shifting UIDs is done by a fixed offset for a container, in my case it’s 1,000,000. You need to figure this out for your system, but likely it’s the same. So by creating the external storage directory with this as owner and then mounting it inside the container without shifting, things start to get working.
export CONTAINER_NAME=mycontainer export DOCKER_STORAGE_DIRECTORY=/mnt/pool/mycontainer/var-lib-docker mkdir -p "$DOCKER_STORAGE_DIRECTORY" chown 1000000:1000000 "$DOCKER_STORAGE_DIRECTORY" lxc config device add "$CONTAINER_NAME" var-lib-docker disk source="$DOCKER_STORAGE_DIRECTORY" path=/var/lib/docker # important, security.nesting is required for nested containers to work! lxc config set "$CONTAINER_NAME" security.nesting=true
docker info | grep Storage finally showed what I wanted:
# docker info | grep Storage Storage Driver: overlay2