{"id":663,"date":"2023-06-27T22:55:13","date_gmt":"2023-06-27T20:55:13","guid":{"rendered":"https:\/\/www.dont-panic.cc\/capi\/?p=663"},"modified":"2023-06-27T22:55:13","modified_gmt":"2023-06-27T20:55:13","slug":"overlay2-for-docker-within-an-unprivileged-lxc-container","status":"publish","type":"post","link":"https:\/\/www.dont-panic.cc\/capi\/2023\/06\/27\/overlay2-for-docker-within-an-unprivileged-lxc-container\/","title":{"rendered":"overlay2 for Docker within an unprivileged LXC container"},"content":{"rendered":"\n

For my Jenkins<\/a> installation I use a Docker<\/a> agent inside an LXC container. I want this container to be unprivileged, so that the host is somewhat protected from misconfiguration (not deliberate attacks). The default setup works fine, but after a bit of experimenting, I noticed that I was soon running out of disk-space. The reason for that turned out that Docker had fallen back to using the vfs<\/code> storage backend instead of overlay2<\/code><\/a>, which basically creates a copy for every layer and every running container.<\/p>\n\n\n\n

# docker info | grep Storage\n Storage Driver: vfs<\/code><\/pre>\n\n\n\n
Further investigation showed, that this was due to the fact that the container was unprivileged. Short experiments with making the container privileged also yielded issues with cgroup management of the outer docker container on the host. So what was the reason for the issues? It seems that the ID mapping \/ shifting of the user IDs prevented the overlay2 driver from working.<\/p>\n\n\n\n
Therefore I decided to try to mount a host directory as a “device” into the container’s \/var\/lib\/docker<\/code>. But using the shift=true<\/code> option, this again fails, since this way the underlying filesystem is shiftfs<\/code> and not plain ext4<\/code> (see supported filesystems<\/a> for various storage drivers). So a solution without “shift” is required.<\/p>\n\n\n\n
Shifting UIDs is done by a fixed offset for a container, in my case it’s 1,000,000. You need to figure this out for your system, but likely it’s the same. So by creating the external storage directory with this as owner and then mounting it inside the container without shifting, things start to get working.<\/p>\n\n\n\n
export CONTAINER_NAME=mycontainer\nexport DOCKER_STORAGE_DIRECTORY=\/mnt\/pool\/mycontainer\/var-lib-docker\n\nmkdir -p \"$DOCKER_STORAGE_DIRECTORY\"\nchown 1000000:1000000 \"$DOCKER_STORAGE_DIRECTORY\"\n\nlxc config device add \"$CONTAINER_NAME\" var-lib-docker disk source=\"$DOCKER_STORAGE_DIRECTORY\" path=\/var\/lib\/docker\n\n# important, security.nesting is required for nested containers to work!\nlxc config set \"$CONTAINER_NAME\" security.nesting=true<\/code><\/pre>\n\n\n\n
After this docker info | grep Storage<\/code> finally showed what I wanted:<\/p>\n\n\n\n
# docker info | grep Storage\n Storage Driver: overlay2<\/code><\/pre>\n","protected":false},"excerpt":{"rendered":"
For my Jenkins installation I use a Docker agent inside an LXC container. I want this container to be unprivileged, so that the host is somewhat protected from misconfiguration (not deliberate attacks). The default setup works fine, but after a bit of experimenting, I noticed that I was soon running out of disk-space. The reason … Continue reading “overlay2 for Docker within an unprivileged LXC container”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[6,5,10],"tags":[247,48,246],"class_list":["post-663","post","type-post","status-publish","format-standard","hentry","category-computer","category-software","category-sysadmin","tag-docker","tag-linux","tag-lxc"],"_links":{"self":[{"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/posts\/663","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/comments?post=663"}],"version-history":[{"count":3,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/posts\/663\/revisions"}],"predecessor-version":[{"id":666,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/posts\/663\/revisions\/666"}],"wp:attachment":[{"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/media?parent=663"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/categories?post=663"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.dont-panic.cc\/capi\/wp-json\/wp\/v2\/tags?post=663"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}