I do terrible things sometimes

Abandon hope…

This is not a how-to, but more of a detailed confession about a terrible thing I’ve done in the last few days. The basic concept for this crime against humanity came during a user’s group meeting where several companies expressed overwhelming interest in containers, but were pinned to older, unsupported versions of CentOS due to 3rd party software constraints. They asked if it would be possible to run a CentOS-4 based container instead of a full VM. While obviously migrating from CentOS-4 to a more recent (and supported) version would be preferable, there are some benefits to migrating a CentOS 4 system to a Docker container. I played around with this off and on over the weekend, and finally came up with something fairly functional. I immediately destroyed it so there could be no evidence linking me to this activity.

The basics for how I accomplied this are listed below. They are terrible. Please do NOT follow them.

Disable selinux on your container host.

Look, I told you this was terrible. Dan Walsh and Vaclav Pavlin of Red Hat were kind enough to provide us patches for SELinux in CentOS-6, and then again for CentOS-5. I’m not going to repay their kindness by dragging them into this mess too. Dan is a really nice guy, please don’t make him cry.

The reason we disable selinux is explained on the CentOS-Devel mailing list. Since there’s no patch for CentOS-4 containers, selinux has to be disabled on the host for things to work properly.

Build a minimal vm.

Initially I tried running a slightly modified version of our CentOS-5 kickstart file for Docker through the usual build process. This mostly worked, however it was somewhat unreliable. The build process did not always exit cleanly, often leaving behind broken loop objects I couldn’t unmount. The resulting container worked, but had no functional rpmdb. The conversion trick used with CentOS-5 didn’t work properly with CentOS-4, even accounting for version differences.

I finally decided to build a normal vm image using virt-install. You could use virt-manager to do this part, it really doesn’t matter. There have been a number of functional improvements to anaconda over the years, and going back to the CentOS-4 installer hammers this home. I had to adjust my kickstart to use the old format, removing several more modern options I’d taken for granted. I ended up with the following. For this install, I made sure to install to an image file for easy extraction later on.

url --url=http://vault.centos.org/4.9/os/x86_64/
lang en_US.UTF-8
network --device=eth0 --bootproto=dhcp
rootpw --iscrypted $1$UKLtvLuY$kka6S665oCFmU7ivSDZzU.
authconfig --enableshadow
selinux --disabled
timezone --utc UTC

clearpart --all --initlabel
part / --fstype ext3 --size=1024 --grow

dd if=/dev/urandom count=50 | md5sum | passwd --stdin root
passwd -l root

rpm -q grub redhat-logos
rm -rf /boot
rm -rf /etc/ld.so.cache

Extract to tarball

Because we’re wiping out /boot and locking the root user, this image really won’t be useful for anything except converting to a container. The next step is to extract the contents into a smaller archive we can use to build our container. In order to do this, we’ll use the virt-tar-out command. This image is not going to be as small as the regular CentOS containers in the Docker index. This is partly due to rpm dependencies, and partly to how the image is created. Honestly, if you’re doing this, a few megs of wasted disk space is the least of your worries.

virt-tar-out -a /path/to/centos-4.img / - | xz --best > /path/to/centos-4-docker.tar.xz

Building the Container

At this point we have enough that we could actually just do a cat centos-4-docker.tar.xz | docker import - centos4, but there are still a few cleanup items that need to be addressed. From here, a basic Dockerfile that provides a vew changes is in order. Since CentOS-4 is End-of-Life and no longer served via the mirrors, the contents of /etc/yum.repos.d/ need to be modified to point to your local mirror as well as /etc/sysconfig/rhn/sources if you intend to still use the up2date utility. To do this, copy your existing yum repo files and sources from your working CentOS-4 systems into the directory with the container tarball, and use a Dockerfile similar to the one below.

FROM scratch
MAINTAINER you <your@emailaddress.com>
ADD centos-4-docker.tar.xz /
ADD CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo
ADD sources /etc/sysconfig/rhn/sources


All that’s left now is to run docker’s build command, and you have a successfully built a CentOS-4 base container to use for migration purposes, or just to make your inner sysadmin cry. Either way. This is completely unsupported. If you’ve treated this as a how-to and followed the steps, I would recommend the following actions:

  1. Having a long think about the decisions in your life that led you to this moment
  2. Drinking
  3. Sobbing uncontrollably
  4. Apologizing to everyone around you.

Reproducible CentOS containers

Around 9 months ago, I took over the creation of the official CentOS images contained in the Docker index. Prior to that, the index officially had one lonely and outdated CentOS 6.4 image that we at the CentOS Project were unaware of. Docker sort of exploded into our view and we spent a bit of time playing catch-up, trying to get things done the way we as a distribution would like to see them done. One of these actions was to do away with minor-versioned containers.

We chose to drop the minor version from our containers, and petitioned the Docker registry maintainer to remove the existing one (not built by us). The reasoning for this is fairly straightforward: A larger percentage of users never updated to the current containers, and so most of the bugs submitted to us were for older versions. Even recently Dan Walsh has had to post reminders to run updates. By only having a centos6 or centos7 image, we tried to remove the mindset for minor versions. Since the containers themselves are a svelte 132 packages, they amount to little more than the dependencies needed for bash and yum. In theory, the differences between a 6.5 image and a 6.6 image should be entirely negligible. In fact, by default any package installed on a 6.5 container would be from the 6.6 repositories.

That said, the number one request since we stopped shipping point releases is… you guessed it: Point releases. While I continue to maintain that our position of updates is the proper one, real world usage often runs counter to ivory tower thinking. A number of valid use cases were brought up in the course of discussions with community members asking for containers to be tagged with minor point releases, and I have agreed to reconsider tagging minor version images.

Beginning with the January monthly rollout, I will add minor tags for the 5, 6 builds in the Docker index. The minor tags will be for 5.11, 6.6. For 7 builds it will correspond to date tagged build name, the same as the installation media. These tags will be built from, and correspond to the respective CentOS installation media, and so will not contain updates. This means if you are using the minor tags, you could potentially / would be exposing your containers to exploits that have been patched in the rolling updates. The latest, 5, 6, and 7 tags will continue to point to the rolling monthly releases, which I would highly recommend using.


CentOS, Docker, and Systemd

Over the last few weeks, we’ve been asked about using systemd inside the CentOS-7 Docker containers for more complex operation. Because systemd offers a number of rather nice features, I can completely understand why people want to use it rather than pulling in outside tools like supervisord to recreate what already exists in centos by default. Unfortunately it’s just not that easy.

There are a couple major reasons why we don’t include systemd by default in the base Docker image. Dan Walsh covered these pretty completely in a blog post, but to recap where we are currently lets hit the highlights

  • systemd requires the CAP_SYS_ADMIN capability. This means running docker with --privileged. Not good for a base image.
  • systemd requires access to the cgroups filesystem.
  • systemd has a number of unit files that don’t matter in a container, and they cause errors if they’re not removed

It’s for these reasons that we ship with fakesystemd in the default image. The fakesystemd package provides dependency resolution and the proper directory structure so that packages install normally, and individual apps can be run inside the container by default. The fakesystemd package isn’t really an elegant fix, but it’s currently the best we can do in the base images. As soon as we’re able to rip it out and ship a proper systemd package, we will.

If you’re okay with running your container with --privileged, then you can follow the steps below to create your systemd enabled docker image from the CentOS-7 base image.

Dockerfile for systemd base image

FROM centos:centos7
MAINTAINER "you" <your@email.here>
ENV container docker
RUN yum -y swap -- remove fakesystemd -- install systemd systemd-libs
RUN yum -y update; yum clean all; \
(cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i ==
systemd-tmpfiles-setup.service ] || rm -f $i; done); \
rm -f /lib/systemd/system/multi-user.target.wants/*;\
rm -f /etc/systemd/system/*.wants/*;\
rm -f /lib/systemd/system/local-fs.target.wants/*; \
rm -f /lib/systemd/system/sockets.target.wants/*udev*; \
rm -f /lib/systemd/system/sockets.target.wants/*initctl*; \
rm -f /lib/systemd/system/basic.target.wants/*;\
rm -f /lib/systemd/system/anaconda.target.wants/*;
VOLUME [ "/sys/fs/cgroup" ]
CMD ["/usr/sbin/init"]

This Dockerfile swaps out fakesystemd for the real deal, but deletes a bunch of the unit files we don’t need. Building this image gives us a usable base to start from.

docker build --rm -t centos7-systemd . 

systemd enabled app container

Once this new base image is built, we can move on to building stuff that actually needs systemd. In this instance we’ll use httpd as an example.

FROM centos7-systemd
RUN yum -y install httpd; yum clean all; systemctl enable httpd.service
CMD ["/usr/sbin/init"]

We build once again:

docker build --rm -t centos7-systemd/httpd

To put this all together and run httpd with systemd in Docker, we do this:

docker run –privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 centos7-systemd/httpd

This container is running with systemd in a limited context, but it must always be run as a privileged container with the cgroups filesystem mounted.

There are plenty of rumors circulating that this will be addressed in the future, either in system or in a systemd-like subpackage. As soon as we’re able to ship systemd in the base image, we will do so.