What’s eating my disk? Docker System Commands explained
You’ve been using Docker for quite a while now and it looks like it’s eating your entire disk space. Wasn’t docker supposed to be fixing all my problems?Luckily there is a solution that arrived in Docker 1.13 and with a couple of simple commands to get an overview on your entire Docker environment and clean it all up.
To show you how quickly a disk can fill up while using Docker and you’re not paying attention, I’m going to give a quick example and to do that I will use my favorite sandboxing tool play-with-docker.com. Create a new instance by click the Add new instance button. This will instantiate a fresh node with the current latest version, 17.03. Since we’re talking ‘disk space’, let’s check out the initial state of our disks:
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev/mapper/... 10.0G 443.3M 9.6G 4% /
tmpfs 60.0G 0 60.0G 0% /dev
tmpfs 60.0G 0 60.0G 0% /sys/fs/cgroup
/dev/xvda1 49.1G 3.7G 43.3G 8% /etc/resolv.conf
/dev/xvda1 49.1G 3.7G 43.3G 8% /etc/hostname
/dev/xvda1 49.1G 3.7G 43.3G 8% /etc/hosts
shm 64.0M 0 64.0M 0% /dev/shm
/dev/mapper/... 10.0G 443.3M 9.6G 4% /graph/overlay2So on a freshly spawned PWD node we get 10GB allocated of which almost 500MB is used. This is our baseline.
Now I’m going to create a Dockerfile, that will build an image from the Alpine base image and in this image I’m writing three random files that each consist of 1 block with block size 1GB, by utilizing the dd command, taking up a total of 3 gigabytes. Since I’m not planning to do anything usefull with this image, the CMD simply defaults to /bin/true.
FROM alpineRUN dd if=/dev/zero of=1g1.img bs=1G count=1
RUN dd if=/dev/zero of=1g2.img bs=1G count=1
RUN dd if=/dev/zero of=1g3.img bs=1G count=1CMD /bin/true
Let’s build that with docker build -t test .. When this file is ready it should result in an image of three-gigabytes large.
$ docker image ls
REPOSITORY TAG CREATED SIZE
test latest 38 seconds ago 3.23GB
alpine latest 5 weeks ago 3.99MBAnd to no surprise that takes the same amount out of our system disk space, three gigabytes less than what I started with, 6.5GB.
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev/mapper/... 10.0G 3.4G 6.5G 34% /
Now, assume I made a mistake, instead of three files of one gigabyte, I actually wanted two… Well, I need to adjust my Dockerfile for that, remove one line. But to prove my point I want to make sure there is no layer caching going on in the build phase, so I will also add an extra command on top. Then build it again.
FROM alpineRUN echo foo
RUN dd if=/dev/zero of=1g1.img bs=1G count=1
RUN dd if=/dev/zero of=1g2.img bs=1G count=1
# RUN dd if=/dev/zero of=1g3.img bs=1G count=1CMD /bin/true
Very naively I could assume that now I’ve saved one gigabyte. But after checking my disk space that’s not true. I made it even worse!
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev/mapper/... 10.0G 5.4G 4.5G 54% /The older image is still lying around. So in the end I start piling up these images and they’re chipping away disk space like hungry hippos! To give you a good view on your usage within the Docker system, Docker 1.13 introduced a docker system df command, similar to the Linux shell command. It will give you a nice overview on everything that’s been going on in the Docker environment.
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 3 0 5.373GB 5.373GB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0BNo surprises here, we have 3 images on the system, alpine, the first ‘test’ image with 3 x 1GB files and the second ‘test’ with 2 x 1GB files, accounting for +5GB of disk space used and since none of the images was ever run as a container, there are no containers associated with these images, so 100% of the images can be cleaned up without breaking anything. Now let’s run our test image docker run test and check the system again.
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 3 1 5.373GB 3.225GB (60%)
Containers 1 0 0B 0B
Local Volumes 0 0 0B 0BThe story is a little bit different, because I ran 1 container, that exited immediately (it executed it’s CMD /bin/true) hence it is not active anymore. This results in the reclaimable amount of disk space being lower, because now an image that is tied to an existing container on the system, flagging it as active and it cannot be cleaned up anymore.
Now let’s get down to business and clean our system. Under the same system namespace Docker provides another command docker system prune, that will help you clean up dangling images, unused containers, stale volumes and networks.
$ docker system prune
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all dangling images
Are you sure you want to continue? [y/N] y
Deleted Containers:
1cdf866157b4a97e151125af3c2a7f186a59b6f63807e2014ce1a00d68f44e1dDeleted Images:
deleted: sha256:f59bb277...
deleted: sha256:695b8e70...
deleted: sha256:93b1cceb...
deleted: sha256:c74d6bcd...
deleted: sha256:df8b9bb1...
deleted: sha256:dfe8340f...
deleted: sha256:ce1ee654...Total reclaimed space: 3.221GB
After this commands greets us with a warning, it will remove all the stopped containers and dangling images. In our case, the intermediary image, the one with the 3 x 1GB files isn’t associated anymore, hence ‘dangling’, so it is pruned. Also all the intermediary images that were created while I was building my image get removed, and the total reclaimed space is about three gigabytes. Win!
To take it one step further, the prune command has a -a parameter to do a deep clean. When I run this, you’ll see that you get a stronger warning.
$ docker system prune -a
WARNING! This will remove:
- all stopped containers
- all volumes not used by at least one container
- all networks not used by at least one container
- all images without at least one container associated to them
Are you sure you want to continue? [y/N] y
Deleted Images:
untagged: test:latest
deleted: sha256:c515ebfa2...
deleted: sha256:07302c011...
deleted: sha256:37c0c6474...
deleted: sha256:5cc2b6bc4...
deleted: sha256:b283b9c35...
deleted: sha256:8a8b9bd8b...
untagged: alpine:latest
untagged: alpine@sha256:58e1a1bb75db1...
deleted: sha256:4a415e366...
deleted: sha256:23b9c7b43...Total reclaimed space: 2.151GB
By running this you’re basically cleaning up your entire system and only keep the things that are actually running on you r system, so be very aware of what you are doing when running this. For instance, you don’t want to run this prune -a command on a production server where you have some sidecar images idling (eg. scheduled backup or rollup, weekly exports, etc.), waiting to be executed every once in a while, because those will be cleaned up and need to be pulled in again when running your sidecar.
In this case, all images that had no container associated will be removed since there were no containers in my system anymore as they were cleaned up when we did the first prune.
$ df -h
Filesystem Size Used Available Use% Mounted on
/dev/mapper/... 10.0G 442.5M 9.6G 4% /So we are back to square one, with all disk space available. Now, this is just the tip of the iceberg, because once you start running actual applications and start allocating volumes and networks, your system might get polluted with stale resources even quicker. To learn more about how you can reclaim all those resources watch this video below (and while your at it, don’t forget to subscribe!) to see the things I talked about in this post in action.
On top of that I’ll dive into a how running a simple WordPress application with multiple containers, volumes and networks, can quickly blow up your system’s resources, and how to keep that under control.
