Docker – Clean Up After Yourself!

UPDATE:

As pointed out by several readers, newer version of Docker Engine ( > 1.13) now provides the following single command to do all this.

docker system prune -a

Please be careful when you execute this command because this will delete and remove any stopped containers / unused images.

 

We started dockerizing some of our applications recently, and I got to say, that I am in love with Docker! It’s an awesome piece of engineering, and on AWS EC2, it made our lives much easier. However, one problem that we came across when we used Docker is the insane disk usage of it. We run Docker on Amazon Linux, and we have a build server that builds docker images as part of our build pipeline. Once built, the image is then pushed to our servers through an Ansible playbook. Probably I will blog more about it when I get a chance.

What we noticed was that over time, docker seem to eat up the disk space of the host. A quick df -h showed that /var/lib/docker was growing to the point where it pretty much covered the entire disk. So we looked around for a solution and came across the following.

A Word of Caution : commands below will delete stopped containers and wipe out their data. So make sure that you don’t run these if you have any containers / valuable data that you need to backup.

1. Make sure that exited containers are deleted.

When a docker container exits, the container is not deleted automatically. You can see all of the containers with ‘docker ps -a’ command. To clean up the exited containers, the command to use is as follows.

docker rm -v $(docker ps -a -q -f status=exited)

This will remove the exited containers. The -v flag is there to remove any containers that will no longer be needed. If there are no such containers you will see an error message like below.

docker: “rm” requires a minimum of 1 argument. See ‘docker rm –help’.

It just means that there’s nothing to delete and you are good to go.

2. Remove unwanted ‘dangling’ images.

Docker keeps all of the images that you have used in the disk, even if those are not actively running. This is done so that it has the necessary images in the local ‘cache’. This is awesome because when you want to pull an image that depends on those, or when you are building an image, all of these are locally available. The bad news is that this eats up disk space!

The command to remove these unwanted images is:

docker rmi $(docker images -f "dangling=true" -q)

Again, you might get an error message that the rmi commands needs an argument if you don’t have any such images.

3. Still not enough space? What is this ‘vfs’ directory?

If your docker directory still takes up space, that probably means that you have unwanted volumes filling up in your disk. The -v flag we passed to rm command usually takes care of this. But sometimes, if your way of shutting down containers do not auto remove the containers, the vfs directory will grow pretty fast. We can reclaim this space by deleting the unwanted volumes. To do this, there’s a docker image that you can use!

Note: If you are using a Docker version that is newer than 1.9, you don’t have to use the below container. Instead, use the following in built docker command.

docker volume rm $(docker volume ls -qf dangling=true)

Here’s how to run it.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

If you want to be safe and see what it’s going to delete, just pass –dry-run as an argument.

When that program runs, it will do the needful to delete all unwanted volumes and you should have your disk space back.

4. That’s all good. Do I have to do this everytime?

Then next question we had was while all that is good, we had to manually run it whenever our servers get filled up. So we decided to automate it. And it is a breeze with crontabs. Just drop all of the above commands into a file under /etc/cron.daily/ directory. We created a file named docker-clean in that directory with execute rights. The file contains the following.

docker rm -v $(docker ps -a -q -f status=exited)

docker rmi $(docker images -f "dangling=true" -q)

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

Linux will run this job every day automatically and clean up after Docker. I personally think that this should be baked into the docker daemon as a housekeeping feature. Nevertheless, kudos to docker team for building such an awesome tool.

42 comments

  1. AWS charges too much for space to let expended docker containers take up all my virtual drive space.

    Your advice and command examples were a real life saver for me. Thanks for volunteering your information.

  2. To avoid the ‘docker: “rm” requires a minimum of 1 argument. See ‘docker rm –help’.’ error you can use xargs with – r (–no-run-if-empty):

    $(docker ps -a -q -f status=exited) | xargs -r docker rm -v

  3. I’ve seen the command `docker rm -v $(docker ps -a -q -f status=exited)` posted in a few variants in a couple of places but I actually find this to be destructive if you use data containers. These are marked as exited and show up in the list returned by the innermost `docker ps …` command. I’ve nuked data container accidentally that way since it was still sitting in `/var/lib/docker/volumes`.

    The reason it was still there was that I didn’t use the `-v` flag in `docker rm -v` as you suggest. That will actually wipe the data when the container gets removed.

    Seeing how your post shows up high on Google when searching for ‘docker clean up’ It’s probably worth pointing that out 🙂

    I haven’t managed to find a way to list exited containers that aren’t data volumes (if that is even possible)

    1. There needs to be a big red warning above that command, or docker should warn before doing it…

      I just wiped out my postgres database trying to free up some space (thank FSM that it was my local dev environment, too bad I did lose some things from it…)

  4. Whoops, I messed up the first paragraph:

    I’ve seen the command `docker rm -v $(docker ps -a -q -f status=exited)` posted in a few variants in a couple of places but I actually find this to be destructive if you use data containers. These are marked as exited and show up in the list returned by the innermost `docker ps …` command. I’ve nuked data container accidentally that way but didn’t lose data since it was still sitting in `/var/lib/docker/volumes`.

  5. docker-cleanup-volumes did the trick and worked like a charm !

    Thanks Yohan for such an informative post. I was struggling with “No Space left on Device”
    issue from the past many days and finally got a solution.

  6. If you want to avoid too few argument errors when there is nothing to clean-up you should use xargs. e.g.

    docker images -q –filter “dangling=true”|xargs -r docker rmi

  7. Is this normal?

    -bash-4.2$ docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker –rm martin/docker-cleanup-volumes
    Unable to find image ‘martin/docker-cleanup-volumes:latest’ locally
    latest: Pulling from martin/docker-cleanup-volumes
    e6c44a677827: Pull complete
    e665190e5bfd: Pull complete
    3f134245a2ac: Pull complete
    Digest: sha256:0cae7e8ac0cbe678c9458f736dea727d5439d92c558993421687c5dda0624e9d
    Status: Downloaded newer image for martin/docker-cleanup-volumes:latest

    Delete unused volume directories from /var/lib/docker/volumes

    Directory /var/lib/docker/vfs/dir does not exist or is empty, skipping.

  8. Thank you very much for this helpful post. I rejoiced after running docker rmi and saw all of those images disappearing. Now my root directory is at 59% instead of 99%, whew! I have added these helpful commands to my daily cron job. thanks!!!

  9. `docker image prune` will remove all dangling images, which means all the images that currently not used by any containers(alive or dead) will be removed.

    This is only be supported by docker engine 1.13 above

  10. People who still stumble across this should note that as of January 2017, one can use this command to do cleanup more easily:
    docker system prune

    There are prune commands for other docker areas as well, like image, container, volume, etc.
    But system takes care of multiple things at once.

Leave a Reply

Your email address will not be published. Required fields are marked *