Docker – Clean Up After Yourself!

We started dockerizing some of our applications recently, and I got to say, that I am in love with Docker! It’s an awesome piece of engineering, and on AWS EC2, it made our lives much easier. However, one problem that we came across when we used Docker is the insane disk usage of it. We run Docker on Amazon Linux, and we have a build server that builds docker images as part of our build pipeline. Once built, the image is then pushed to our servers through an Ansible playbook. Probably I will blog more about it when I get a chance.

What we noticed was that over time, docker seem to eat up the disk space of the host. A quick df -h showed that /var/lib/docker was growing to the point where it pretty much covered the entire disk. So we looked around for a solution and came across the following.

1. Make sure that exited containers are deleted.

When a docker container exists, the container is not deleted automatically. You can see all of the containers with ‘docker ps -a’ command. To clean up the exited containers, the command to use is as follows.

docker rm -v $(docker ps -a -q -f status=exited)

This will remove the exited containers. The -v flag is there to remove any containers that will no longer be needed. If there are no such containers you will see an error message like below.

docker: “rm” requires a minimum of 1 argument. See ‘docker rm –help’.

It just means that there’s nothing to delete and you are good to go.

2. Remove unwanted ‘dangling’ images. ¬†

Docker keeps all of the images that you have used in the disk, even if those are not actively running. This is done so that it has the necessary images in the local ‘cache’. This is awesome because when you want to pull an image that depends on those, or when you are building an image, all of these are locally available. The bad news is that this eats up disk space!

The command to remove these unwanted images is:

docker rmi $(docker images -f "dangling=true" -q)

Again, you might get an error message that the rmi commands needs an argument if you don’t have any such images.

3. Still not enough space? What is this ‘vfs’ directory?

If your docker directory still takes up space, that probably means that you have unwanted volumes filling up in your disk. The -v flag we passed to rm command usually takes care of this. But sometimes, if your way of shutting down containers do not auto remove the containers, the vfs directory will grow pretty fast. We can reclaim this space by deleting the unwanted volumes. To do this, there’s a docker image that you can use!

Here’s how to run it.

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

If you want to be safe and see what it’s going to delete, just pass –dry-run as an argument.

When that program runs, it will do the needful to delete all unwanted volumes and you should have your disk space back.

4. That’s all good. Do I have to do this everytime?

Then next question we had was while all that is good, we had to manually run it whenever our servers get filled up. So we decided to automate it. And it is a breeze with crontabs. Just drop all of the above commands into a file under /etc/cron.daily/ directory. We created a file named docker-clean in that directory with execute rights. The file contains the following.

docker rm -v $(docker ps -a -q -f status=exited)

docker rmi $(docker images -f "dangling=true" -q)

docker run -v /var/run/docker.sock:/var/run/docker.sock -v /var/lib/docker:/var/lib/docker --rm martin/docker-cleanup-volumes

Linux will run this job every day automatically and clean up after Docker. I personally think that this should be baked into the docker daemon as a housekeeping feature. Nevertheless, kudos to docker team for building such an awesome tool.

[Note To Self] Spring Security Default Target URL

Just a quick note to self on Spring Security Default Target URLs. In one of my recent projects, I noticed that suddenly my Spring Security based login does not use the specified default target url in the configuration. Instead, it was hitting the root of the application always. This application was working perfectly fine until recently, and default target URL has not been changed since.

The Spring Security definition was:

<security:http auto-config="true">
      access="ROLE_LOGIN" />
      authentication-success-handler-ref="authenticationSuccessHandler" />
   <security:logout logout-url="/!/signout" logout-success-url="/!/signin" />

After debugging through Spring Security code, I noticed that the defaultTargetURL of¬†AbstractAuthenticationTargetUrlRequestHandler is not set to my value, but it uses the default ‘/’. Then after some digging up, it turned out that I’ve added a new Authentication Success Handler to my definition for a different purpose, and when an authentication-success-handler-ref is present in the configuration, the ‘default-target-url’ element in XML configuration is not used.

To fix this, the solution was  to specify the default target URL on my authentication success handler bean as follows.

<bean id="authenticationSuccessHandler" 
   <property name="defaultTargetUrl" value="/!/" />

The reason behind this is, the value we provide on the XML configuration goes to the default authentication success handler only. When we define our own, that value goes no where, so we need to specify it manually on the bean itself. This ate up about 15 mins of my time, before luckily noticing that the success handler change was the reason.


Mac Book Air & WiFi Routers

Two days back, I got my ADSL router fried due to a near field lightening strike, and I had to replace the router. I bought a D-Link DSL 2750u and configured it, my MBA happily connected, and everything was working fine.

This morning, when I woke up, I was in for a surprise, because suddenly my MBA stopped recognising the WiFi Network. It sees all other WiFi networks, connects without any hiccups to those, but not this one. I tried the typical first aids – restarting router, restarting my laptop, no luck. My mobile and other devices can connect to the WiFi network without any problems, but not the Mac Book Air. At random times, MBA sees the WiFi network, but when I try to connect it says ‘A Connection Timeout Has Occurred’ (WTH?).

After few minutes spent on Google, saw that Apple devices are known to have some problems with particular WiFi Channels. I checked the WiFi Channel on my router and saw that it was set to AUTO. I changed it to Channel 6, and voila – the MBA see’s and connects to the WiFi smoothly.

So if you are facing any WiFi issues on your Mac devices, try changing the Wireless Channel. On my D-Link Router, it was found in the Administration Console under Wireless -> Basic Settings.

DustJS – Getting ‘Duster’ to work on OS X / Linux

This is going to be a short post on one of the issues I came across when I was trying to use LinkedIn DustJS on my Mac Book. I was trying out DustJS for one of the new projects I’m working on, and I came across ‘DusterJS’ (, which is a watcher that could monitor your .dust files and convert them on-the-fly to compiled dust templates. This is a life saver when it comes to developing dust templates, and kudos to the author for writing such a great tool.
Continue reading

Integration Testing with MongoDB & Spring Data

Integration Testing is an often overlooked area in enterprise development. This is primarily due to the associated complexities in setting up the necessary infrastructure for an integration test. For applications backed by databases, it’s fairly complicated and time-consuming to setup databases for integration tests, and also to clean those up once test is complete (ex. data files, schemas etc.), to ensure repeatability of tests. While there have been many tools (ex. DBUnit) and mechanisms (ex. rollback after test) to assist in this, the inherent complexity and issues have been there always.

But if you are working with MongoDB, there’s a cool and easy way to do your unit tests, with almost the simplicity of writing a unit test with mocks. With ‘EmbedMongo’, we can easily setup an embedded MongoDB instance for testing, with in-built clean up support once tests are complete. In this article, we will walkthrough an example where EmbedMongo is used with JUnit for integration testing a Repository Implementation.
Continue reading