I've been disturbed by having my services unavailable on my machine because of "no space left on device". Every time I was forced to do some df -h / and du -h / commands and dig down to what leads to the lack of disk space.

Today I again encountered this issue and I'm done with it. I decided to find out what has been swallowing my disk space.

$ history
...
55  du -sh /*
56  du -sh /var
57  du -sh /var/*
58  du -sh /var/lib/*
59  du -sh /var/lib/docker/*
...

(be sure to sudo su first if you want to use du utility)

I ran du -ch /var/lib/docker/containers/*/*-json.log,  and I got an interesting result:

$ sudo sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
...
13G    /var/lib/docker/containers/../..-json.log
...

Holy crap. That's a log file of 13 gigabytes. I then moved to Google on how to clear Docker logs because there's no official implementation of so. The fastest and cleanest way is sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log', from https://stackoverflow.com/a/43570083/2465955.

Problem solved. The next thing is how do I prevent this from happening again in the future? And under the previous linked Stackoverflow answer, here's a quick answer to my next question: Rotate the log periodically.

Create /etc/logrotate.d/docker-logs, and add the following to the file:

/var/lib/docker/containers/*/*.log {
 rotate 7
 daily
 compress
 size=50M
 missingok
 delaycompress
 copytruncate
}

And if everything is fine, logrotate.d will do the jobs.