/ devops

DevOps: Truncate Docker container logs periodically to free up server disk space

I've been disturbed by having my services unavailable on my machine because of "no space left on device". Every time I was forced to do some df -h / and du -h / commands and dig down to what leads to the lack of disk space.

Today I again encountered this issue and I'm done with it. I decided to find out what has been swallowing my disk space.

$ history
55  du -sh /*
56  du -sh /var
57  du -sh /var/*
58  du -sh /var/lib/*
59  du -sh /var/lib/docker/*

(be sure to sudo su first if you want to use du utility)

I ran du -ch /var/lib/docker/containers/*/*-json.log,  and I got an interesting result:

$ sudo sh -c "du -ch /var/lib/docker/containers/*/*-json.log"
13G    /var/lib/docker/containers/../..-json.log

Holy crap. That's a log file of 13 gigabytes. I then moved to Google on how to clear Docker logs because there's no official implementation of so. The fastest and cleanest way is sudo sh -c 'truncate -s 0 /var/lib/docker/containers/*/*-json.log', from https://stackoverflow.com/a/43570083/2465955.

Problem solved. The next thing is how do I prevent this from happening again in the future? And under the previous linked Stackoverflow answer, here's a quick answer to my next question: Rotate the log periodically.

Create /etc/logrotate.d/docker-logs, and add the following to the file:

/var/lib/docker/containers/*/*.log {
 rotate 7

And if everything is fine, logrotate.d will do the jobs.

Birkhoff Lee

Birkhoff Lee

I'm Birkhoff Lee, a developer since the age of 7. I'm still learning cyber security and site reliability engineering. Recently having Harvard CS50.

Read More