containerized nginx log rotation with logrotate containerized nginx log rotation with logrotate nginx nginx

containerized nginx log rotation with logrotate


You can get the process id from the Pid attribute using docker inspect and use kill -USR1 {pid} to have nginx reopen the logs.

Here's the /etc/logrotate.d/nginx file I created:

/var/log/nginx/access.log{    size 2M    rotate 10    missingok    notifempty    compress    delaycompress    postrotate        docker inspect -f '{{ .State.Pid }}' nginx | xargs kill -USR1    endscript}


If you want to run logrotate in a dedicated container (e.g to rotate both nginx logs and Rails' file log) rather than on the host machine, here's how I did it. The trickiest part by far was as above, getting the reload signals to nginx, Rails, etc so that they would create and log to fresh logfiles post-rotation.

Summary:

  • put all the logs on a single shared volume
  • export docker socket to the logrotate container
  • build a logrotate image with logrotate, cron, curl, and jq
  • build logrotate.conf with postrotate calls using docker exec API as detailed below
  • schedule logrotate using cron in the container

The hard part:

To get nginx (/etcetera) to reload thus connect to fresh log files, I sent exec commands to the other containers using Docker's API via socket. It expects a POST with the command in JSON format, to which it responds with an exec instance ID. You then need to explicitly run that instance.

An example postrotate section from my logrotate.conf file:

postrotate    exec_id=`curl -X POST --unix-socket /var/run/docker.sock \      -H "Content-Type: application/json" \      -d '{"cmd": ["nginx", "-s", "reopen"]}' \      http:/v1.41/containers/hofg_nginx_1/exec  \    | jq -r '.Id'`    curl -X POST --unix-socket /var/run/docker.sock \      -H "Content-Type: application/json" \      -d '{"Detach": true}' \      http:/v1.41/exec/"$exec_id"/startendscript

Commentary on the hard part:

  exec_id=`curl -X POST --unix-socket /var/run/docker.sock \

This is the first of two calls to curl, saving the result into a variable to use in the second. Also don't forget to (insecurely) mount the socket into the container, '/var/run/docker.sock:/var/run/docker.sock'

  -H "Content-Type: application/json" \  -d '{"cmd": ["nginx", "-s", "reopen"]}' \

Docker's API docs say the command can be a string or array of strings, but it only worked for me as an array of strings. I used the nginx command line tool, but something like 'kill -SIGUSR1 $(cat /var/run/nginx.pid)' would probably work too.

  http:/v1.41/containers/hofg_nginx_1/exec  \

I hard-coded the container name, if you're dealing with something more complicated you're probably also using a fancier logging service

  | jq -r '.Id'`

The response is JSON-formatted, I used jq to extract the id (excuse me, 'Id') to use next.

  curl -X POST --unix-socket /var/run/docker.sock \      -H "Content-Type: application/json" \      -d '{"Detach": true}' \

The Detach: true is probably not necessary, just a placeholder for POST data that was handy while debugging

  http:/v1.41/exec/"$exec_id"/start

Making use of the exec instance ID returned by the first curl to actually run the command.

I'm sure it will evolve (say with error handling), but this should be a good starting point.