Log rotating with a Bash script

Okay, here's an idea, inspired by http://en.wikibooks.org/wiki/Bourne_Shell_Scripting/Files_and_streams

1. make a named pipe:

   mkfifo /dev/mypipe

2. redirect stdout and stderr to the named pipe:

   &> /dev/mypipe

3. read from mypipe into a file:

   cat < /dev/mypipe > /var/log/log.txt &

4. when you need to log-rotate, kill the cat, rotate the log, and restart the cat.

Now, I haven't tested this. Tell us how it goes.

Note: you can give the named pipe any name, like /var/tmp/pipe1 , /var/log/pipe , /tmp/abracadabra , and so on. Just make sure to re-create the pipe after booting before your logging-script runs.

Alternatively, don't use cat, but use a simple script file:

#!/bin/bashwhile : ; do  read line  printf "%s\n" "$line"done

This script guarantees an output for every newline read. (cat might not start outputting until its buffer is full or it encounters an EOF)

Final -- and TESTED -- attempt

IMPORTANT NOTE: Please read the comments from @andrew below. There are several situations which you need to be aware of.

Alright! Finally got access to my Linux box. Here's how:

Step 1: Make this recorder script:

#!/bin/bashLOGFILE="/path/to/log/file"SEMAPHORE="/path/to/log/file.semaphore"while : ; do  read line  while [[ -f $SEMAPHORE ]]; do    sleep 1s  done  printf "%s\n" "$line" >> $LOGFILEdone

Step 2: put the recorder into work:

1. Make a named pipe:

   mkfifo $PIPENAME

2. Redirect your application's STDOUT & STDERR to the named pipe:

   ...things... &> $PIPENAME

3. Start the recorder:

   /path/to/recorder.sh < $PIPENAME &

   You might want to nohup the above to make it survive logouts.

4. Done!

Step 3: If you need to logrotate, pause the recorder:

touch /path/to/log/file.semaphoremv /path/to/log/file /path/to/archive/of/log/filerm /path/to/log/file.semaphore

I suggest putting the above steps into its own script. Feel free to change the 2nd line to whatever log-rotating method you want to use.

Note : If you're handy with C programming, you might want to make a short C program to perform the function of recorder.sh. Compiled C programs will certainly be lighter than a nohup-ed detached bash script.

Note 2: David Newcomb provided a helpful warning in the comments: While the recorder is not running then writes to the pipe will block and may cause the program to fail unpredictably. Make sure the recorder is down (or rotating) for as short time as possible.

So, if you can ensure that rotating happens really quickly, you can replace sleep (a built-in command which accepts only integer values) with /bin/sleep (a program that accepts float values) and set the sleep period to 0.5 or shorter.

First of all, you really should not reinvent the square wheel here. Your peers are probably against rotating the logs on daily schedule which automatically applies to all scripts in /etc/logrotate.d/ - this can be avoided by placing the script elsewhere.

Now, the standard approach to log rotation (that is implemented in logrotate) can be implemented by any other facility just as well. E.g. here's a sample implementation in bash:

MAXLOG=<maximum index of a log copy>for i in `seq $((MAXLOG-1)) -1 1`; do    mv "log."{$i,$((i+1))}    #will need to ignore file not found errors heredone mv log log.1    # since a file descriptor is linked to an inode rather than path,                #if you move (or even remove) an open file, the program will continue                #to write into it as if nothing happened                #see https://stackoverflow.com/questions/5219896/how-do-the-unix-commands-mv-and-rm-work-with-open-files<make the daemon reopen the log file with the old path>

The last item is done by sending SIGHUP or (less often) SIGUSR1 and having a signal handler in the daemon that replaces the corresponding file descriptor or variable. This way, the switch is atomic, so there's no interruption in logging availability. In bash, this would look like:

trap { exec &>"$LOGFILE"; } HUP

The other approach is to make the writing program itself keep track of the log size each time it writes to it and do the rotation. This limits your options in where you can write to and what rotation logic is to what the program itself supports. But it has the benefit of being a self-contained solution and checking the log size at each write rather than on schedule. Many languages' standard libraries have such a facility. As a drop-in solution, this is implemented in Apache's rotatelogs:

<your_program> 2>&1 | rotatelogs <opts> <logfile> <rotation_criteria>

You can also pipe your output thru Apache rotatelogs utility.Or following script:

#!/bin/ksh#rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]numberOfFiles=10while getopts "n:fltvecp:L:" opt; do    case $opt in  n) numberOfFiles="$OPTARG"    if ! printf '%s\n' "$numberOfFiles" | grep '^[0-9][0-9]*$' >/dev/null; then      printf 'Numeric numberOfFiles required %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$numberOfFiles" 1>&2      exit 1    elif [ $numberOfFiles -lt 3 ]; then      printf 'numberOfFiles < 3 %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$numberOfFiles" 1>&2    fi  ;;  *) printf '-%s ignored. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$opt" 1>&2  ;;  esacdoneshift $(( $OPTIND - 1 ))pathToLog="$1"fileSize="$2"if ! printf '%s\n' "$fileSize" | grep '^[0-9][0-9]*[BKMG]$' >/dev/null; then  printf 'Numeric fileSize followed by B|K|M|G required %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$fileSize" 1>&2  exit 1fisizeQualifier=`printf "%s\n" "$fileSize" | sed "s%^[0-9][0-9]*\([BKMG]\)$%\1%"`multip=1case $sizeQualifier inB) multip=1 ;;K) multip=1024 ;;M) multip=1048576 ;;G) multip=1073741824 ;;esacfileSize=`printf "%s\n" "$fileSize" | sed "s%^\([0-9][0-9]*\)[BKMG]$%\1%"`fileSize=$(( $fileSize * $multip ))fileSize=$(( $fileSize / 1024 ))if [ $fileSize -le 10 ]; then  printf 'fileSize %sKB < 10KB. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$fileSize" 1>&2  exit 1fiif ! touch "$pathToLog"; then  printf 'Could not write to log file %s. rotatelogs.sh -n numberOfFiles pathToLog fileSize[B|K|M|G]\n' "$pathToLog" 1>&2  exit 1filineCnt=0while read linedo  printf "%s\n" "$line" >>"$pathToLog"  lineCnt=$(( $lineCnt + 1 ))  if [ $lineCnt -gt 200 ]; then    lineCnt=0    curFileSize=`du -k "$pathToLog" | sed -e 's/^[  ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g' | cut -f1 -d" "`    if [ $curFileSize -gt $fileSize ]; then      DATE=`date +%Y%m%d_%H%M%S`      cat "$pathToLog" | gzip -c >"${pathToLog}.${DATE}".gz && cat /dev/null >"$pathToLog"      curNumberOfFiles=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | wc -l | sed -e 's/^[   ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g'`      while [ $curNumberOfFiles -ge $numberOfFiles ]; do        fileToRemove=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | head -1`        if [ -f "$fileToRemove" ]; then          rm -f "$fileToRemove"          curNumberOfFiles=`ls "$pathToLog".[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]_[0-9][0-9][0-9][0-9][0-9][0-9].gz | wc -l | sed -e 's/^[   ][  ]*//' -e 's%[   ][  ]*$%%' -e 's/[  ][  ]*/[    ]/g'`        else          break        fi      done    fi  fidone