How to log to a single file with concurrent processes in Django without exclusive locks

An alternative would be for all Django logging to be sent to a queue (e.g. a Redis queue, using something like this, or a multiprocessing.Queue) and then a single process reads the queue and writes records to file. There are more moving parts, so this may or may not be appropriate for your needs, but it would eliminate the file contention. See this post for more options when using logging from multiple processes.

You can of course also set up a socket server and use a SocketHandler to send logging events from all Django processes to the server, which writes to file. The Python docs contain a working example of such a server.

A scalable pure-logging approach (as in using only system logging components) can be implemented using syslog-ng

syslog-ng is syslog++, so it will not break any syslog configurations and logging if you already have some. Installation on Ubuntu is straightforward:

sudo apt-get install syslog-ng

for installation on windows, refer:http://www.syslog.org/logged/running-syslog-ng-on-windows/

Multiple django applications or multiple servers log to syslog-ng which uses UDP or TCP (depending on your configuration) to send it to a central syslog-ng server which logs it on that machine. You can use a basic regex to identify the application in syslog-ng and route it accordingly.

Django logging configuration

'syslog-verbose': {    'format': 'myapp %(levelname)s %(asctime)s %(module)s %(process)d %(thread)d %(message)s'}'ConcurrentLogHandler':{    'level': 'DEBUG',    'class': 'logging.handlers.SysLogHandler',    'formatter': 'syslog-verbose',},

Note the "myapp" string added to the formatter to identify the application by syslog-ng.

You can even configure nginx, apache and other servers to log to syslog-ng. E.g. for apache:

CustomLog "| /usr/bin/logger -t 'apache' -u /var/run/apache_access_log.socket" combined

Syslog-ng client configuration

Append this to the end of /etc/syslog-ng/syslog-ng.conf

filter filter_loghostclient {    program("^myapp$");};destination dest_loghostclient {    tcp("destination_logserver.example.com" port (514));};log {    source(s_all);    filter(filter_loghostclient);    destination(dest_loghostclient);};source s_apache_access {    #apache access logs getting written directly to a socket (as described above)    unix-stream("/var/run/apache_access_log.socket" max-connections(512) keep-alive(yes)); };log{    source(s_apache_access);    destination(dest_loghostclient);};filter f_apache_err {    #Apache error logs    program("apache") and level(err); }; log{    source(s_all);    filter(f_apache_err);    destination(dest_loghostclient);};

Syslog-ng aggregator configuration

Append the following to the syslog-ng config file on destination_logserver.example.com

source src_loghostserver {    tcp(port(514) keep-alive(yes) max_connections(1000));};destination dest_loghostserver {    file("/var/log/myproject/request_\$R_YEAR\$R_MONTH\$R_DAY.log");};log {    source(src_loghostserver);    destination(dest_loghostserver);};