Django Celery Logging Best Practice

When your logger initialized in the beginning of "another module" it links to another logger. Which handle your messages. It can be root logger, or usually I see in Django projects - logger with name ''.

Best way here, is overriding your logging config:

LOGGING = {    'version': 1,    'disable_existing_loggers': True,    'formatters': {        'simple': {            'format': '%(levelname)s %(message)s',             'datefmt': '%y %b %d, %H:%M:%S',            },        },    'handlers': {        'console': {            'level': 'DEBUG',            'class': 'logging.StreamHandler',            'formatter': 'simple'        },        'celery': {            'level': 'DEBUG',            'class': 'logging.handlers.RotatingFileHandler',            'filename': 'celery.log',            'formatter': 'simple',            'maxBytes': 1024 * 1024 * 100,  # 100 mb        },    },    'loggers': {        'celery': {            'handlers': ['celery', 'console'],            'level': 'DEBUG',        },    }}from logging.config import dictConfigdictConfig(LOGGING)

In this case I suppose it should work as you assume.

P.S. dictConfig added in Python2.7+.

To fix duplicate logging issue, what worked for me is to set the propagate setting to false when declaring my settings.LOGGING dict

LOGGING = {    'version': 1,    'disable_existing_loggers': False,    'handlers': {        'console': {            'level': 'DEBUG',            'class': 'logging.StreamHandler',            'formatter': 'verbose'        },    },    'formatters': {        'verbose': {            'format': '%(asctime)s %(levelname)s module=%(module)s, '            'process_id=%(process)d, %(message)s'        }    },    'loggers': {        'my_app1': {            'handlers': ['console'],            'level': 'DEBUG',            'propagate': False #this will do the trick        },        'celery': {            'handlers': ['console'],            'level': 'DEBUG',            'propagate': True        },    }}

lets say your django project layout looks like:
my_project/
- tasks.py
- email.py

and lets say one of your tasks makes a call to some function in email.py; the logging will happen in email.py and then that logging will get propagated to the 'parent' which in this case happens to be your celery task. Thus double logging. But setting propagate to False for a particular logger means that for that logger/app, its logs wont get propagated to the parent, hence their will be no 'double' logging.By default 'propagate' is set to True

Here's a link to the django docs section about that parent/children loggers stuff

It is troubling that Celery interferes with the root logger (which is not best practice and can't be controlled completely), but it does not disable your app's custom loggers in any way, so use your own handler names and define your own behavior rather than trying to fix this issue with Celery. [I like to keep my application logging separate anyway). You could use separate handlers or the same for Django code and Celery tasks, you just need to define them in your Django LOGGING config. Add formatting args for module, filename, and processName to your formatter for sanity, to help you distinguish where messages originate.

[this assumes you have setup a handler for 'yourapp' in the LOGGING settings value that points to an Appender - sounds like you are aware of this though].

views.py

log = logging.getLogger('yourapp')def view_fun():    log.info('about to call a task')    yourtask.delay()

tasks.py

log = logging.getLogger('yourapp')@taskdef yourtask():    log.info('doing task')

For the logging that Celery generates - use the celeryd flags --logfile to send Celery output (eg, worker init, started task, task failed) to a separate place if desired. Or, use the other answer here that sends the 'celery' logger to a file of your choosing.

Note: I would not use RotatingFileHandlers - they are not supported for multi-process apps. Log rotation from another tool like logrotate is safer, same goes with logging from Django assuming you have multiple processes there, or the same log files are shared with the celery workers. If your using a multi-server solution you probably want to be logging somewhere centralized anyway.