How to limit log file size in python

Lose basicConfig() and use RotatingFileHandler:

import loggingfrom logging.handlers import RotatingFileHandlerlog_formatter = logging.Formatter('%(asctime)s %(levelname)s %(funcName)s(%(lineno)d) %(message)s')logFile = 'C:\\Temp\\log'my_handler = RotatingFileHandler(logFile, mode='a', maxBytes=5*1024*1024,                                  backupCount=2, encoding=None, delay=0)my_handler.setFormatter(log_formatter)my_handler.setLevel(logging.INFO)app_log = logging.getLogger('root')app_log.setLevel(logging.INFO)app_log.addHandler(my_handler)while True:    app_log.info("data")

When you use logging.basicConfig with a file, the log is attached with a file handler to handle writing to the file.afterwards you created another file handler to the same file with logging.handlers.RotatingFileHandler

Now, once a rotate is needed, RotatingFileHandler is trying to remove the old file but it can't becuase there is an open file handler

this can be seen if you look directly at the log file handlers -

import loggingfrom logging.handlers import RotatingFileHandlerlog_name = 'c:\\log.log'logging.basicConfig(filename=log_name)log = logging.getLogger()handler = RotatingFileHandler(log_name, maxBytes=1024, backupCount=1)log.addHandler(handler)[<logging.FileHandler object at 0x02AB9B50>, <logging.handlers.RotatingFileHandler object at 0x02AC1D90>]

To use BasicConfig and RotatingFileHandler, add RotatingFileHandler as Handler in BasicConfig.

main.py:

import loggingrfh = logging.handlers.RotatingFileHandler(    filename='foo.log',     mode='a',    maxBytes=5*1024*1024,    backupCount=2,    encoding=None,    delay=0)logging.basicConfig(    level=logging.DEBUG,    format="%(asctime)s %(name)-25s %(levelname)-8s %(message)s",    datefmt="%y-%m-%d %H:%M:%S",    handlers=[        rfh    ])logger = logging.getLogger('main')logger.debug("test")

other.py

import loggingclass Other():    def __init(self):        self.logger = logging.getLogger('other')        self.logger.info("test2")

"test" will be written into foo.log with the tag 'main'

"test2" will be written into foo.log with the tag 'other'