Django - Loggers are overwriting the previous log file along with the new one

Question

I am using Django logger for logging along with gunicorn workers. Currently, I am testing with 3 workers and I am using the TimedRotatingFileHandler. I configured when = m and interval = 1, I tried sending some 1000 requests and it is writing in the new file and overwriting the old file as well.

Here is my config from settings.py

'debug': {
            'level': 'DEBUG',
            'filename': BASE_DIR + '/Log/debug.log',
            'class': 'logging.handlers.TimedRotatingFileHandler',
            'when': 'M',
            'interval': 1,
            'formatter': 'verbose'
        },

This will happen only if i use more than 1 gunicorn worker. But if I use 1 gunicorn worker this is not happening.

Answer 1

You shouldn't be using file-based handlers directly from multiple processes concurrently, which is what you get if you use gunicorn with this configuration. Instead, use a SocketHandler or QueueHandler to send logs from the webapp to a separate process which logs them to disk. The Python logging cookbook has more information and example code, which is too long to reproduce here.

Writing to a single file from multiple concurrent processes without them treading on each other isn't supported in Python (and a lot of other languages too).