Scrapy Logging: configure logging to rewrite log file

Question

Here is my Scrapy code ... which I copied from Scrapy official documentation

import logging
from scrapy.utils.log import configure_logging

configure_logging(install_root_handler=False)
logging.basicConfig(
    filename='log.txt',
    format='%(levelname)s: %(message)s',
    level=logging.INFO,
    filemode = 'w'
)

Now the issue is that even though the error threshold is 'INFO' its still printing 'DEBUG' messages in the log.

Does anyone why this is happening? Is there any other setting somewhere else I need to configure?

Also, If I set the following flags in settings.py file, it works ok (as expected) ...

LOG_FILE = 'mylog.txt'
LOG_LEVEL = 'INFO'

I don't see any 'DEBUG' messages printed in the log. But then it just appends to the log file every time crawler is run. I need to clear the file and write again on every crawl.

Thanks

Answer 1

you can log all scrapy logs to file by first disabling root handle in scrapy.utils.log.configure_logging and then adding your own log handler and customize log level (DEBUG to INFO) and formatter as required.

In settings.py file of scrapy project add the following code:

import logging
from logging.handlers import RotatingFileHandler

from scrapy.utils.log import configure_logging

LOG_ENABLED = False
# Disable default Scrapy log settings.
configure_logging(install_root_handler=False)

# Define your logging settings.
log_file = '/tmp/logs/CRAWLER_logs.log'

root_logger = logging.getLogger()
root_logger.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
rotating_file_log = RotatingFileHandler(log_file, maxBytes=10485760, backupCount=1)
rotating_file_log.setLevel(logging.INFO)
rotating_file_log.setFormatter(formatter)
root_logger.addHandler(rotating_file_log)

Hope this helps!