Python logger in the hierarchy under root and over my project loggers?

Question

First of all, I have read real python article on the subject.

Having learnt that loggers have a hierarchy, I want to create this new one called MyProjectLogger in such a hierarchy:

Root logger
    · MyProjectLogger
         · File logger 1
         · File logger 2
         · ... and so on with all the loggers in my project...

so that MyProjectLogger is used for all the descendant loggers, because right now I'm using all the same handlers with same configuration in all the loggers in my project (quite a lot). Although doing it through an only method, it doesn't feel right. In this way I would add the handlers only once to MyProjectLogger and all descendant loggers would just go up in the hierarchy using MyProjectLogger's handlers.

I don't want to use the default root logger for this because I have some third party libraries which are logging on it and right now I want the loggers in my project to log separately from the loggers in the libraries.

So, in summary:

• I want to define a logger MyProjectLogger in the hierarchy
• I want it to be direct descendant of root logger
• I want it to be the parent of all the loggers in my source code
• I believe I should use propagate=False so I can add the handlers to MyProjectLogger and getting it to handle of descendant loggers

My only doubt is: how do I give it such a name so that is under root and over the rest?

I know that:

logging.getLogger()  # Gets the root logger
logging.getLogger(__name__)  # Gets a logger for the present file
logging.getLogger(__package__)  # Gets a logger for the present module

so let's say, if my project has this folder layout:

aaaBot/
   main.py  # Only file visible in this example.
            # Please assume the rest of folders have files
   common/
      utils/ 
   config/
   database/
   exceptions/
   model/
   wizards/

In every file for each folder I use logging.getLogger(__name__). __package__ in the root returns None and in the main executable main.py __name__ is '__main__'.

Should I add a prefix + '.' for all the loggers in my project and create MyProjectLogger with that prefix (like getLogger(prefix+'.'))?

If not, what should I do?

Answer 1

Here's a working example illustrating the logger hierarchy mimicking the module structure:

so-57021706
└── aaaBot
    ├── __init__.py
    ├── common
    │   ├── __init__.py  # empty
    │   └── utils.py
    └── main.py

Source

aaaBot/__init__.py:

import logging
import sys


PKG_LOGGER = logging.getLogger(__name__)


def setup_logging():
    msg_format = '%(asctime)s [%(levelname)8s] %(message)s (%(name)s - %(filename)s:%(lineno)s)'
    date_format = '%Y-%m-%d %H:%M:%S'
    formatter = logging.Formatter(fmt=msg_format, datefmt=date_format)
    console_handler = logging.StreamHandler(stream=sys.stdout)
    console_handler.setLevel(logging.DEBUG)
    console_handler.setFormatter(formatter)
    PKG_LOGGER.addHandler(console_handler)
    PKG_LOGGER.setLevel(logging.DEBUG)
    PKG_LOGGER.propagate = False
    PKG_LOGGER.info('finished logging setup!')

aaaBot/common/utils.py:

import logging


UTILS_LOGGER = logging.getLogger(__name__)


def spam():
    UTILS_LOGGER.debug('entered spam() function')
    output = 'eggs'
    UTILS_LOGGER.debug('leaving spam() function')
    return output

aaaBot/main.py:

import sys
from aaaBot import setup_logging
from aaaBot.common.utils import spam


if __name__ == '__main__':
    if sys.argv[-1] == '--with-logging':
        setup_logging()
    print(spam())

Execution

Normal run:

$ python -m aaaBot.main
eggs

Debug run (turns logging on):

$ python -m aaaBot.main --with-logging
2019-07-15 13:16:04 [    INFO] finished logging setup! (aaaBot - __init__.py:18)
2019-07-15 13:16:04 [   DEBUG] entered spam() function (aaaBot.common.utils - utils.py:8)
2019-07-15 13:16:04 [   DEBUG] leaving spam() function (aaaBot.common.utils - utils.py:10)
eggs

Explanation

In this example project, the PKG_LOGGER under aaaBot/__init__.py is the "project" logger, having the name aaaBot. It's also the one and only logger that is configured; all the child loggers do nothing else than propagate the records up to PKG_LOGGER. The example of a child logger is UTILS_LOOGER from aaaBot/common/utils.py - not configured and having the name aaaBot.common.utils. The hierarchy in this case is:

root logger           code: logging.getLogger()
                      name: "root"
                      configured: no
                      propagates to parent: no (already root)
└── PKG_LOGGER        code: logging.getLogger('aaaBot')
                      name: "aaaBot"
                      configured: yes
                      propagates to parent: no (because of propagate = False)
    └── UTILS_LOGGER  code: logging.getLogger('aaaBot.common.utils')
                      name: "aaaBot.common.utils"
                      configured: no
                      propagates to parent: yes (default behaviour)

The configuration possibilities are endless and depend on your particular use case. For example, you can configure the root logger only and make sure all loggers propagate (they do that by default anyway). This will also print all messages from third-party libraries you are using, should they want to log anything. Or you can introduce an additional logger aaaBot.common that will write records from child loggers to file, in addition to propagating and emitting them to console etc.

Regarding your comment:

It seems like that thing would be ok for developing a library, but my project is an application.

It doesn't matter; it only depends on the value of the __name__ variable (importing a module versus executing it). For example, using logging.getLogger(__name__) in aaaBot/main.py will not create a logger with the name aaaBot.main with aaaBot as parent. Instead, a logger named __main__ will be created.

The difference in logging setup between a library and an application is only the logger configuration. An application always configures logging explicitly if in need of logging. A library doesn't configure any logging at all, except a NullHandler to the library's root logger. For example, if aaaBot would be a library, the logging setup in aaaBot/__init__.py could look like this:

import logging
LIB_LOGGER = logging.getLogger('aaaBot')
if not LIB_LOGGER.handlers:
    LIB_LOGGER.addHandler(logging.NullHandler())