logging with the same logging levels across modules

Question

I'm somewhat new to Python, especially in writing modules and functions in several files and not just raw scripts.

I'm writing a command line app and I would like to have a single function (I call it argpconf) that will parse the command line arguments and set a log level accordingly. Most importantly, I'd like the log level to be set once in this function and to be the same across all modules with minimum overhead when creating their loggers. Additionally, I would like to be able to identify the module from which the message came while using the common formatter:

logging.Formatter("%(levelname)s : %(name)s : %(message)s")

Partially based on a cookiecutter template, I've created the following files:

├── api
│   ├── __init__.py
│   └── some_functionality.py
├── cli.py
├── core
│   ├── argpconf.py
│   ├── __init__.py
│   ├── logger.py
│   └── __version__.py
├── __init__.py
└── __main__.py

core/logger.py has the following content:

from logging import Formatter, Logger as _Logger, NullHandler, StreamHandler

class Logger(_Logger):
    def __init__(self, name=None):
        super(Logger, self).__init__(name or __name__.split(".")[0])
        self.addHandler(NullHandler())  # default to no output
    def start(self, level="WARN", stream=None,
              fmt="%(levelname)s : %(name)s : %(message)s"):
        handler = StreamHandler(stream)
        handler.setFormatter(Formatter(fmt))
        self.addHandler(handler)
        self.setLevel(level.upper())
    def stop(self):
        for handler in self.handlers[1:]:
            # Remove everything but the NullHandler.
            self.removeHandler(handler)

logger = Logger()

Compared to the ideas suggested in the answers to these questions:

• How to share a file between modules for logging in python
• Python logging - multiple modules
• How to use logging.getLogger(__name__) in multiple modules
• Using Python logging in multiple modules

I really like the approach taken in the cookiecutter template with the logger since it allows you just to import logger and have a logger object that it's log level is the same across all modules. However, I'm not totally satisfied with it, because in my case the argpconf.py is the first module that starts the logger so all log messages from all modules have their %(name)s substituted with core since it's argpconf.py's __name__.split(".")[0].

How can I improve the logger module so it will detect the module that called it and print log messages with module as the %(name)s and perhaps even the function that prints them?

Answer 1

This approach seems to be making things more complicated than they need to be. I realise this is coming from the cookiecutter template you used, and it's just my opinion, but the approach to logging taken in that template isn't what I would regard as best practice. You know your use case best, but if all I wanted was to

have a single function that will parse the command line arguments and set a log level accordingly. Most importantly, I'd like the log level to be set once in this function and to be the same across all modules with minimum overhead when creating their loggers. Additionally, I would like to be able to identify the module from which the message came while using the common formatter

then the simplest approach is to import argparse and logging in your main script, process command line arguments and set the logging level accordingly, call basicConfig() (as suggested in Brian M. Sheldon's comment) and then dispatch to your application endpoint as determined by command line arguments. Every module you use that needs to log something just needs to import logging and logger = logging.getLogger(__name__) and then logger.debug(...) or whatever, wherever needed in that module. If you stick to this, all modules will use the logging level set in basicConfig() automatically, and if you use the fragment %(name)s in the format= argument to basicConfig(), then you will see the full module name in that place in the logged messages, as a fully-qualified dotted name (like api.some_functionality). This approach will certainly have less overhead in creating loggers than the cookiecutter template does.

Update: I'll update the Python Logging Cookbook with an example. For now, here's a Gist with just the code.