Show extra fields when logging to console in python

Question

I have a basic setup from a logging.ini file so that my console prints human readable logs, but I print JSON dict-like logs to file, so my ELK can process the logs.

[formatter_json]
class=pythonjsonlogger.jsonlogger.JsonFormatter
format=%(asctime)s %(name)s %(levelname)s %(message)s

[formatter_simpleFormatter]
format=%(asctime)s %(name)s - %(levelname)s:%(message)s

When logging messages, I often add extra fields, ad-hoc, like so:

logger.info("hello world, I like the word %s", "cowabanga", extra={"more":15})

And so the field more:15 appears in the log line in my file. But it does not appear in the console, which would just say 2019-06-12 13:25:02,189 root - INFO:hello world, I like the word cowabanga . What do I need to do to have 15 or more:15 appear (I don't care exactly in what order the extra parameters would appear, just that they appear)?

Answer 1

The extra parameter is just ignored by the standard formatter.

That means that you will need a custom Formatter to nicely process the additional arguments. For example this class looks for extra parameters in the LogRecord and if any adds - extra {key: value, ...} to the message:

class ExFormatter(logging.Formatter):
    def_keys = ['name', 'msg', 'args', 'levelname', 'levelno',
            'pathname', 'filename', 'module', 'exc_info',
            'exc_text', 'stack_info', 'lineno', 'funcName',
            'created', 'msecs', 'relativeCreated', 'thread',
            'threadName', 'processName', 'process', 'message']
    def format(self, record):
        string = super().format(record)
        extra = {k: v for k,v in record.__dict__.items()
             if k not in self.def_keys}
        if len(extra)>0:
            string += " - extra: " + str(extra)
        return string

Demo:

>>> log = logging.Logger('foo')
>>> hnd = logging.StreamHandler(sys.stdout)
>>> hnd.setFormatter(ExFormatter())
>>> log.addHandler(hnd)
>>> log.log(logging.INFO, 'foo', extra={'bar': 'baz'})
foo - extra: {'bar': 'baz'}