Python: flush logging only at end of script run

Question

Currently I use for logging a custom logging system that works as follow:
I have a Log class that ressemble the following:

class Log:
    def __init__(self):
        self.script = ""
        self.datetime = datetime.datetime.now().replace(second=0, microsecond=0)
        self.mssg = ""
        self.mssg_detail = ""
        self.err = ""
        self.err_detail = ""

I created a function decorator that perform a try/except on the function call, and add a message either to .mssg or .err on the Log object accordingly.

def logging(fun):
    @functools.wraps(fun)
    def inner(self, *args):
        try:
            f = fun(self, *args)
            self.logger.mssg += fun.__name__ +" :ok, "                
            return f               
        except Exception as e:
            self.logger.err += fun.__name__ +": error: "+str(e.args) 
    return inner 

So usually a script is a class that is composed of multiple methods that are run sequentially. I hence run those methods (decorated such as mentionned above) , and lastly I upload the Log object into a mysql db.

This works quite fine and alright. But now I want to modify those items so that they integrate with the "official" logging module of python.

What I dont like about that module is that it is not possible to "save" the messages onto 1 log object in order to upload/save to log only at the end of the run. Rather each logging call will write/send the message to a file etc. - which create lots of performances issues sometimes. I could usehandlers.MemoryHandler , but it still doesn't seems to perform as my original system: it is said to collect messages and flush them to another handler periodically - which is not what i want: I want to collect the messages in memory and to flush them on request with an explicit function.

Anyone has any suggestions?

Answer 1

Here is my idea. Use a handler to capture the log in a StringIO. Then you can grab the StringIO whenever you want. Since there was perhaps some confusion in the discussion thread - StringIO is a "file-like" interface for strings, there isn't ever an actual file involved.

import logging
import io
def initialize_logging(log_level, log_name='default_logname'):
    logger = logging.getLogger(log_name)
    logger.setLevel(log_level)

    log_stream = io.StringIO()

    if not logger.handlers:
        ch = logging.StreamHandler(log_stream)
        ch.setLevel(log_level)
        ch.setFormatter(logging.Formatter(
            '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
        ))
        logger.addHandler(ch)
        logger.propagate = 0
    return logger, log_stream

And then something like:

>>> logger, log_stream = initialize_logging(logging.INFO, "logname")
>>> logger.warning("Hello World!")

And when you want the log information:

>>> log_stream.getvalue()
'2017-05-16 16:35:03,501 - logname - WARNING - Hello World!\n'