Passing extra context for tornado log entries

Question

Our log formatter has a user defined parameter. It is defined as:

'%(asctime)s|%(levelname)s|%(name)s|REQID:%(req_id)s|%(module)s:%(lineno)s|%(message)s'

where req_id is a request id, generated by application code for every request. When we are processing the requests, on our application code, we have access to this req_id, and we use it for logging purposes like this:

logger = logging.LoggerAdapter(logging.getLogger(service_name), {'req_id': req_id})
logger.debug('A debug message')

I am trying to make the tornado logger conforming to our log format, but since tornado has no access to our application level req_id it fails with:

KeyError: 'req_id'

How can I tell tornado to use a LoggerAdapter for tornado.access, with a user provided context?

EDIT

As a workaround, I tried the following:

Since it is not possible for me to tell tornado what loggers to use, I managed to hack my way around this limitation by reconfiguring the tornado logger in each request, adding the contextual information using a logging filter.

Unfortunately, reconfiguring the log for each request does not work since tornado will be serving requests in parallel, and we will get an inconsistent state.

How can we pass user context for the tornado loggers then?

Answer 1

The tornado.access log can be controlled by overriding Application.log_request in a subclass or using the log_function application setting. This method defaults to writing to the tornado.access log, but you can override it to log however you want.

Note however that the tornado.general and tornado.application loggers cannot be overridden in this way, so your log formatters/filters must still be able to handle messages that do not have the req_id field.