Logging Framework considerations

Question

I am designing a server for logging. The business logic for this application is written in multiple languages (C++ & Java for now, but other languages might be added to the mix at a later stage.

I am considering making this a separate server with a well defined interface to ensure that I need not port this for other languages at a later date. For scalability, the main application has the ability to run multiple instances on multiple machines supported by load balancers.

One of the important considerations for the design (other than the usuals like the logging level) is performance and support for multiple logging targets (flat file, console, DB(?) etc) .

How do I ensure that the logger is not impacting the performance of the application? Would communicating using a socket make sense? Is there a better way to do this?

Answer 1

Is there a need to have all your logs shared? I would use whatever logging mechanism is best for each stage of the logic (log4j or java's logging in java, and I guess I don't know C++'s logging libraries enough to suggest one.)

For the most part, logs should only be used for debugging and outside-the-app parsing. I would not recommend integrating logging as part of your business logic. If you really need data in the logs, you'll be much better off making a direct communication rather than spitting out the log to have it slurped in by another application.

If you absolutely need it, you can have an external (very low priority) application that feeds off the logs and sends them back to a centralized logging server.

Answer 2

There is data you need to see in near real time and data which needs to be recorded for offline processing. They have different requirements.

real time data needs to be in a machine readable format and is usually directed to the places where it is used. The central logger can be on this path provided it doesn't delay the real time information unacceptably. For this I would use a sockets (or JMS) rather than a buffered file.

offline processing logs can be machine readable format (for over night reports) or be human readable (for debugging) For this I would use a file or a database or both. File can be simpler to manage, esp if that are large. A database makes building reports easier.

In either case I would pass the information which needs to be send via socket or written to a file, to another thread so that any random delays in the system do not impact the code which is producing the log. In fact, I would consider delaying send any logs until whatever the critical process is complete. i.e. You process everything which needs to be done first, then log everything of interest later.