Python logging vs. write to file

Question

Which is more efficient? Is there a downside to using open() -> write() -> close() compared to using logger.info()?

PS. We are accumulating query logs for a university, so there's a perchance that it becomes big data soon (considering that the min-max cap of query logs per day is 3GB-9GB and it will run 24/7 constantly for a lifetime). It would be appreciated if you could explain and differentiate in great detail the efficiency in time and being error prone aspects.

Answer 1

Use the method that more closely describes what you're trying to do. Are you making log entries? Use logger.*. If (and only if!) that becomes a performance issue, then change it. Until then it's an optimization that you don't know if you'll ever need.

Pros for logging:

• It's semantic. When you see logging.info(...), you know you're writing a log message.
• It's idiomatic. This is how you write Python logs.
• It's efficient. Maybe not extremely efficient, but it's so thoroughly used that it has lots of nice optimizations (like not running string interpolation on log messages that won't be emitted because of loglevels, etc.).

Cons for logging:

• It's not as much fun as inventing your own solution (which will invariably turn into an unfeatureful, poorly tested, less efficient version of logging).

Until you know that it's not efficient enough, I highly recommend you use it. Again, you can always replace it later if data proves that it's not sufficient.