What's the best way to capture Log4J (2) log entries during a test?

Question

I'm writing a logger testing Framework in which I intend to support multiple logging backends and test harnesses. In each test harness there's a setup/teardown cycle that gives me a chance to inject some form of "log capturing" instance into the logging system.

In JDK logging it's easy to install a new handler to a specific logger to capture everything it sees, but in Log4J2 I've struggled to get something with the same behaviour.

I've read all the docs in https://logging.apache.org/log4j/log4j-2.1/manual/architecture.html and I fully understand why programmatic configuration of logging is a bad idea, but for tests it's really the only reasonable approach.

In Log4J I'm guess that adding an Appender is the best way to go, but getting an Appender added (and then removed) to the right place in the configuration hierarchy is difficult.

In particular, I need to be able to add an Appender for a package (i.e. for which no logger exists in the code-under-test), so I'd like a way to say:

"Get me the logger config for this name, creating it if it doesn't exist"

Instead, the various config getters I've found will just search up the hierarchy and find the parent config (typically root in my case).

My requirements are:

1. Add a capturing appender at a specific point in the config namespace.
2. Have the appender capture logs for all loggers at or below that point in the config namespace.
3. Have a robust way to remove the appender when I'm done.
4. Don't affect actual log levels or any other part of the existing logger configuration.

I've managed to get an Appender installed for an existing logger, but any "child" loggers seem to not use this.

    // The "loggerName" can be a package name and no logger need exist for it before this point.
    Logger logger = (Logger) LogManager.getLogger(loggerName);
    Configuration configuration = ((LoggerContext) LogManager.getContext()).getConfiguration();
    configuration.addLoggerAppender(logger, appender);
    LoggerConfig config = configuration.getLoggerConfig(loggerName);
    // A callback to remove the appendr after the test.
    return () -> {
      try {
        // I don't understand why there's no way to remove the append instance via its reference,
        // so I hacked it to use a random name string since I don't want to get into issues
        // with multiple tests running in parallel. I'd like a solution that avoids this.
        config.removeAppender(probablyUniqueAppenderName);
      } catch (RuntimeException e) {
        // Ignored on close().
      }
    };

One unfortunate thing I've noticed is that in various places during other attempts I've seen the existing log level of a logger be modified by the seemingly simple act of adding an Appender. This is clearly unacceptable to what I'm trying to do and I've no idea why that's desireable behaviour (when a new config is created it inherits the parent config's level rather than using any log level already set on the logger).

I also don't believe what I'm asking is obviously identical to any of the previous (often years old) questions I've seen here about Log4J, so please don't just assume I should use one of those (the ones I've tried didn't work).

Edit: The other code snippet I tried first (which also doesn't work) is:

    Logger logger = (Logger) LogManager.getLogger(loggerName);
    // This *changes* the logger's level because there's no existing config.
    // For example, existing loggers set to TRACE level become set to ERROR after
    // this call because the act of creating a new config for this logger inherits
    // from the parent (root) instead of using the level set on the instance.
    logger.addAppender(appender);
    return () -> {
      try {
        logger.removeAppender(appender);
      } catch (RuntimeException e) {
        // Ignored on close().
      }
    };

Edit 2: I've also looked at resetting the configuration by merging the existing configuration with a custom one I created programmatically, but several things didn't work when trying that (in particular, the CompositeConfiguration class won't let me merge instances of the Configuration interface, only sublcasses of the AbstractConfiguration class (which isn't what LogManager returns).

Edit 3: I've now found https://logging.apache.org/log4j/2.x/manual/customconfig.html#programmatically-modifying-the-current-configuration-after-initi which looks promising (if a little complex), but the docs don't explain:

1. what an AppenderRef does, or why it's needed
2. how to undo any modifications to re-installl the original configuration after testing is complete (though I can probably guess that).

Answer 1

I discovered that using the Configurator (internal API) was good enough to change a logger's level without it being reverted when I add the appender.

So in my tests, instead of using:

• logger.setLevel(Level.FOO)

I now use:

• Configurator.setLevel(logger, Level.FOO)

Then to add/remove appenders for the duration of the test, I can just use my original idea via logger.addAppender() / removeAppender() (using the "core" API).

Basically what I've discovered from this is the Log4J has very strong opinions about programmatic configuration (which is fine, they mostly match mine), but doesn't do a good job of explaining how some logger state is transitory and will be overridden as a side-effect of any operations on the configuration.

So if you are needing to do this sort of programmatic manipulation, you'll end up chasing your tail. This isn't a 100% what I wanted to achieve, but it does help me move forward.

I'm also having to assume they'll never actually remove the configurator (Hyrum's Law probably means it's too well used to be removable at this point).