log4j2 kafka appender over TLS/SSL

Question

I am trying to use the Apache Kafka appender for log4j2. I am able to publish my logs over Kafka successfully. The question I have is whether I can publish over TLS/SSL. I could not find documentation on how to configure a secure transport for the log4j2 kafka appender, or maybe I am lazy. Any help is appreciated.

Answer 1

1) To set up SSL on the Kafka server, follow the steps outlined on docs.confluent.io/2.0.0/kafka/ssl.html

2) On the Kafka server.properties, add the following

listeners=SSL://:9092
ssl.keystore.location=<Folder>/kafka.server.keystore.jks
ssl.keystore.password=<password>
ssl.key.password=changeme
ssl.truststore.location=<Folder>/kafka.server.truststore.jks
ssl.truststore.password=<password>
ssl.client.auth=required
security.inter.broker.protocol=SSL

3) On the Kafka client (log4j2.xml file of the application that wants to send its logs over Kafka), add the following SSL properties under the Kafka appender configuration, as shown below

<Kafka name="Kafka"  topic="kafka.ssl.log4j2.test.topic">
    <!--JsonLayout complete="true" compact="false" eventEol="true" properties="true" locationInfo="true"/-->
    <PatternLayout>
        <Pattern>[%-5level] [%t] [%C : %M : %F : %L] : %X : %m%n</Pattern> 
    </PatternLayout>
    <Property name="bootstrap.servers">localhost:9092</Property>
    <!--
    <Property name="security.protocol">SSL</Property>
    <Property name="ssl.truststore.location">truststore location of client</Property>
    <Property name="ssl.truststore.password">password</Property>
    <Property name="ssl.keystore.location">keystore location of client</Property>
    <Property name="ssl.keystore.password">password</Property>
    <Property name="ssl.key.password">key password</Property>
    -->
</Kafka>

4) That's it. You should now be able to send your logs to LogStash or any other application that has a Kafka plugin to pick up logs from Kafka.