Spring Boot and Kafka: Broker disconnected

Question

I have setup a Spring Boot application to receive Kafka messages from an existing and working Kafka producer. The setup is standard, and based on the following: https://www.codenotfound.com/spring-kafka-consumer-producer-example.html

Messages are not received, and the following is continually displayed in the console:

WARN org.apache.clients.NetworkClient :Bootstrap broker <hostname>:9092 disconnected 

In addition, the following debug message is logged:

org.apache.common.errors.Timeout: Failed to update metadata after 60000 ms.

The console message is discussed in the following link: https://community.hortonworks.com/content/supportkb/150148/errorwarn-bootstrap-broker-6668-disconnected-orgap.html

The logged message is discussed here: https://community.cloudera.com/t5/Data-Ingestion-Integration/Error-when-sending-message-to-topic-in-Kafka/td-p/41440

Very likely, the timeout will not happen when the first issue is resolved.

The solution to the console message which is given is to explicitly pass --security-protocol SSL as an argument to the producer or consumer command.

Given that I am listening on an existing Kafka broker and topic, no settings can be changed there. Any changes must be on the Spring Boot side.

Is it possible to configure application.yml so that --security-protocol SSL is passed an an argument to the consumer? Also, has anyone experienced this before, and is there another way to resolve the issue using the configuration options available in Spring Boot and Spring Kafka?

Thanks

Answer 1

See the documentation.

Scroll down to Kafka. Arbitrary Kafka properties can be set using

spring:
  kafka:
    properties:
      security.protocol: SSL

applies to consumer and producer (and admin in 2.0).

In the upcoming 2.0 release (currently RC1), there is also

spring:
  kafka:
    properties:
      consumer:
        some.property: foo

for properties that only apply to consumers (and similarly for producers and admins).