Kafka consumer fails silently to connect to broker

Question

We had an issue today where a Kakfa consumer failed to connect to any brokers, and there was no error message in the logs. It failed to connect as it was connecting over SSL and was pointing to the wrong truststore. But although in this instance it was due to an SSL failure, I have seen other times where the consumer fails to connect, and there be no error message in the logs.

Is there a way to configure a Kafka consumer so that an error message is logged if the consumer cannot connect to a broker? I am looking for a generic solution, not specific to SSL connection issues.

(We are using: org.apache.kafka:kafka-clients:0.9.0.1 and org.springframework.kafka:spring-kafka:1.0.4.RELEASE)

Answer 1

• consider upgrading your client to newer version of kafka-clients (in later versions some operations like seek started to throw instead of hanging - that would let you know of issues faster)
• set up log4j, especially org.apache.kafka.clients.NetworkClient class (it logs if it cannot find a broker)
• consider configuring default.api.timeout.ms or in older versions metadata.fetch.timeout.ms to have timeouts faster