What may cause huge load in Kafka `__consumer_offsets` topic?

Question

I have simple observation in my Kafka cluster (Kafka 0.11.0.0).

According to jmx information the __consumer_offsets topic constantly is loaded with 10 times more messages than sum of all messages in all other topics. I had also conected console consumer to this topic and I can measure similar values.

• What could be the reason?
• How can I check what is Kafka broker doing and generate such a load on its own?

Answer 1

to read __consumer_offsets topic:

bin/kafka-console-consumer.sh --topic __consumer_offsets --bootstrap-server brokers --formatter "kafka.coordinator.GroupMetadataManager\$OffsetsMessageFormatter" --new-consumer --consumer.config consumer.conf

for kafka 11 use formatter "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

where consumer.conf has one line

exclude.internal.topics=false

Answer 2

For version 2.2 of kafka, this is a demo:

kafka_2.12-2.2.1/bin #

./kafka-console-consumer.sh --bootstrap-server  your-broker   --topic __consumer_offsets --formatter  "kafka.coordinator.group.GroupMetadataManager\$OffsetsMessageFormatter"

Output will be:

[MyGroup-7cc5f948df-f9tqz,__MY_TOPIC,2]::OffsetAndMetadata(offset=30, leaderEpoch=Optional.empty, metadata=, commitTimestamp=1568121107017, expireTimestamp=None)