I understand that I cannot control when to poll if I use @KafkaListener, and I read from this answer that
the next
poll()is performed after the last message from the previous poll has been processed by the listener.
So I'm wondering how to know when each poll() is executed? Or equivalently, how long does it take to process all messages received in each poll() call?
I am asking because my program got "Offset commit failed ... The request timed out" exceptions, and I would like to tune my consumer config, i.e. max.poll.interval.ms and max.poll.records, but I need to know the current performance first.
Here is part of my @KafkaListener method if it helps:
@KafkaListener(id = "dataListener", topics ="${spring.kafka.topic}", containerFactory = "kafkaListenerContainerFactory")
public void listen(@Payload(required = false) ConsumerRecord payload, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partition,
@Header(KafkaHeaders.OFFSET)Long offset, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY)String messageKey){
// processing messages
}
You can see polling activity by turning on DEBUG logging.
this.logger.debug(() -> "Received: " + records.count() + " records");
logged after each poll().
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With