How to increase the number of messages consumed by Spring Kafka Consumer in each batch?

Question

I am building a Kafka Consumer application that consumes messages from a Kafka Topic and performs a database update task. The messages are produced in a large batch once every day - so the Topic has about 1 million messages loaded in 10 minutes. The Topic has 8 partitions.

The Spring Kafka Consumer (annotated with @KafkaListener and using a ConcurrentKafkaListenerContainerFactory) is triggered in very short batches.

The batch size is sometimes just 1 or 2 messages. It would help performance if it can consume about 1000 messages at once and process it together (for example, I could update the database in a single update SQL), instead of connecting to the database for each message.

I have already tried to decrease the concurrency in the factory to avoid multiple threads consuming smaller number of messages.

I also increased the socket.send.buffer.bytes property in Kafka's server.properties to 1024000, from 102400.

These steps have not increased the batch size.

Is there any other configuration I could use to increase the bath size of the consumer?

Answer 1

See kafka consumer properties max.poll.records, fetch.min.bytes, fetch.max.wait.ms, fetch.max.bytes, max.partition.fetch.bytes.

Most likely fetch.min.bytes and fetch.max.wait.ms is what you need.