Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Kafka Streams - Multiple Threads or Multiple Instances with same application id

I have a Kafka Streams application that consumes from topic 'A' with 10 partitions with around 10k messages per second. I am confused about what will be better for my application.

To run multiple Kafka Streams application instances with same consumer group.
  OR 
To run single kafka streams application with more num.stream.threads
like image 678
Nandish Kotadia Avatar asked Oct 15 '25 18:10

Nandish Kotadia


1 Answers

As mentioned in the confluent blog

The maximum parallelism at which your application may run is bounded by the maximum number of stream tasks, which itself is determined by maximum number of partitions of the input topic(s) the application is reading from. For example, if your input topic has 5 partitions, then you can run up to 5 applications instances.

So there is no difference when you run 10 application instance or a single streams application with 10 thread in processing of messages. Except when you run 10 application instance you can run it on different JVMs spread across different machines which can help in some throughput improvement.

Also see this

like image 125
asolanki Avatar answered Oct 18 '25 20:10

asolanki



Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!