Is there an upper cap on the number of partitions in Apache Kafka

Question

I have read that its 10000. But what i want to know is that this limit is on 1 broker or the entire cluster. I want to have a topic with event name and capture all the event streams of the user. Now i wish to record activity to every user on the dedication partition where that particular user's event will go. Now what if the number of users go beyond 10000. Can i increase the number of partitions just by adding another broker or this approach is not feasible at all.

Answer 1

This question is a bit old, but there is a lot of confusion around this issue.

The answer is closely related to the version of the Kafka broker that you are running. As you can see in this Apache blog post, a reasonably up to date cluster can hold up to 4,000 partitions per broker and up to 200,000 partitions in a cluster.

Answer 2

First of all there is a nice post about Kafka Partitions on Confluent blog by one of the author of Kafka. Maybe it will help you.

I don't think it's a good idea to create a partition for each user. Besides other things it's hard to dynamically add new partitions to Kafka cluster and I suppose you would need it. What is the main reason for such architecture?

I think It should be sufficient to use let's say ten partitions and you can send to every partition data of 1000 users. When you read the messages you will always know to which user the message belongs because there would be some userId. You just need to ensure that two events from the same user will end up in the same partition. That's all.

Hope it helps, Lukas