Why do we have __consumer_offsets partitions default value 50?

Question

Why do we need __consumer_offsets partition count having default value 50? i.e offsets.topic.num.partitions default value if 50. We can even use offsets.topic.num.partitions=1.

Answer 1

Kafka really scales by splitting data over partitions allowing to distribute them across many servers. This is particularly true for this internal topic that is used to store consumer groups data.

Also as this setting can't be changed after deployment, I make sense to have a relatively large default value. This allows clusters to grow from an handful of brokers to dozens of them without hitting scaling issues with this internal topic.

For development purpose and if you have very limited hardware resources, you could set it to 1 but I wouldn't recommend it. From my experience, I've found the cost of having 50 partitions in my development environment negligible.