When does Kafka rotates partition leader? Do we have any property to control it?

Question

I'm trying to understand how often Kafka rotates a partition leader? Do we have any property to control it?

Answer 1

The partition leader gets assigned at the creation of the topic. According to the Kafka documentation on Replica Management the assignment of partitions is happening on a round-robin fashion:

We attempt to balance partitions within a cluster in a round-robin fashion to avoid clustering all partitions for high-volume topics on a small number of nodes. Likewise we try to balance leadership so that each node is the leader for a proportional share of its partitions.

In an ideal world, the leader partition will not rotate. However, there are scenarios when the leader has to be moved to another broker, e.g. broker stops and starts for maintenance or updates. Then the leader will be shifted to another in-sync replica.

To manage these shifts in leader partitions (e.g. to handle unbalances) you have two options.

Option 1: Script kafka-preferred-replica-election.sh

This command line tool can be used to help bring leadership back to replicas as they existed on creation of a topic. Please refer to online documentation for more details and how to use it.

The most simple example is shown below. This will trigger re-assignment of all topics.

kafka-preferred-replica-election.sh --zookeeper localhost:2181

Option 2: Use broker configuration for automatic rebalance

At broker level you have the following configuration that help you steer the automated rebalancing of topic leaders:

• auto.leader.rebalance.enable: Enables auto leader balancing. A background thread checks the distribution of partition leaders at regular intervals, configurable by leader.imbalance.check.interval.seconds. If the leader imbalance exceeds leader.imbalance.per.broker.percentage, leader rebalance to the preferred leader for partitions is triggered. (Defaults to true)

• leader.imbalance.check.interval.seconds: The frequency with which the partition rebalance check is triggered by the controller. (Defaults to 300)

• leader.imbalance.per.broker.percentage: The ratio of leader imbalance allowed per broker. The controller would trigger a leader balance if it goes above this value per broker. The value is specified in percentage. (Defaults to 10)