Kafka partition in relation to a broker

Question

Is a partition always on the same physical machine as a broker or can a partition reside on a machine which is not also a broker?

I am pretty sure a partition or multiple partitions can reside on a broker node but I am not sure if a partition can reside on a non broker node?

Answer 1

Partition is just a structure / object which resides inside the broker. Without running Kafka broker, there is no partition. They cannot exist outside of broker.

The Kafka brokers run in clusters - a Kafka cluster can consist only from one broker, but it can be 1000s of brokers. When you create a topic with a defined number of partitions, they will be distributed (either automatically, or you can specify this distribution) across the brokers in the cluster. So if you want to use multiple machines for your topics / partitions, you will need to run a Kafka broker on each of these machines and connect them into a cluster.