Setting up a Multi broker kafka on every node on a 5 node cluster Setting up a Multi broker kafka on every node on a 5 node cluster hadoop hadoop

Setting up a Multi broker kafka on every node on a 5 node cluster


Each node in your cluster should only have one configuration file and kafka-server-start should only be run once on each node. For example, server 1 only needs to have a single configuration file that contains, e.g. broker.id = 1.

Each time you run kafka-server-start you are starting a broker (i.e. server). When the broker starts, Kafka will locate the other brokers via ZooKeeper. This allows new brokers to be added and removed from the cluster without any additional configuration file specifying the other nodes in the cluster and without having to do any reconfiguration on the existing nodes.

If you're running kafka-server-start multiple times on the same node then you are indeed starting multiple brokers on the same node, but that's not what you want.