Understanding distributed partition management
- thierry sinassamy
Owned by thierry sinassamy
How the Kafka producer splits the messages across the different partitions !
Partitions assigned to each broker by Zookeeper at the creation of topic (MY_TOPIC_A)
The assignment of the partition by Zookeeper
Leaders of topic’s partitions found by Producer
Partitions found by producer
Leaders of topic’s partitions found by Consumer
The more partitions the greater the Zookeeper overhead !
The more partitions the longer the leader fail-over time !