...
Kafka as a Messaging System
Kafka Cluster
Principles of distributed systems (multiple workers or nodes)
In Kafka, worker nodes are the Kafka brokers. Within a distributed system, there are different roles and responsibilities and there is a hierarchy that starts with a controller or supervisor. A controller is just a worker node like any other.
Once the controller is established and the workers are assigned and available, the cluster is distributed.
Reliable work distribution
The work that Cluster of brokers performs is receiving messages, categorizing them into topics and reliably persisting them for eventual retrieval.
Distributed consensus with Apache Zookeeper
| Info |
|---|
Apache Zookeeper is a software developed by Apache that acts as a centralized service and is used to maintain a robust synchronization for distributed systems. It is used to managed and coordinate Kafka brokers : https://kafka.apache.org/documentation/#ecosystem |
...
Scaling out will increase levels of reliability and availability.