Kafka Streams simplifies application development by building on the Kafka producer and consumer libraries and leveraging the native capabilities of Kafka to offer data parallelism, distributed coordination, fault tolerance, and operational simplicity. In this section, we describe how Kafka Streams works underneath the covers : https://kafka.apache.org/23/documentation/streams/architecture
Kafka as a Messaging System
Kafka Cluster
Principles of distributed systems (multiple workers or nodes)
In Kafka, worker nodes are the Kafka brokers. Within a distributed system, there are different roles and responsibilities and there is a hierarchy that starts with a controller or supervisor. A controller is just a worker node like any other.
Once the controller is established and the workers are assigned and available, the cluster is distributed.
Reliable work distribution
The work that Cluster of brokers performs is receiving messages, categorizing them into topics and reliably persisting them for eventual retrieval.
Distributed consensus with Apache Zookeeper