Understanding Reliability with Kafka

Owned by thierry sinassamy
Last updated: Feb 17, 2022
1 min read

How partitions enable work not only to be distributed, but reliably distributed ! What happens if the disk fails or the data is inaccessible ?

Reliability with Apache Kafka Replication

There is no redundancy between the nodes, there could be unrecoverable data loss because the data being managed by the failed broker, unless replicated to another cluster, is now accessible.

The property “Replication-Factor” will enable to handle failed broker or server.

kafka/bin/kafka-topics.sh --create \ --zookeeper localhost:2181 \ --replication-factor 3 \ --partitions 3 \ --topic unique-topic-name

The replication set to 2 or 3 at minimum so that failures or machine maintenance will not interrupt the cluster’s operation.

Replication with a factor to 3

If one broker is down, Kafka search for a new following peer to replace the quorum member.