Kafka and Stateful Architecture

Local storage : Each consumer has a local database (RocksDB as file key-value database).

Local storage is partitioned by key. So, each consumer has a subset of keys.

The main benefit of the local storage is that there is no need to send a network request to fetch data to process each message and we can process incoming records much faster.