How to handle mass processing with Kafka Producer

Kafka Producer Batching : https://www.conduktor.io/kafka/kafka-producer-batching

A producer will have up to 5 requests in flight (controlled by the max.in.flight.requests.per.connection setting), meaning up to 5 message batches will be sent at the same time.

This smart batching allows Kafka to increase throughput while maintaining very low latency. Batches have a higher compression ratio so a better disk and networking efficiency. Batching is mainly controlled by two producer settings - linger.ms and batch.size.

Producer Batching and no message compression

By default, the messages are not compressed.

Kafka Producer Batching

The max size is represented by the setting “batch.size”. The default value of batch.size is 16 KB so,16Kb x 1024 bytes = 16384 bytes (binary and not decimal). We can increase to 32KB or 64KB, so it can help the increase the compression, throughput and efficiency of requests.

var producerConfig = new ProducerConfig
{
       Acks = Acks.All,
       BootstrapServers = "localhost:9092",
       BatchSize = 32768, //32KB more than the default
       LingerMs = 20 //some lag or delay to increase the throughput and compression
 };

Producer Batching with message compression enabled

Compression is enabled on the producer-side event if we can do it on the broker-side.

Kafka Producer Batching with compressed messages

var producerConfig = new ProducerConfig
{
      Acks = Acks.All,
      BootstrapServers = "localhost:9092",
      BatchSize = 32768,
      LingerMs = 20,
      CompressionType = CompressionType.Snappy // or lz4
};