How to handle mass processing with Kafka Producer

Kafka Producer Batching : https://www.conduktor.io/kafka/kafka-producer-batching

A producer will have up to 5 requests in flight (controlled by the max.in.flight.requests.per.connection setting), meaning up to 5 message batches will be sent at the same time.

This smart batching allows Kafka to increase throughput while maintaining very low latency. Batches have a higher compression ratio so a better disk and networking efficiency. Batching is mainly controlled by two producer settings - linger.ms and batch.size.

Producer Batching and no message compression

The max size is represented by the setting “batch.size”. The default value of batch.size is 16 KB so,16Kb x 1024 bytes = 16384 bytes (binary and not decimal). We can increase to 32KB or 64KB, so it can help the increase the compression, throughput and efficiency of requests.

var producerConfig = new ProducerConfig
{
       Acks = Acks.All,
       BootstrapServers = "localhost:9092",
       BatchSize = 32768, //32KB more than the default
       LingerMs = 20 //some lag or delay to increase the throughput and compression
 };

Producer Batching with message compression enabled

Compression is enabled on the producer-side event if we can do it on the broker-side.

var producerConfig = new ProducerConfig
{
      Acks = Acks.All,
      BootstrapServers = "localhost:9092",
      BatchSize = 32768,
      LingerMs = 20,
      CompressionType = CompressionType.Snappy // or lz4
};

Speed versus Compression : Consider testing snappy or lz4 for optimal speed / compression ratio.