Kafka Producer Batching : https://www.conduktor.io/kafka/kafka-producer-batching
A producer will have up to 5 requests in flight (controlled by the max.in.flight.requests.per.connection setting), meaning up to 5 message batches will be sent at the same time.
This smart batching allows Kafka to increase throughput while maintaining very low latency. Batches have a higher compression ratio so a better disk and networking efficiency. Batching is mainly controlled by two producer settings - linger.ms and batch.size.
Producer Batching and no message compression
By default, the messages are not compressed.
...
| Info |
|---|
The max size is represented by the setting “batch.size”. The default value of batch.size is 16 KB so,16Kb x 1024 bytes = 16384 bytes (binary and not decimal). We can increase to 32KB or 64KB, so it can help the increase the compression, throughput and efficiency of requests. |
...
| Note |
|---|
Speed versus Compression : Consider testing snappy or lz4 for optimal speed / compression ratio. |