Kafka Producer Configuration

 This article will show the Kafka producer config.

Acks=0

Producer assumes a message written successfully to the topic at the same time the message is sent, without waiting for the broker to accept it all, it 

If the broker goes offline at the same will lose the data 

Suitable for the scenario where potentially okay to lose the data, like metrics collections.

In this case, producer throughput is highest as less network overhead 

Acks=1

Producer assumes a message written successfully to the topic if gets ack from the leader.

It's the default behavior for Kafka 1 and 2.8

Leader response is requested, but replication is not guaranteed as it happens in the background.

If the broker leader goes offline unexpectedly, at the same time replication does not happen possible data loss.

If acks are not received, the Producer will retry to send the message again.

Acks=All [-1]

it's default for kafka V3+.

In this case will wait for acceptance from all sync replicas. No data loss.

With min.insync.replicas Value, this is the optimization of this setting. 

Let's say if having 3 replicas, and setting for min.insync.replicas =2 , will just wait for leader and one replica response,

Idempotent Producer 

This producer guarantees no duplication of the message, in case of broker's response gets failed due to network error, it's by default behaviour of Kafka V3.

The default configs

retries = Integer.maxValue(2^31-1)

max.in.flight.requests=1 [kakfka 0.11]Or [Note number of parallel request]

max.in.flight.requests=5 [kakfka >V1, Higher performance and keep ordering KAFKA-5494]

The default config for Kafka v3+ is above.[Safe Kakfka]

acks=All

enabled.idempotence=true;

max.in.flight.requests=5

timeout  2 minute 

min.insync.replicas=2

The default config for Kafka v2.8 and below.

acks=1

enabled.idempotence=false;

Batching Performance Improvement 

Increase liger.ms time will help the producer to wait that many milliseconds to fill up the batch

The batch size can be increased to take more messages.

Add producer-level compression for efficiency.

Default Partitioner and Sticky Partitioner  

2.4 > above sticky partitioner [performance improvement in as message sent into batches with full capacity ]

The below version is a round-robin in this case more batches are less efficient.

Advanced Setting max.block.ms and buffer.memory

When the Broker is down or busy, the Producer keeps buffering the messages into the buffer as per setting 

producer send method and throws the exception as per the defined time max.block.ms if it is not able to deliver the messages that got buffered. It simply means the Broker is down or some issues need action on it.