Kafka Consumer Configuration

At Most once 


Offsets are committed as soon as messages are received, in case of processing fails those messages will be lost, let's say the consumer went down, when the consumer comes back it will start reading from the point of the last committed offsets.



At least once

In this case, offsets are committed after processing the message batch, in case of consumer failure, the same messages are read twice so processing also happens twice, so make sure for idempotency for the system.


Exactly Once

This can be achieved with Kadka Transactional APIS, [easy with Kafka stream apis],



Consumer Offset Reset behavior 


auto.offset.reset:latest -> start reading from end 

auto.offset.reset:earlierest -> start reading from start.

auto.offset.reset.none-> throw NP if the offset was not found.


Note- consumers offset can be lost 

kafkaV<2.0 [consumer hasn't read data in 1 day ]

kafkaV>2.0 [consumer hasn't read data in 7 days]

This can be controlled by offset.retention.minutes  property



Comments

Popular posts from this blog

Stay Updated As Software Engineer

12 Must-Know LeetCode+ Links for Coding Excellence

Getting Master in Hibernate and JPA