Skip to main content

Kafka Consumer Configuration

At Most once 


Offsets are committed as soon as messages are received, in case of processing fails those messages will be lost, let's say the consumer went down, when the consumer comes back it will start reading from the point of the last committed offsets.



At least once

In this case, offsets are committed after processing the message batch, in case of consumer failure, the same messages are read twice so processing also happens twice, so make sure for idempotency for the system.


Exactly Once

This can be achieved with Kadka Transactional APIS, [easy with Kafka stream apis],



Consumer Offset Reset behavior 

auto.offset.reset:latest -> start reading from end 

auto.offset.reset:earlierest -> start reading from start.

auto.offset.reset.none-> throw NP if the offset was not found.


Note- consumers offset can be lost 

kafkaV<2.0 [consumer hasn't read data in 1 day ]

kafkaV>2.0 [consumer hasn't read data in 7 days]

This can be controlled by offset.retention.minutes  property


Comments

Post a Comment

Popular posts from this blog

Handling Kafka Retries in Spring Boot: Blocking vs. Reactive Approaches

  Introduction Apache Kafka is designed for high availability, but failures still happen—network issues, broker crashes, or cluster downtime. To ensure message delivery, applications must implement retry mechanisms. However, retries behave differently in traditional (blocking) vs. reactive (non-blocking) Kafka producers. This guide covers: ✅ Kafka’s built-in retries ( retries ,  retry.backoff.ms ) ✅ Blocking vs. non-blocking retry strategies ✅ Reactive Kafka retries with backoff ✅ Fallback strategies for guaranteed delivery ✅ Real-world failure scenarios and fixes 1. Kafka Producer Retry Basics When Do Retries Happen? Kafka producers automatically retry on: Network errors (e.g., broker disconnect) Leader election (e.g., broker restart...
Post a Comment
Read more

🔄 Kafka Producer Internals: send() Explained with Delivery Semantics and Transactions

Kafka Producer Internal Working Apache Kafka is known for its high-throughput, fault-tolerant message streaming system. At the heart of Kafka's data pipeline is the Producer —responsible for publishing data to Kafka topics. This blog dives deep into the internal workings of the Kafka Producer, especially what happens under the hood when send() is called. We'll also break down different delivery guarantees and transactional semantics with diagrams. 🧠 Table of Contents Kafka Producer Architecture Overview What Happens When send() is Called Delivery Semantics Kafka Transactions & Idempotence Error Handling and Retries Diagram: Kafka Producer Internals Conclusion 🏗️ Kafka Producer Architecture Overview Kafka Producer is composed of the following core components: Serializer : Converts key/value to bytes. Partitioner : Determines which partition a record should go to. Accumulator : Buffers the records in memory be...
Post a Comment
Read more

Choosing Between Envoy and NGINX Ingress Controllers for Kubernetes

As Kubernetes has become the standard for deploying containerized applications, ingress controllers play a critical role in managing how external traffic is routed to services within the cluster. Envoy and NGINX are two of the most popular options for ingress controllers, and each has its strengths, weaknesses, and ideal use cases. In this blog, we’ll explore: How both ingress controllers work. A detailed comparison of their features. When to use Envoy vs. NGINX for ingress management. What is an Ingress Controller? An ingress controller is a specialized load balancer that: Manages incoming HTTP/HTTPS traffic. Routes traffic to appropriate services based on rules defined in Kubernetes ingress resources. Provides features like TLS termination, path-based routing, and host-based routing. How Envoy Ingress Controller Works Envoy , initial...
Post a Comment
Read more