What is Consumer Lag?
Consumer lag is the gap between the latest message in a partition and the offset of the consumer.
- Latest offset = position of the newest message in the partition.
- Current offset = where the consumer has read so far.
- Lag = Latest offset – Current offset.
Example:
- Partition has messages up to offset
100. - Consumer has processed up to offset
90. - Lag = 100 – 90 = 10 messages behind.
Why Consumer Lag Matters
- Low lag: Consumer is keeping up with producers.
- High lag: Consumer is too slow or overloaded.
- Critical risk: If lag grows larger than the topic’s retention period, old messages are deleted before the consumer can read them → data loss for that consumer.
Diagram: Consumer Lag
sequenceDiagram
participant P as Producer
participant B as Broker / Partition 0
participant C as Consumer (Current Offset = 90)
P->>B: Append messages up to Offset 100
Note right of B: Latest offset = 100
C->>B: Fetch from Offset 90
B-->>C: Deliver messages 90..n
Note right of C: Consumer lag = 100 - 90 = 10
Key Points
- Offsets are bookmarks that tell consumers where to continue.
- Consumer groups allow many consumers to share the load.
- Offset management can be automatic or manual, each with pros and cons.
- Together, they make Kafka a scalable and fault-tolerant system.
Pages: 1 2
Category: Kafka
