What is Consumer Lag?

Consumer lag is the gap between the latest message in a partition and the offset of the consumer.

• Latest offset = position of the newest message in the partition.
• Current offset = where the consumer has read so far.
• Lag = Latest offset – Current offset.

Example:

• Partition has messages up to offset 100.
• Consumer has processed up to offset 90.
• Lag = 100 – 90 = 10 messages behind.

Why Consumer Lag Matters

• Low lag: Consumer is keeping up with producers.
• High lag: Consumer is too slow or overloaded.
• Critical risk: If lag grows larger than the topic’s retention period, old messages are deleted before the consumer can read them → data loss for that consumer.

Diagram: Consumer Lag

sequenceDiagram
    participant P as Producer
    participant B as Broker / Partition 0
    participant C as Consumer (Current Offset = 90)

    P->>B: Append messages up to Offset 100
    Note right of B: Latest offset = 100

    C->>B: Fetch from Offset 90
    B-->>C: Deliver messages 90..n
    Note right of C: Consumer lag = 100 - 90 = 10

Key Points

• Offsets are bookmarks that tell consumers where to continue.
• Consumer groups allow many consumers to share the load.
• Offset management can be automatic or manual, each with pros and cons.
• Together, they make Kafka a scalable and fault-tolerant system.