Producers and Consumers: How Data Flows in Kafka

Batch Consuming in Kafka

Consumers often read messages in batches (a group of messages) instead of fetching them one at a time. This makes consumption faster and more efficient.

Normal Batch Flow

1. Consumer polls messages: Fetches a batch (e.g., 10 messages) from a partition.
2. Process messages one by one inside the batch.
3. Commit offset: After the batch is done, the consumer commits the last processed offset.
4. Next time the consumer starts, it resumes after the committed offset.

Example: Batch of 5 Messages

• Consumer polls offsets [100, 101, 102, 103, 104].
• Processes all successfully.
• Commits offset = 104.
• Next poll starts at offset 105.

Negative Case: Error in the Middle

What if something goes wrong while processing the batch?

Example

• Batch = [100, 101, 102, 103, 104]
• Consumer processes 100, 101 successfully
• Error happens at 102 → the app crashes

Effect Depends on Offset Commit Strategy:

1. Auto Commit (default)
   • Kafka may have already committed offset 104 (end of batch).
   • On restart, the consumer starts at 105.
   • Messages 102, 103, 104 are skipped → data loss for this consumer.
2. Manual Commit After Batch
   • Offset is only committed when the whole batch succeeds.
   • On restart, consumer starts again from 100.
   • Messages 100, 101 are reprocessed → possible duplicates.
3. Manual Commit Per Message
   • Consumer commits after each message.
   • On crash at 102, offsets 100, 101 are safe, and restart continues from 102.
   • No loss, minimal duplicates, but slightly slower.

Diagram: Batch Consuming with Error

sequenceDiagram
    participant C as Consumer
    participant B as Broker (Partition)

    C->>B: Poll batch [100-104]
    B-->>C: Deliver messages

    C->>C: Process 100 ✅
    C->>C: Process 101 ✅
    C->>C: Process 102 ❌ (Error)

    alt Auto Commit
        Note over C,B: Offset 104 already committed → messages 102-104 skipped
    else Manual Commit (after batch)
        Note over C,B: Batch not committed → reprocess 100-104 (duplicates possible)
    else Commit Per Message
        Note over C,B: Offset 101 committed → restart from 102
    end