Apache Kafka may look complex at first, but its main building blocks are simple. If you understand brokers, topics, partitions, and offsets, you can understand how Kafka works.
What is a Broker?
A broker is a Kafka server.
- Each broker stores messages and serves producers and consumers.
- Kafka is usually run as a cluster with many brokers working together.
- If one broker goes down, the others keep the system running.
Think of a broker as a post office that receives and delivers messages.
What is a Topic?
A topic is like a folder or category of messages.
- Producers send messages into a topic.
- Consumers read messages from a topic.
- Example:
- Topic
ordersstores all purchase events. - Topic
usersstores all user registration events.
- Topic
What is a Partition?
Each topic can be split into partitions.
- A partition is an ordered log where messages are stored in sequence.
- Instead of putting all messages in a single log, Kafka splits them into partitions.
- Example: Topic
ordersmay have 3 partitions:orders-0orders-1orders-2
Why Do We Need Partitions?
- Scalability
- If there was only 1 partition, only 1 consumer could read it at a time.
- With many partitions, Kafka can split work across multiple consumers in a consumer group.
- This means more throughput and faster processing.
- Distribution
- Partitions can be stored on different brokers (servers).
- This spreads the load across the cluster, instead of putting all the pressure on one machine.
- Result: Kafka can handle huge volumes of data.
- Replication and Fault Tolerance
- Each partition can have replicas on different brokers.
- If one broker fails, another broker has a copy and can continue serving data.
- This makes Kafka highly reliable in a distributed system.
👉 In short: Partitions make Kafka both scalable and fault-tolerant. Without partitions, Kafka would be just a single log, and it would not be able to handle big data at large scale.
Category: Kafka