Overview
In the micro-service world, we often heard about Event Driven Architecture (EDA) which uses events to communicate between services. A publisher publishes events without caring who will consume the events, this async communication (background operations) makes it decoupled between components. Reducing latency is one of its benefits.
There are some famous tools that implement Event Driven Architecture, ie: Kafka, NSQ, Amazon SQS, GCP Pub Sub, RabbitMQ, and soon. In this article, I want to talk a bit about RabbitMQ.
RabbitMQ is a message broker based on Advanced Message Queuing Protocol (AMQP). A message is published to an exchange with metadata, then it is forwarded to queue according the specific rule called bindings that reads from the metadata. Once there is a consumer active on that queue, RabbitMQ pushes that message to the client (Yes, different with Kafka which uses pull model, RabbitMQ uses push model instead). In general, RabbitMQ’s topology can be seen in picture below:
A publisher publishes a message to an exchange inside the broker, then the message is forwarded to queue and being pushed to consumer. By default it will be acknowledged automatically once a consumer receives the message, but we can setup to sent acknowledgement message after the consumer processes the message.
Another characteristic is by default, once a consumer get the message, the message will be gone from the broker (while Kafka still saving the message based on retention config). Absolutely we can configure as well to make the message durable and survive even though the broker restarted. But of course it comes without free lunch!!!
Terminologies
Publisher
A message sender to a broker. It contains specific logic to construct message and its metadata. Usually exchange name, routing key, and its message body. The message body can be plain string, json, or protobuf. It depends on the requirements.
Exchange
Exchange is a AMQP entity that are messages sent to. Exchange will forward the message to a queue through specific rule, called bindings. There are some type of exchanges:
| Exchange type | Description |
| Direct Exchange | Exchange will forward the message to a queue based on message routing key. |
| Fanout Exchange | Exchange will forward the message to all of queues that are bound to it. |
| Topic Exchange | Exchange will forward the message to a queue based on message routing key and binding key pattern. |
| Header Exchange | Exchange will forward the message to a queue based on metadata from message header. |
Queue
Queue stores messages that will be pushed to consumers. Queue is connected to exchange through bindings. Queue has some properties:
| Attributes | Description |
| Name | Queue name |
| Durable | If true, the message survives even though the broker restarted |
| Exclusive | If true, it will only use one connection, and the queue will be deleted automatically once the connection closed. |
| Auto-delete | If true, the queue will be deleted automatically once the last consumer unsubscribes. |
| Arguments | Other options, like TTL, etc |
Binding
Binding is rules that connecting between exchange to queue (usually). But it also possible to bind between exchanges. A binding contains routing key, that will be compared with the message routing key. One exchange can have multiple bindings. One queue also able to have multiple bindings.
Consumer
Consumer receives message(s) that is pushed from queue. By default, once a message pushes to a consumer, then immediately broker got acknowledgement and the actual message will be deleted. But we can configure it, so that we can send a acknowledgement after the consumer processes it.
Another interesting fact, consumer can send back negative acknowledgement in case consumer wont process the message at the moment.
Virtual Host
Similar to vhosts that are used in many popular web servers. It makes a single broker can have multiple isolated virtual hosts. Example, we have an application that are used in some countries, if we want to differentiate configuration, then we can utilize this so that one country one environment.
Policy
Policy is a way to centralize queue configurations. Some queues that are doing similar things may have the same configurations. So instead of configure one by one queue, we can easily manage them by creating a policy then applying to them.
Topology
As I mention above, there are 4 types of exchange: direct exchange, fanout exchange, topic exchange, and header exchange. RabbitMQ also provides default exchange which has type direct exchange. In order to understand these type of exchanges, Iets see detailed example below:
Default Exchange
Basically default exchange in RabbitMQ is an direct exchange without exchange name and binding. Since there is no routing key, we can use queue name when publishing a message. Then default exchange will forward the message to that queue.
Direct Exchange
In direct exchange, message is published to specific exchange and route. Direct exchange is suitable when we have some consumers and want to differentiate its responsibility. Lets take a look the use case:
- exchange type: direct exchange
- queue name 1: resize and bind to direct exchange with routing key img.resize
- queue name 2: crop and bind to direct exchange with routing key img.crop
- 2 consumers listen to queue resize (resizer.1 and resizer.2)
- a consumer listens to queue crop cropper.1
When publisher publishes a message to direct exchange with routing key img.resize, then the message is forwarded to resize queue and being pushed to either resizer.1 and resizer.2. Please note, if there are more than one consumer that listen to 1 queue, then the message will be distributed in round-robin fashion. Yes, the resize queue acts as load balancer!!
Now, lets take a look the second use case:
- exchange type: direct exchange
- queue name 1: resize.1 and bind to direct exchange with routing key img.resize.1
- queue name 2: resize.2 and bind to direct exchange with routing key img.resize.2
- queue name 3: crop and bind to direct exchange with routing key img.crop
- 2 consumers listen to queue resize.1 (resizer.1.1 and resizer.1.2)
- a consumer listens to queue resize.2 (resizer.2.1)
- a consumer listens to queue crop (cropper.1)
Because of 2 queues are bind to direct exchange with the same routing key, then the exchange copies the message and distributed to both queues (resize.1 and resize.2).
Fanout Exchange
Once a message is sent to a fanout exchange, then it will be replicated as much as connected queues and forwarded to the queues regardless the routing key. You are right!!, fanout exchange will ignore the routing keys. Below is the example of use case:
- exchange type: fanout exchange
- queue name 1: room a
- queue name 2: room b
- queue name 3: room c
- there are 2 printers act as consumers in room a
- in room b and room c, each have one consumer
Let say we are in the newspaper factory, we have 3 rooms that are responsible to print the newspaper. Then we can send a message that contains command for printing newspaper to fanout exchange, once it is arrived in the exchange, then it will broadcast the message to room a, room b, and room c.
Topic Exchange
Topic exchange extends capability of direct exchange by using pattern binding key (* and #). The binding key follows these expression rules:
- * (star) can substitute for exactly one word
- # (hash) can substitute for zero or more words
Thus, it will reduce repeated similar binding configuration. Below is an example configuration:
- exchange type: topic exchange
- queue name 1: only error and bind to topic exchange with routing key logs.error
- queue name 2: all logs and bind to topic exchange with routing key logs.*
- queue name 3: only success and bind to topic exchange with routing key #success
- 2 consumers listen to only error queue
- other queues connect to one consumer each
Scenarios:
- Message with message routing key logs.error, it will be sent to only error and all logs queue.
- Message with message routing key logs.warning, it will be sent to all logs queue.
- Message with message routing key logs.success, it will be sent to only success and all logs queue.
- Message with message routing key success, it will be sent to only success queue.
- Message with message routing key logs.only.success, it will be sent to only success queue.
Header Exchange
A header exchange routes message based on arguments in header instead of routing key. Header exchange is suitable to be used in a case that more complex than topic exchange. There is special property in binding called x-match. This property has 2 possible values: all (default) and any. A value all means all header pairs (key, value) must match with the header binding properties, while any means at least one header pair (key, value) is match with the header binding property. Below is an example configuration:
- exchange type: header exchange
- queue name 1: image.jpg and bind to header exchange with these binding properties:
- x-match: all
- type: image
- format: jpg
- queue name 2: image and bind to header exchange with these binding properties:
- x-match: any
- type: image
- format: png
- 2 consumers listen to image.jpg queue
- 1 consumer listens to image queue
Scenarios:
- message with header properties (type: image) and (format: jpg), then it will be forwarded to image.jpg queue (since it matches with all of binding properties) and image queue (since it matches the type)
- message with header properties (type: image) and (format: png), then it will be forwarded only to image queue.
Conclusion
RabbitMQ is a message broker based on AMQP and uses push model as its implementation. There are 4 exchange types: direct exchange, fanout exchange, topic exchange, and header exchange. RabbitMQ also have default exchange which is basically direct exchange without exchange name and binding configuration.
In next articles, I will write about RabbitMQ implementation for all of exchange types.