What are message brokers?
A message broker, or message bus, is a solution that provides asynchronous data transmission from one component to another via an intermediate system.
In contrast to direct connections, where senders know the receivers to which data will be sent and connect directly to them, messaging solutions decouple the sending of data from data consumption. Senders do not know which receivers will see that data or when they will see it.
In addition to routing data, messaging systems can also provide capabilities for logging, distributed processing, flow control, and resiliency. One example is the concept of dead-letter queue or topic, a space to store messages that could not be delivered.
Messaging is a broad term that covers several models that differ based on how data is moved from senders to recipients. There are two primary categories of messaging models: message queue and publish-subscribe (often referred to as pub-sub).
A message queue system consists of message producers, message queues, and message consumers. The queue receives messages created by a producer and ensures that each message for a given queue is delivered to and processed by exactly one consumer.
Message queues can support high rates of consumption by adding multiple consumers for each queue, but only one consumer will receive each message. Which consumer receives which message is determined by the implementation of the message queue. To ensure that a message is only processed by one consumer, each message is deleted from the queue once it has been received, processed, and acknowledged by a consumer.
Message queues support use cases where it is important that each message is processed only once, but it is not necessary to process messages in order, such as a task list or work queue. For example, in the case of network or consumer failures, the message queue will try resending an affected message at a later time (not necessarily to the same consumer) and as a result, that message may be processed out of order.
A publish-subscribe system consists of message publishers, topics, and message subscribers. The topic receives messages created by a publisher, but in contrast to a message queue, it allows each message to be delivered to multiple subscribers.
Publish-subscribe messaging systems ensure that each subscriber receives messages from a topic in the exact order in which they were received by the messaging system. Moreover, subscribers can have two types of subscriptions: ephemeral subscriptions, where the subscription is only active as long the subscriber is up and running, and durable subscriptions, where the subscription is maintained as long as it is not explicitly deleted.
Publish-subscribe use cases are frequently associated with stateful applications, where the order of received messages is important because they determine the state of the application and will, as a result, impact the correctness of whatever processing logic the application needs to apply.
Application decoupling. Using message brokers, services do not have knowledge of other services. They are notified of new events, process that information and produce/publish new information. This new information then can be consumed by the required service, or any number of services. Loose coupling allows microservices to be smaller and more independent.
Non-blocking asynchronicity. Microservices should perform as efficiently as possible, and it is a waste of resources to have many threads blocked waiting for a response. With asynchronous messaging applications can send a request and process another request instead of waiting for a response.
Scalability. Since each service is small and only performs one task, they should be able to grow or shrink as needed. Event driven architecture and messaging make it easy for microservices to scale since they are decoupled and do not block. This also makes it easy to determine which service is the bottleneck and add additional instances of just that service, instead of blindly scaling up all services, which can be the case when microservices are chained together with synchronous communications.
Greater resiliency. Messaging platforms that offer guaranteed delivery can act as the source of truth in the event of system failures and enable rapid recovery without data loss. In the case of service failures, the use of messaging allows healthy services to continue working since they are not blocked by the failed service. Once the failed service is back online, it will start processing the messages that had accumulated during the downtime, making the system eventually consistent. Additionally, the message broker handles the retry of unacknowledged messages which frees the services from network retry and error handling logic.
Complexity. Message brokers require adding an extra component in the architecture, making the system more complex, but it will be worth it specially in high-scalable systems.
Message delivery issues. Messages can be duplicated or produced without order. To mitigate these issues, we can be deterministic and plan for it. For example, in the case of a duplicated message, by making actions idempotent: checking if the action has already been executed in the system, in such case it would be skipped and the message acknowledged, and in the case of unordered messages, by performing the action and reacting with an error and not acknowledging the message. Unacknowledged messages will then be requeued and picked up again by another service, leading to eventual consistency.
If you're using dark mode, do you like the code blocks's theme? I have it available for VS Code, feel free to check it.