Outbox Pattern in microservices

src: Outbox Pattern in Microservices | Baeldung on Computer Science - 2024-03-18

1. Overview

In the sprawling landscape of microservices, the Outbox Pattern emerges as a paragon of order and consistency. This pattern orchestrates event communication, ensuring data’s sanctity amidst asynchronicity.

In this tutorial, we’ll dive deep into the Outbox Pattern, unearthing its core mechanics and real-world applicability, and highlight its unparalleled adaptability and importance.

2. What Are Microservices?

Microservices are an architectural style of developing software applications as a collection of small, independent, and loosely coupled services. By doing so, we enable each service to handle a specific business function, and these services communicate through well-defined APIs. This approach better facilitates scalability, flexibility, and maintainability than traditional monolithic architectures.

2.1. Example

Let’s say our application features discrete components: catalog, shopping cart, discount, and ordering, each functioning as an independent microservice with dedicated databases.

UUUsers browse the catalog, fill their shopping carts, add discounts, and complete their orders. To enhance this experience, developers use an API Gateway. It acts as a single entry point, directs requests to the right microservices, and combines responses to present a unified user interface.

This approach ensures that each microservice can be scaled independently, facilitating optimal resource allocation based on real-time demands. As a result, our application showcases enhanced resilience, agility, and user satisfaction.

Outbox Pattern is a method of maintaining data consistency.

At first, when we consider microservices, we imagine standalone, independent services. But, challenges pop up when these services have to talk or exchange data. That’s when we turn to this pattern.

3.1. Goals

First, the Outbox Pattern defines temporary storage (outbox table) where microservices log their outgoing data changes. This pattern serializes the data, preparing it for transmission. Then, it sends the serialized records from the outbox table to other services or external systems. Throughout this operation, the pattern ensures data consistency and reliability.

Additionally, the outbox table provides a safety net. If a network problem or service downtime prevents immediate data dispatch, it holds onto the data to avoid loss. When technicians fix these issues, the pattern sends the data correctly.

At its core, Outbox Pattern isn’t just a pattern. It stands as a mechanism for resilience and consistency in microservices. As systems expand and complexity increases, relying on strategies like the Outbox Pattern becomes favorable and necessary.

3.2. Architecture

Here’s how it works:

Whenever a business transaction unfolds within a microservice, it makes the required changes within a database transaction. Rather than forging a direct link with external services such as RabbitMQ, EventHub, or Kafka, we store all the events (table changes) in the outbox table in the same database that the microservice uses.

A background service consistently checks the outbox table and relays the archived events to a pre-chosen message broker or event-streaming platform.

Transitioning further, the services subscribed to the message broker get notified about the events and react according to their business rules.

4. Implementation Steps

Having defined the Outbox Pattern, let’s now learn how to implement it:

4.1. Crafting Our Dedicated Outbox Database Table

Every microservice under our domain needs its outbox table:

Here’s its explanation:

• is the unique identifier for each entry
• denotes the kind of entity we’re dealing with
•   is where we store the ID of said entity
• represents the message type
• is the message content, often serialized for efficiency
• is the timestamp marking when we added the message
• shows us if the message is pending or processed

4.2. Streamlining Data Modifications and Event Relay

When there’s an operation leading to data alterations:

1. The modifications are initially recorded in the primary database.
2. Instead of immediately dispatching an event to associated services, the event gets serialized and stored in our outbox table with a “Pending” status.

To ensure the seamless transition of these pending events, we use a periodic task or a persistent background service to scan the outbox table.

This mechanism routinely inspects our outbox table for pending entries. After fetching these events, it sends them to their designated destination (possibly a message broker):

Upon successful dispatch, we update the event’s status to indicate it’s been processed.

4.3. Relayer Resilience and Outbox Maintenance

Should our relayer falter, the messages remain pending. We bolster resilience with retry protocols, assuring each message is eventually sent.

As processed messages accumulate, we archive or discard them, keeping the outbox table as small as possible.

4.4. Vigilant Operations: From Receiving to Monitoring 

Services processing our messages need to handle potential duplicates. Moreover, we must stay alert at all times. We actively monitor our outbox’s growth and any delivery issues and keep a close watch on our pending messages to make sure we don’t overlook any:

By taking these steps, we strengthen our microservices’ communication and boost our system’s resilience. In doing so, we fully embrace the Outbox Pattern, creating a smooth data flow throughout our ecosystem.

5. Real-World Application

Jumping into the e-commerce scene, the Outbox Pattern shines. Let’s say we have the ordering, inventory, and notification services, each operating independently. When a shopper hits the buy option, a sequence of actions kicks off:

First, the ordering service captures the order and sends an “Order Created” alert to its outbox. Right after, the inventory service sees this alert. Immediately, it updates the item’s stock. Each action flows naturally into the next, like a perfectly timed dance.

At the same time, the notification service, following the same alert, informs the user about their order status.

The real magic of this pattern shows up during unexpected downtimes. For instance, if the inventory or notification services malfunction, the outbox ensures no event is lost since the services can take the saved events when they start working again. This means inventory stays accurate, and shoppers always receive timely updates.

6. Advantages and Disadvantages

Let’s check out this pattern’s advantages and disadvantages:

Advantages	Disadvantages
Consistency	Increased complexity
Reliability	Performance
Decoupling	Additional storage

It maintains data consistency and doesn’t lose events. If a service fails before sending an event, the event isn’t lost; it remains in the outbox and can be processed when the service is back online. Further, the pattern decouples microservices from one another and external event systems such as RabbitMQ or Kafka. However, it adds complexity since we have to implement Outbox table scanning, which also slows down the system. Furthermore, storing events means spending more space than without this pattern.

7. Conclusion

In this article, we talked about the Outbox Pattern.

It addresses the data consistency challenges typical in microservices architectures. Using this pattern, we strengthen consistency across our services and guarantee their decoupling, resilience, and scalability.

While integrating the Outbox Pattern may add layers of complexity, its benefits typically outweigh the possible drawbacks, especially in larger and mission-critical systems. Like any design pattern, we must evaluate its fit meticulously, considering our system’s needs and limitations.

Difference with change data capture

The Change Data Capture (CDC) and Outbox patterns are both used to capture and process changes to data in a database, but they differ in their approach and use cases.

Change Data Capture (CDC) is a pattern that involves capturing changes to data in a database by reading the database’s transaction log. This log contains a record of all changes made to the database, including inserts, updates, and deletes. CDC tools can read this log and extract the changes, which can then be processed and sent to other systems, such as data warehouses, data lakes, or message queues.

The Outbox pattern, on the other hand, is a design pattern that involves storing changes to data in a separate table or message queue, often referred to as an “outbox”. When data is updated or inserted, a corresponding message is added to the outbox, which can then be processed and sent to other systems.

The key differences between CDC and the Outbox pattern are:

• Source of truth: In CDC, the database’s transaction log is the source of truth, whereas in the Outbox pattern, the application itself is responsible for generating the messages.
• Data consistency: CDC ensures that the changes are captured in real-time, as they are written to the transaction log. The Outbox pattern relies on the application to generate the messages, which may introduce latency or inconsistencies.
• Data format: CDC typically captures changes in a raw, database-specific format, whereas the Outbox pattern allows for more flexibility in the format of the messages.
• Complexity: CDC often requires specialized tools and expertise to read and process the transaction log, whereas the Outbox pattern can be implemented using standard application code.

In general, CDC is suitable for use cases where:

• High data consistency and real-time capture are critical
• The database is the system of record, and changes need to be captured accurately
• The application is not capable of generating messages for changes

The Outbox pattern is suitable for use cases where:

• The application has control over the data changes and can generate messages accordingly
• Flexibility in message format is required
• The application needs to handle complex business logic or validation before sending messages