What is Debezium? – An Introduction to Change Data Capture

Outbox Pattern (Sequence Diagram)

sequenceDiagram
  autonumber
  participant App as Application Service
  participant DB as Database (with Outbox Table)
  participant DBZ as Debezium Connector
  participant K as Kafka
  participant C as Consumer Service

  App->>DB: Write business data (orders)
  App->>DB: Insert outbox event (status=PENDING)
  DBZ->>DB: Read transaction log (binlog/WAL)
  DBZ-->>K: Publish event to topic (outbox.events)
  K-->>C: Deliver event
  C->>C: Process event (e.g., send email, update cache)
  C-->>DB: (optional) Ack/Update status via API

1. Application Service (App)
   • The application performs a normal business action, for example:
     • “Create a new order”.
   • At the same time, it also writes an outbox event into a special table called the outbox table inside the same database.
   • The outbox event usually has a status like PENDING, which means it is waiting to be processed.

2. Database (DB with Outbox Table)
   • The database now contains two things:
     1. The business data (like order details).
     2. The outbox event (which describes what happened, e.g., “OrderCreated”).
   • Both are stored in the same transaction, so they are always consistent.

3. Debezium Connector (DBZ)
   • Debezium monitors the database transaction log (binlog for MySQL, WAL for PostgreSQL, etc.).
   • When it sees the new outbox event row, Debezium captures it.

4. Apache Kafka (K)
   • Debezium publishes the outbox event into a Kafka topic, for example outbox.events.
   • Kafka acts as a broker, keeping the event safe and allowing many consumers to subscribe to it.

5. Consumer Service (C)
   • A consumer (another microservice, for example an email service or inventory system) reads the event from Kafka.
   • It processes the event — for example:
     • Send a confirmation email to the customer.
     • Update the product stock.
     • Refresh a cache.

6. (Optional) Acknowledgment
   • After processing, the consumer can send back an acknowledgment to the application or database.
   • For example, it might update the outbox row from PENDING to SENT or DONE.
   • This step is optional but can be useful for monitoring and reliability.

Why Use the Outbox Pattern?

• Ensures consistency between business data and events (no missing messages).
• Avoids “dual-write problem” (where DB and Kafka writes could get out of sync).
• Makes microservices more reliable and event-driven.