Example Event in Kafka

{
  "order_id": "ORD-20250927-123",
  "customer_id": 789,
  "status": "CREATED",
  "total": 149.95,
  "created_at": "2025-09-27T13:10:00Z"
}

End-to-End Flow

sequenceDiagram
  autonumber
  participant App as Order Service
  participant DB as Postgres (order_outbox)
  participant DBZ as Debezium Connector
  participant K as Kafka (12 partitions)
  participant S as Shipping Service
  participant B as Billing Service
  participant M as Monitoring System

  App->>DB: Insert order + outbox event
  DBZ->>DB: Read WAL (INSERT into order_outbox)
  DBZ-->>K: Publish event (partitioned by order_id)
  K-->>S: Deliver event for shipping
  K-->>B: Deliver event for billing
  DBZ-->>M: Publish DB heartbeat + Kafka heartbeat
  S-->>M: Send consumer lag metrics
  B-->>M: Send consumer lag metrics

Best Practices

1. Replication Factor ≥ 3: prevent data loss if broker fails.
2. Partitions: choose partition key carefully (order_id, customer_id, etc.).
3. Monitor Lag: measure event delay from DB → Debezium → Kafka → Consumer.
4. Dead Letter Queue: always enabled.
5. Heartbeats: check both DB heartbeat and Kafka heartbeat.
6. Consumer Metrics: expose Prometheus metrics for lag and throughput.
7. Security: use Vault/KMS for DB password, enable TLS for Kafka.

Conclusion

A production-grade Debezium connector for PostgreSQL with Kafka requires more than a simple config. You need:

• Outbox Pattern for data-event consistency.
• Partitioning for ordered events.
• Exactly-once delivery for safety.
• Dead Letter Queue for resilience.
• Heartbeats + Consumer Metrics for observability.

With these elements, you can confidently run real-time streaming pipelines in production for critical domains like e-commerce, finance, or logistics.