Five Kafka patterns for event-driven microservices

Why microservices need events

The usual explanation of event-driven architecture is that it decouples services. That is true but incomplete. The more precise benefit is that it decouples them at the data boundary rather than the network boundary. When services publish events to Kafka topics instead of calling each other's APIs, you gain independent deployability, replay, and the ability to add a new consumer without touching the producer.

At Rocket India, on a regulated mortgage servicing platform, this was a practical necessity rather than a preference. With 50+ source systems producing data and 12+ downstream teams needing to react to business events, point-to-point calls would have produced a dependency graph nobody could maintain. Kafka topics became the canonical event record for the platform.

Event-driven architecture is not one pattern. It is a family of them, each solving a different problem, and using the wrong one makes a system harder to reason about. These are the five I find most useful and most often misapplied.

Pattern 1: event notification

The simplest pattern. A service publishes a minimal event to say something happened, usually an event type and an entity ID. Consumers that care fetch the current state from the producing service's API.

{
  "eventType": "customer.profile.updated",
  "customerId": "cust-98765",
  "occurredAt": "2026-05-05T10:30:00Z"
}

When to use it: the event data is large or changes often and you do not want to copy state into every consumer. Consumers need the state at processing time, not at event time. The producer owns the canonical record and consumers should always read from it.

When not to use it: consumers cannot afford a synchronous API call after receiving the event. The producer may be down when the consumer runs, which is temporal coupling. You need to replay history, because replaying notifications does not reproduce the state at the time of each event.

The hidden coupling. Event notification is often mistaken for a fully decoupled pattern. It is not. Consumers stay coupled to the producer's API, and the coupling simply moves from compile time to runtime. If that API is unavailable, the consumer is blocked. For consumers that must stay highly available, prefer event-carried state transfer.

Pattern 2: event-carried state transfer

Event-carried state transfer, a term from Martin Fowler, puts the relevant state in the event itself. Consumers need no callback to the producer, because everything required is in the message.

{
  "eventType": "payment.processed",
  "paymentId": "pay-44321",
  "customerId": "cust-98765",
  "amount": 1250.00,
  "currency": "USD",
  "status": "COMPLETED",
  "processedAt": "2026-05-05T10:30:00Z",
  "loanId": "loan-11234",
  "remainingBalance": 187500.00
}

When to use it: consumers must work independently of the producer's availability. Replay has to reproduce the state at the time of each event. Several consumers need the same data and you want to avoid N×M API calls. Most Kafka-native architectures end up here.

When not to use it: the state is a large document, because Kafka is not designed for big messages. Use the claim check pattern instead: store the payload in object storage and put the reference in the event. Also avoid it when the payload contains sensitive fields that not every consumer should see, because every subscriber receives every field.

Schema evolution matters most here. Events are persisted in the log and replayed, so the schema is effectively a public API. Add fields with defaults. Never remove or rename required fields without a versioning strategy. Use Confluent Schema Registry with backward-compatible Avro or Protobuf schemas to enforce the contract at the producer.

Pattern 3: transactional outbox

The outbox solves the dual-write problem: a service must update its database and publish a Kafka event as one atomic act. Without it, you can write to the database and fail to publish, or the reverse, and leave the system inconsistent.

The pattern works like this:

1. The service writes its business change and an outbox record in the same database transaction. The outbox row holds the topic, key, payload and timestamp.
2. A separate relay reads the outbox and publishes to Kafka. It can poll the table on a schedule, or use change data capture, with a Debezium connector tailing the database transaction log.
3. Once the event is confirmed published, the outbox row is marked processed or deleted.

-- In the service's database transaction
BEGIN;
  INSERT INTO payments (id, amount, status) VALUES ('pay-44321', 1250.00, 'COMPLETED');
  INSERT INTO outbox_events (id, topic, key, payload, status, created_at)
    VALUES (gen_random_uuid(), 'payment.processed', 'cust-98765',
            '{"paymentId":"pay-44321",...}', 'PENDING', NOW());
COMMIT;

-- A Debezium CDC connector tails the outbox table
-- and publishes to Kafka automatically

When to use it: whenever a service must update its own database and reliably notify consumers. The alternative, a distributed two-phase commit, adds coordination overhead that hurts throughput. At Rocket India, payment events, loan status changes and customer data updates that needed guaranteed downstream notification used the outbox with Debezium.

The CDC relay advantage. Tailing the transaction log removes the polling interval, so the event is published within milliseconds of the commit. A polling relay is simpler to build but adds latency equal to the poll interval, typically 1 to 15 seconds.

Pattern 4: CQRS with Kafka as the synchronisation bus

Command query responsibility segregation separates the write model, which handles commands that change state, from the read model, which serves queries. Kafka carries state changes from the write side to read-side materialized views, so each side can scale and be optimised on its own.

On a mortgage platform, the loan servicing service owns the authoritative loan state and publishes loan.status.changed events. A read-side service consumes them and maintains a denormalised view in a different store: Elasticsearch for text search, Redis for low-latency lookups, Snowflake for analytics. Each store is tuned to its query pattern without touching the write database.

When to use it: read and write loads differ sharply. Different consumers need the same data in different shapes. You need to scale reads independently of writes. One topic should power several specialised query interfaces.

The eventual consistency trade-off. After a command, the event is published, consumed and written to the read store. That pipeline adds latency, typically seconds in a well-tuned setup. A user who writes and immediately reads back may see stale data. That is usually fine for reporting. For interactive screens it needs explicit handling, such as optimistic UI updates or read-your-writes from the write-side API.

Rebuilding read models from the topic. With sufficient retention, a read store can be rebuilt after a consumer bug, a schema migration or a move to a new store type. Replay the topic from offset 0 and reconstruct the view without touching the write database. The read model becomes disposable and the write model stays the source of truth.

Pattern 5: dead letter stream

This pattern handles consumer failures without data loss or pipeline stalls. When a consumer cannot process a message because of a deserialisation error, a validation failure, an unavailable downstream service or a business rule violation, the message goes to a dead letter topic instead of being retried forever or dropped.

A robust implementation has three parts:

1. Error capture. The consumer catches the failure and publishes the message to a dead letter topic with context: original topic, partition and offset, error type, error message, stack trace and timestamp. That context is what makes diagnosis and reprocessing possible.
2. Monitoring. Alert on consumer lag for the dead letter topic. Any growth signals a systematic failure. In production, alert when it is above zero.
3. Reprocessing. A separate service replays messages to the original topic, or a corrected path, once the cause is fixed. Make it a manual trigger for one-off failures and automatic for transient ones such as a short downstream outage.

// Consumer error handler: publish to the DLQ instead of stopping
@KafkaListener(topics = "payment.processed")
public void handlePayment(ConsumerRecord<String, String> record) {
  try {
    processPayment(record.value());
  } catch (ProcessingException e) {
    DeadLetterRecord dlr = DeadLetterRecord.builder()
        .originalTopic(record.topic())
        .originalPartition(record.partition())
        .originalOffset(record.offset())
        .payload(record.value())
        .errorType(e.getClass().getSimpleName())
        .errorMessage(e.getMessage())
        .failedAt(Instant.now())
        .build();
    deadLetterProducer.send("payment.processed.dlq", dlr);
  }
}

When to use it: always. The real question is what your reprocessing workflow looks like. Consumers without a dead letter strategy either stall when configured to stop on errors, or silently drop messages when errors are swallowed. Both are worse than parking the message.

Connector support. Kafka Connect sink connectors, including managed ones on Confluent Cloud and the Snowflake connector, can route bad records to a dead letter queue through the standard errors.tolerance and errors.deadletterqueue.topic.name settings. For custom consumers, build it into your own error path.

How these patterns fit together on a customer data platform

On the Customer Data Platform at Rocket India, the patterns worked as one system:

• Transactional outbox published loan status changes, payments and customer updates from the source-of-record database without dual-write risk, with Debezium tailing the outbox tables.
• Event-carried state transfer put full customer and loan state on each event, so downstream consumers did not call back to source systems and cascade failures from upstream coupling went away.
• CQRS powered the unified customer view: writes stayed distributed across source systems, and a dedicated consumer materialised a unified customer record into a read store that downstream analytics and customer-facing features read from.
• Dead letter streams on every consumer kept a single bad message from stalling a pipeline. At millions of events a day, schema surprises and validation edge cases are certain, and the DLQ turned them into monitored exceptions rather than incidents.
• Event notification was used selectively for high-frequency, low-urgency signals such as document upload confirmations and preference updates, where the consumer always wanted current state.

Pitfalls that cost teams months

Publishing before the database commit. If a service publishes to Kafka before or outside its transaction, a rollback leaves an event for data that does not exist, and consumers act on it. The outbox is the fix. Do not publish to Kafka inside an application-level transaction without outbox semantics.

Ordering by timestamp across partitions. Kafka message timestamps reflect production time, not when the business event happened. Clock skew, network jitter and producer retries mean timestamps are not monotonic across partitions. Use offsets for ordering within a partition. For global ordering, use a single partition, which caps throughput at one consumer, or add a watermark-based ordering window in your stream processor.

Consumers that cannot handle duplicates. At-least-once delivery means the same message can arrive twice after a restart, a rebalance or a timeout. Every consumer should be idempotent, so processing an event twice gives the same result as once. The simplest approach is an event ID on every event and a deduplication check in a database or Redis before processing.

Events designed as implementation details. An event is a public contract once the first consumer subscribes. Teams that expose internal field names, mirror the database schema or add undocumented fields build debt that gets expensive as consumers multiply. Design events like public APIs from day one, with clear names, a documented schema and an explicit versioning strategy.

Frequently asked questions

What is the difference between event notification and event-carried state transfer?

Event notification is a minimal signal that something happened, and the consumer must call back to the producer to fetch current state. Event-carried state transfer embeds the relevant state in the event so consumers can process it on their own. The second is more resilient because there is no upstream coupling at processing time, but it needs careful schema management because the event becomes a contract for every consumer.

Is the transactional outbox pattern expensive to implement?

The database change is small: an outbox table, written in the same transaction as the business data. The relay is where the cost sits. A polling relay is simple to build but adds the poll interval as latency. A CDC relay with Debezium has near-zero added latency but needs access to the database transaction log and connector infrastructure. For most services where reliable publishing matters, the setup cost is paid back quickly.

Does CQRS require event sourcing?

No. CQRS separates the read and write models. Event sourcing stores state as a sequence of events instead of the current state. They are often used together but neither requires the other. You can run CQRS with an ordinary database on the write side and Kafka-driven materialized views on the read side. Event sourcing adds real complexity and suits audit-heavy domains more than general application architecture.

How do I choose between these five patterns for a new microservice?

Ask four questions. Does the consumer need state as it was at event time (event-carried state transfer) or current state (event notification)? Does the producer need to update its database and publish atomically (transactional outbox)? Does the consumer need to survive bad messages without losing data (dead letter stream, always)? Does the same write-side data have to serve several query shapes (CQRS)? A common baseline is outbox, event-carried state transfer and a dead letter stream, with CQRS added when read and write scaling diverge.

How large should a Kafka event payload be?

The default broker limit is about 1 MB (message.max.bytes). In practice, keep payloads well under 100 KB for healthy throughput and consumer memory. For larger payloads such as documents or big JSON blobs, use the claim check pattern: store the payload in object storage and put only the reference in the event.