Kafka concepts

Kafka offers a reliable, scalable and high-performant solution for implementing a Pub-Sub model with asynchronous communication between services.

Let’s take a closer look at a Pub-Sub model and async communication to get the point.

Pub-Sub model

The Pub-Sub model is a pattern which allows us to decouple a caller and a receiver. This is extremely useful for building maintainable solutions because we’re not bothered with knowing who is going to process a call.

Let’s look at a figure that depicts the model:

You are sure to notice the third component between a publisher and a subscriber — an event channel. It’s an independent component, for example, message broker, that receives data from a publisher and carries it down to a subscriber.

This channel allows a publisher not to know anything about a subscriber. BTW, this makes a difference between Observer pattern and Pub-Sub model. For more information you can visit Observer vs Pub-Sub pattern which explains the difference pretty clearly.

Async messaging

Systems can interact with each other either synchronously (sync) or asynchronously (async).

Sync communication forces the caller to wait for the receiver to process the request. In contrast, async communication doesn’t enforce the caller to do that: the caller just continues its execution without any response from a receiver.

At first glance, sync communication looks as a robust and reliable way of building services communication but once we take a closer look to it, we will discover a bunch of drawbacks:

1. Tight coupling to the receiver which results in:

• Service unavailability during the receiver redeployment
• Lack of scaling opportunities since we are coupled to a single instance of a receiver

2. Intermittent failures due to unreliable network

3. Increased latency due to:

• unbounded delays in unreliable networks
• overloaded receiver when there are lots of incoming requests
• network congestion (each caller caller needs to hold a connection to a receiver)
• …

On one hand, async communication solves these issues and also brings other benefits:

• Block-freeing a caller
• Increased throughput because the network is not used when there is no data to send
• Adoption of the underlying network nature which is based on async packet delivery

On the other hand, async communication brings another challenges to consider:

• How would we know that the request was processed/failed or is being processed right now?
• Can a message be delivered more than one time?
• Not suitable for real-time processing because a request may stay not processed for weeks
• …

To get a better understanding of these subtle issues you can read an excellent book Designing Data-Intensive Application by Martin Kleppmann.

Next

Next we are going to look at Kafka components and their role in the whole system. Stay tuned!