Kafka Producer API (C#)
Let’s explore APIs which Kafka provides for its producers, consumers and admins.
I’m C# developer so I’ll provide examples using Confluent.Kafka library. In fact, internally it delegates calls to high-performant library librdkafka.
From my experience, it’s easier to grasp new technology if it’s applied to a meaningful domain. Let’s imagine that we are building an application which handles weather information in numerous ways and computations are divided by microservices to increase the overall performance. All microservices need a source of weather info to make calculations and here Kafka comes into play. Suppose we have the following architecture:
Let’s start with the Weather Loader component which acts as a producer to Kafka.
In this article we will:
- Publish messages one by one
- Publish a batch of messages in a performant way
You can find the code base in my github repository. If you have any questions, feel free to ask!
Publish messages one by one
Let’s rush to the code and see what it takes to produce a message to a topic:
The producer provided by Confluent.Kafka API publishes messages and the goal is achieved.
But it’s not the best implementation from a performance point of view. Let’s take a closer look at what happens: for each message we are waiting for it to be delivered to all brokers. In my local setup delivery of a single message (without replication) took approximately 15 ms. Looks pretty fast but imagine you have to deliver 10000 messages and you’ll end up waiting for 150 seconds.
This issue motivates us to investigate further and luckily there is a way to increase the performance by publishing a batch of messages.
Publish a batch of message in a performant way
The following change brings us lightning-fast publication speed:
In this implementation delivery of a single message took approximately 0.05 ms which is 300 times faster than successive publication!
The reason for the performance acceleration is a producer queue. Producer queue is a local queue which stores messages until some number of messages are collected or some time passed to send accumulated messages as a single batch:
Conclusion
We’ve just figured out how to publish messages to a Kafka topic using the Producer API provided by Confluent.Kafka and also learnt about producer queue which is important to increase performance of your applications. If you’re looking for a Docker setup for Kafka, see my previous article: Docker setup for Kafka
Also there are some articles which may be interesting for Kafka learners:
