Kafka Client APIs Overview
Introduction
In this article, I will give an overview of different Client APIs that are available in Kafka and when to use them to solve different use cases.
Getting Started with Kafka
Kafka is a distributed streaming platform, which is really popular and a default choice in many organizations for streaming data.
- It is a high throughput, scalable , highly available system.
- It has the capability to store the data permanently for some specific use-cases.
It is used by thousands of companies for high-performance data pipelines, streaming analytics, data integration, and mission-critical applications.
If you wish to learn more about Kafka then please check this link.
At the heart of Kafka, we have the Kafka broker.
- Kafka broker is where the actual data resides.
- Data in Kafka are stored in the form of topics.
- You can relate this as the equivalent of a Table in a Relational DB.
- Data in Kafka are stored in the form of topics.
- Kafka broker is also responsible for handling request from the clients efficiently.
- It also is responsible for a lot of other optimizations such as high throughput, data replication, and etc.,
In this article, we are going to focus on the different ways to interact with Kafka Broker from a KafkaClient perspective.
Kafka Clients APIs
Fundamentally we have four APIs that we can use to interact with Kafka:
- Kafka Producer API
- Kafka Consumer API
- Kafka Connect API
- Source Connector - Reading Data from an external data source and send it to the Kafka topic.
- Sink Connector - Reading Data from a Kafka topic and send it to an external data source (eg.,DB).
- Kafka Streams API
| API | Description | Usage pattern | Example |
|---|---|---|---|
| Producer API | This is used to produce or create data in to the Kafka topic. | Common usage pattern is to have the producer sit behind a Rest API. | 1. Online Food delivery updates to the user.2. Click Streams. 3. Uber driver location updates. |
| Consumer API | This is used to read the records from a Kafka Topic and act on these events | Read an event and act on it. | 1. Your online food order is approaching notification.2. Your uber driver is approaching closer notification.3. Huge banking transaction alert. |
| Source Connector API(Kafka Connect) | This is used to read data from an extenal data soucrce and publish that event in to the Kafka topic. Behind the scenes it uses the Producer API to write the data in to the Kafka Topic. | Connect an external data source by creating a source connector and have that data flow into the Kafka Streaming Pipelines. Most of the usecases can be solved without writing any code. | 1. Connect to a legacy DB and have that data flow in to the modern streaming pipelines. 2. Data Migration using event driven approach to move away from legacy DB. |
| Sink Connector API(Kafka Connect) | This is used to emit data from a Kafka topic to an external datasource. Behind the scenes it uses the Consumer API to read the data from the Kafka Topic. | Specifically used to replicate data from a Kafka topic to another datasource such as MongoDB, RelationDB ,Elastic and etc., Most of the usecases can be solved without writing any code. | 1. Use Kafka as a Streaming platform to replicate data from one data source into another. |
| Streams API | This is used to stream the data the from the kafka topic and write it back to the kafka topic. Behind the scenes it uses the Producer API and Consumer API to read and write data into the Kafka Topic. | Specifically used to data enrichment, aggregating data, joining data from multiple kafka topics. | 1. Total number of orders recieved on a given time frame. 2. Total number of active users. 3. Total Revenue generated in a given time period. 4. Joining Data from muliple kafka topics. |
Conclusion
I hope this article helped you to understand the different Kafka Client APIs available and its usage to solve different use-cases.