Kafka with NodeJS, ExpressJS, and WebSocket

In this article, I will try to set up a NodeJS project with ExpressJS (for http REST API communication) and express-ws for consumer API push events and explain how we can interact with the Kafka server using javascript APIs.

Create NodeJS application with Kafkajs
We will use KafkaJS npm package to communicate with Kafka server. Let’s create a NodeJS project and install Kafkajs, ExpressJS, express-ws packages, if it’s not set up using npm init, you may want to do that step to create package.json before you install those npm packages.

npm init
npm i kafkajs

First, we will just write a static code to interact with Kafka from the NodeJS application. Publish message and Consume message from the Kafka Server. Complete code for this simple interaction between NodeJS and Kafka Server can be found here.
If you try to produce a message to a new topic, i.e. without creating a topic first on Kafka server you will get an error as soon as you will run your producer

"There is no leader for this topic-partition as we are in the middle of a leadership election"

Create a topic on Kafka Server
You can’t send a message to a Kafka server without a topic created on it. You can create a topic using the admin APIs of Kafka, which kafkajs module exposes, once you create a topic, you will no longer receive the above-mentioned error. If you try to publish without creating a topic, Kafka Server will create a topic for you but you will receive an error as explained above.

Another thing I noticed is when we are listening to a message stream on a particular topic, we have to specify the groupId while creating a Consumer API object. Without groupId you will get an error when you will try to run a consumer app. GroupId can be any unique string value. Kafka assigns the partitions of a topic to the consumer in a group, such that each partition is consumed by exactly one consumer in a group this way Kafka ensures that the message is read by a single consumer in the group.
To check the number of consumer group ids from the command window, if we are running Kafka server locally, we can use the following command on Windows OS:

.\bin\windows\kafka-consumer-groups.bat --list --bootstrap-server localhost:9092

Add ExpressJS and express-ws packages to NodeJS Application
For API communication with NodeJS application we will add expressJS and express-ws npm package to project, complete code for the project can be found here

npm i kafkajs express express-ws 

Create Kafka Topic using Rest API
Now we will write REST APIs to interact with NodeJS application using ExpressJS and then use the code we wrote earlier to use kafkajs APIs to interact with Kafka Server.
We will create POST API to create a topic on Kafka server using Kafka Admin-client API through KafkaJS. I am using the code we wrote previously to create a topic on Kafka server, but wrapping it with POST API, so that the user can send data through API call.

Push message to Kafka Queue using POST API
We will create POST API to send messages to the NodeJS application and then using Producer API sends a message to Kafka Server Queue.

Subscribe to messages from Kafka using WebSocket Connection
We will create a WebSocket API on the server side so that client app can create a WebSocket connection to our NodeJS application and NodeJS application Consumer API can send back messages on WebSocket to all connected Clients subscribed for a particular topic, as and when Consumer receives new messages posted by Producer on a particular topic. So Consumer API will be using a push mechanism to send messages to connected clients, rather than Client pulling the message from the queue.

Testing the complete application
Make sure Zookeeper and Kafka Server are running. You can have multiple Kafka server broker instance running, or just single, if you haven’t seen the previous post on how to setup Kafka Server Locally please check it and setup your Kafka Local environment first, before you can setup and test NodeJS locally, make sure you update the config file present in util folder of the project. You can test REST APIs using Postman.
Step1: Create a topic using Postman
As screenshot below, using POST API you can create a topic on Kafka Server. We will use this topic later to send message to using Producer API and consume message using Consumer API.

Step2: Post Messages to topic created in previous step
Let’s use the above created topic and use POST API as displayed in screenshot below to send message to express server and it will send it to Kafka Server.

Step3: Start Client-APP with WebSocket connection to Consumer API
For testing Consumer API, you can use the Node Client app, provided with the project or you can use some 3rd party apps like chrome extension to do WebSocket API testing. You can use “Smart Websocket Client” or some other chrome extension to test it. Below is the screenshot of the client app receiving messages from NodeApp, make sure you send the topic name to listen to when the connection is opened.

This was my end to end local development and use of some of Kafka APIs with NodeJS and Javascript package which provided interaction with Kafka server. The KafkaJS has very detailed documentation with lots of examples on all the APIs which you can use to interact with Kafka and detailed explanation of each Kafka API.

My next target is create a serverless environment and cloud and use Kafka either in Azure or AWS and use Functions or Lambda to communicate with Kafka server. Will share my experience on how to setup serverless environment in cloud offering. Although both Azure and AWS have there own internal messaging solution offerings like Event-Grid from Azure or Kinesis from AWS.