Apache Kafka

Download Apache Kafka from https://kafka.apache.org/downloads. Unzip the folder in any drive of your choice. In case of Windows, try to keep the Kafka unzipped folder closer to the root drive path. Check the SS posted below, kafka directory is directly inside D drive.

  1. Start zookeeper, run the command inside the Kafka folder. For unix-based systems you need to run the .sh file. By default, zookeeper runs on 2181 port. You can check the logs to get your port number.

  2. Start Kafka server, by running the kafka-server-start.bat file in Windows & .sh file in Unix-based system. Also, give the config server.properties file. By default, it will run on 9092 port.

  3. Create a topic - You can specify the bootstrap server port, the topic name, replication factor and number of partitions you want. You can create n number of topics by running the same command, just change the topic name.

Download Offset Explorer -> https://www.kafkatool.com/download.html

Open the Offset Explorer GUI. Here, just create a cluster, and specify the Zookeeper port and click on Test -> after that you will be able to see the below GUI.

If you click on the topic name, you will see that currently no messages are present in any of the topics. Now, we need a Producer to send messages.

Open a CMD terminal, and run the below.

You'll get a prompt here. Simultaneously, open another CMD terminal and run the Consumer as well.

Now, start sending messages from the Producer prompt. Just write anything you want, it will be reflected in the Consumer terminal.

At this moment, you can open Offset Explorer and refresh topics/partitions to visualize the messages being sent. You can also send bulk CSV data (download from Internet).

Github Repo for Kafka Consumer & Producer Application :

  1. https://github.com/Ninjavin/Apache-Kafka-Producer

  2. https://github.com/Ninjavin/Apache-Kafka-Consumer

References : https://www.youtube.com/watch?v=xGwzuz8F9k0&list=PLVz2XdJiJQxwpWGoNokohsSW2CysI6lDc&index=1

Last updated