Setting Up Apache Kafka locally

November 10th, 2017

This Guide is for the linux based OS & directories in the Kafka setup can be slightly different in the windows setup. Mainly we deal with the /bin folder.

  • for mac - kafka_2.10-0.10.2.0/bin/
  • for Windows - kafka_2.10-0.10.2.0\bin\windows\


Note : Every command must be executed in a separate terminal or console except for creating the topic and ending a server.


Step 1 - Download the source code & extract it

  • Download Apache kafka from here

  • Navigate to the downloaded directory & untar the kafka_2.10-0.10.2.0.tar

  tar -xzf kafka_2.10-0.10.2.0.tgz
  cd kafka_2.10-0.10.2.0


Step 2 - Starting the Server

  • You first need to start a ZooKeeper serve, since Kafka uses Zookeeper for the tasks such as Electing a controller, Topic configuration, ACLs & for many more.

  • As you are now in the kafka_2.10-0.10.2.0 directory execute the following command

  bin/zookeeper-server-start.sh config/zookeeper.properties
  • Starting the Kafka Server
  bin/kafka-server-start.sh config/server.properties


Step 3 - Creating a Topic

  • Creating a topic located at zookeeper at the localhost:2181 named mytopic with a single partition & single replica.
  bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic mytopic
  • To ensure that our topic is created, execute following command
  bin/kafka-topics.sh --list --zookeeper localhost:2181


Step 4 - Running the Producer

  • This is for feed the mytopic Topic (do it in a separate console)
  bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic
  • You can send messages to the Kafka cluster from the console even except the standard file inputs. just type the message in the console.
  This is a Sample message


Step 5 - Running the Consumer

  • This is for listen to the producer at that port (do it in a separate terminal).
  • Following command will listens for the topic inputs and outputs
    bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic mytopic
  • If you type messages into the producer terminal you should see them appear in the consumer terminal.


Step 6 - Stopping the Kafka Server

bin/kafka-server-stop.sh


Step 7 - Stopping the Zookeeper Server

bin/zookeeper-server-stop.sh

Omal Perera

Software Engineer at 99X Technologies
Full Stack Developer expertise in React-native
Information Systems graduate.