Installing Zookeeper and Kafka

22 Nov 2017

I've been learning Kafka, and figured I'd leave some installation notes here.

Installing Kafka is really installing Java and then installing Zookeeper and then installing Kafka.

Java

Download the latest Java 8 from Oracle:

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

$ su -

# cd /usr/local

# cp /home/mwood/Downloads/jdk-8u152-linux-x64.tar.gz .

# tar -xzvf jdk-8u152-linux-x64.tar.gz 

# ./jdk1.8.0_152/bin/java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

# ./jdk1.8.0_152/bin/javac -version
javac 1.8.0_152

# rm jdk-8u152-linux-x64.tar.gz 

# exit

$ cd $HOME
$ vim .bashrc 

Edit .bashrc so that a JAVA_HOME is exported and so that PATH has $JAVA_HOME/bin in it.

############## JAVA #####################

export JAVA_HOME=/usr/local/jdk1.8.0_152
export PATH="$JAVA_HOME/bin:$PATH"

#########################################

Now log out and back in again so that you don't have to worry about sourcing the new .bashrc all the time.

Be sure java is in your path:

$ java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

Now it's time to instal...

Zookeeper

This link is useful-ish:

http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html

$ cd Downloads/
$ wget http://mirror.metrocast.net/apache/zookeeper/zookeeper-3.4.11/zookeeper-3.4.11.tar.gz

Let's just run Zookeeper as the local user for now.

$ cd ..
$ cp Downloads/zookeeper-3.4.11.tar.gz .
$ tar -xzvf zookeeper-3.4.11.tar.gz 
$ cd zookeeper-3.4.11/

Let's keep the data local to our zookeeper dir, so that blowing away zookeeper is as easy as nuking one dir, rather than a few dirs spread around the filesystem.

$ mkdir data

$ cat > conf/zoo.cfg
tickTime=2000
dataDir=/home/mwood/zookeeper-3.4.11/data
clientPort=2181
^D

$ ./bin/zkServer.sh start

$ ./bin/zkServer.sh stop

That seems to have worked.

While it was running, we were able to insert and get keys with the provided CLI:

$ ./bin/zkCli.sh -server 127.0.0.1:2181

Later, maybe we can play around with getting this running on multiple machines.

But for now, let's instal...

Kafka

(Do we need zookeeper to be running? For now, I am leaving it turned off.)

$ cd /home/mwood/Downloads/
$ wget http://download.nextag.com/apache/kafka/1.0.0/kafka_2.11-1.0.0.tgz

The same way we did with zookeeper, let's install kafka in our home directory, instead of installing it as root and spreading its files around the filesystem.

$ cd ..
$ cp ./Downloads/kafka_2.11-1.0.0.tgz .
$ tar -xzvf kafka_2.11-1.0.0.tgz 

Now let's start zookeeper again

$ ./zookeeper-3.4.11/bin/zkServer.sh start

I'm assuming zookeeper is running on a standard port, so now let's start Kafka and see how that works.

$ cd kafka_2.11-1.0.0/
$ vim config/server.properties 

I don't like the look of this line:

log.dirs=/tmp/kafka-logs

so let's change it to this:

log.dirs=/home/mwood/kafka_2.11-1.0.0/logs

and then create that directory:

$ mkdir /home/mwood/kafka_2.11-1.0.0/logs

Let's see if we can get it to start

$ ./bin/kafka-server-start.sh ./config/server.properties 

That seems to have worked. It has also taken over our console with logs, so let's go to another terminal and interact with Kafka.

May as well try to create a topic.

$ ./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Created topic "test".

That seems to have worked.

$ ./bin/kafka-topics.sh --list --zookeeper localhost:2181
test

$ ./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
>This is a message
>This is another message
>^D

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
^CProcessed a total of 2 messages

Usefully, we can run this again, because Kafka holds on to the messages:

$ ./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
^CProcessed a total of 2 messages