Thursday, July 8, 2021

Setup a Zookeeper Cluster

Apache Zookeeper is distributed coordination service to build distributed applications. Kafka uses the Zookeeper to manage cluster.


Kafka is included with Zookeeper and we can start as service however we can also install standalone zookeeper. This article demonstrates the setting up zookeeper cluster.

 

Prerequisite


Latest Zookeeper is required minimum JAVA version 1.8 and above (JAVA_HOME environment variable need setup in the system)

 

Need 7zip software to extract .tgz files in windows.

 

Software's and Tools


 

Windows 10

Java 1.8 or higher

Zookeeper 3.7.0

 

 

 

Download and Extract



Download latest zookeeper binary tar.gz from following apache zookeeper download page.

 

https://zookeeper.apache.org/releases.html

 

Direct link as follow

 

https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.7.0/apache-zookeeper-3.7.0-bin.tar.gz

 

Extract downloaded “apache-zookeeper-3.7.0-bin.tar.gz” using 7z software in local drive. We are going to setup 3 node cluster so clone extracted directory 2 more times.

 



Name the directories as follows


 

zookeeper-node1

zookeeper-node2

zookeeper-node3

 

 

 



Configure Cluster


Zookeeper Node1


Step:1


Data Directory


Create data directory in “zookeeper-node1” and we will use this directory path in zookeeper configuration.


Example:


C:\\kafka-workspace\\zookeeper-node1\\data

 





 

 

Step:2


Create “myid” file


We need to create “myid” file in data directory and update file name with node number that is “1”.





Step:3


Zookeeper Configuration


Now have to update few zookeeper configuration properties. Go to zookeeper-node1 directory and rename “zoo_sample.cfg” to “zoo.cfg

 



 

 

Add/Update “zoo.cfg” file with following properties.

 

 

clientPort=2181

dataDir=C:\\kafka-workspace\\zookeeper-node1\\data

 

autopurge.snapRetainCount=3

autopurge.purgeInterval=24

 

server.1=localhost:2666:3666

server.2=localhost:2667:3667

server.3=localhost:2668:3668

 

tickTime=2000

initLimit=10

syncLimit=5

 

 

 

 

server.<myid>=<hostname>:<leaderport>:<electionport>

 

 

We are setting up 3 node cluster so we have to configure server.[myid] property which have host name, leader and elector ports. Zookeeper use this information to form cluster.


Note:

we are using same machine to setup cluster so we have to make sure there is no port conflicts.

 

Zookeeper Node2 & Zookeeper Node2


Repeat Zookeeper Node1 Steps 1,2 and 3 for Zookeeper Node2 and Node3 and maintain unique port numbers.

 

Zookeeper Node2


myid file content is “2

 

Add/Update “zoo.cfg” file with following properties.

 

 

clientPort=2182

dataDir=C:\\kafka-workspace\\zookeeper-node1\\data

 

autopurge.snapRetainCount=3

autopurge.purgeInterval=24

 

server.1=localhost:2666:3666

server.2=localhost:2667:3667

server.3=localhost:2668:3668

 

tickTime=2000

initLimit=10

syncLimit=5

 

 

Zookeeper Node3


myid file content is “3

 

Add/Update “zoo.cfg” file with following properties.

 

 

clientPort=2183

dataDir=C:\\kafka-workspace\\zookeeper-node1\\data

 

autopurge.snapRetainCount=3

autopurge.purgeInterval=24

 

server.1=localhost:2666:3666

server.2=localhost:2667:3667

server.3=localhost:2668:3668

 

tickTime=2000

initLimit=10

syncLimit=5

 

 

Start Zookeeper Node 1,2 and 3


Zookeeper Node1


Open command prompt and locate to “zookeeper-node1” bin directory and use following command to start zookeeper server.

 

 

zkServer.cmd

 

 



 

Zookeeper console logs will give us the more details on server startup.

 




Note:


You may see “java.net.ConnectException: Connection refused: connect” warning and its due to other nodes are not yet started.


Zookeeper Node2


Open command prompt and locate to “zookeeper-node2” bin directory and use following command to start zookeeper server.

 

 

zkServer.cmd

 

 

Zookeeper Node3

 

Open command prompt and locate to “zookeeper-node3” bin directory and use following command to start zookeeper server.

 

 

zkServer.cmd

 

 

In the cluster only one node will be leader and other all nodes are followers. Leader and follower’s information can be view in the logs.

 

Node1( Leader)






Node2 (Follower)






Node3 (Follower)




 

Now successfully configured zookeeper cluster on windows.


References


https://zookeeper.apache.org/doc/current/index.html



Author





0 comments :

Post a Comment

Recent Posts

Recent Posts Widget

Popular Posts