Kafka is distributed event streaming software based on publish and subscribe to model. Its very powerful streaming software used by many organizations.
More details are find in the below link
Prerequisite
Kafka is required minimum JAVA version 1.8 and
above (JAVA_HOME environment variable need setup in the system)
Need 7zip software to extract .tgz files
in windows.
Download and Extract
Download latest Kafka software from below location.
http://kafka.apache.org/downloads.html
Direct link is below
https://www.apache.org/dyn/closer.cgi?path=/kafka/2.8.0/kafka_2.13-2.8.0.tgz
Extract “kafka_2.13-2.8.0.tgz” in local drive.
Start Zookeeper
Kafka is required zookeeper service to manage Kafka
cluster. Latest version of Kafka has embedded zookeeper. Older version of Kafka
was required to install zookeeper as separate service outside of Kafka.
Open “zookeeper.properties” file in the config
directory and update location for zookeeper data directory.
|
dataDir=C:\\kafka-workspace\\kafka_2.13-2.8.0\\zookeeper\\data |
Zookeeper uses the default port 2181. There are many zookeeper properties can update
based on requirement.
Open command prompt and locates to Kafka root
directory and use following command to start zookeeper service.
|
bin\windows\zookeeper-server-start.bat
.\config\zookeeper.properties |
Finally, zookeeper is running on localhost:2181
We can see startup logs in console.
Once zookeeper started successfully, we can see data
directory with snapshot files.
Start Kafka
Locate to Kafka config directory and update “server.properties”
Update Kafka logs directory to below location or any
other valid directory.
|
log.dirs=C:\\kafka-workspace\\kafka_2.13-2.8.0\\kafka-logs |
It also have important property “zookeeper.connect”
and it uses the default value. If any changes in port or host name need to
update same property.
|
zookeeper.connect=localhost:2181 |
Open command prompt and locate to Kafka root
directory. Run following command to start Kafka broker service.
|
bin\windows\kafka-server-start.bat
.\config\server.properties |
Once Kafka started successfully we can see “started
(kafka.server.KafkaServer)” message in the logs.
Verify Installation
We create Kafka topic and produce some messages on topic.
Other end we will run consumer to receive messages.
Create Kafka Topic
Open command prompt and locate to Kafka bin windows
directory. Use following create topic command.
|
kafka-topics.bat
--create --zookeeper localhost:2181 --replication-factor 1 --partitions 1
--topic first-kafka-topic |
List topics
There are many other commands we can see by typing --help
Start Producer
Open command prompt and locate to Kafka bin windows
directory. Use following producer command to start producer and post messages
on specific topic.
|
kafka-console-producer.bat
--broker-list localhost:9092 --topic first-kafka-topic |
Broker-list is Kafka server and which is running on 9092
default port.
Start Consumer
Open command prompt and locate to Kafka bin windows
directory. Use following consumer command to start consumer.
|
kafka-console-consumer.bat --bootstrap-server
localhost:9092 --topic first-kafka-topic |
Now type some messages on producer command prompt and
same messages are receiving at consumer command prompt. This confirms the installation
of Kafka is successful.
References
https://kafka.apache.org/quickstart
Posts
0 comments :
Post a Comment