Friday, July 23, 2021

Kafka Cluster Monitoring with Prometheus and Grafana

Grafana is popular open-source solution for monitoring applications. It will provide graphical dashboards to build monitoring visualization.


Any graphical view required data or metrics, so metrics data will be provided by Prometheus.


Prometheus is another monitoring tools to pull data from different application with help of JMX Exporter Agent.


Grafana is ability to connect Prometheus to pull metrics data and it will be represented as graphical view. We can build nice dashboards to show metrics data in different graphical view in the Grafana web UI.

 






 

Prerequisite



Setup a Zookeeper Cluster


http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html

 


Setup a Kafka Cluster


http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html

 


Following are steps to Demonstrate Kafka Cluster Monitoring.

 

Setup Prometheus for Kafka Cluster

Install Grafana

Configure Prometheus Data source in Grafana

Create Kafka Cluster Dashboard

 


Setup Prometheus for Kafka Cluster


Follow the below article to setup Prometheus for Kafka cluster.


http://www.liferaysavvy.com/2021/07/kafka-monitoring-with-prometheus.html


 

Install Grafana



Follow the below article to install Grafana on windows.

 

http://www.liferaysavvy.com/2021/07/grafana-installation-on-windows.html



Configure Prometheus Data source in Grafana



Now its time to configure Prometheus Data source in Grafana. Prometheus already have metrics data which is pulled from Kafka with help of JMX exporter Java agent. That is already covered in the previous step.

 

Access Grafana Web UI with following URL

 

 

 

http://localhost:3000/

 

 

 

Home page click on Setting and Under Configuration click on “Data sources

 



 

Click on Add Data source.

 



 

Select Prometheus Data source in the List.

 



 

Provide Prometheus URL where its running. Default port is 9090. Once provided required information click on Test &Save.  Grafana successfully connected to Prometheus data source.

 


 

http://localhost:9090/

 

 




 

Create Kafka Cluster Dashboard



Grafana home page click on Dashboard icon and Click on Manage

 





There is free available Grafana dashboards for Kafka. Creating Grafana dashboard is very easy and its just need to import Dashboard JSON file.

 


Go to following URL and get Grafana Kafka overview dashboard file to local.

 


https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka-overview.json

 

OR

 


https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.0.1-post/jmxexporter-prometheus-grafana/assets/prometheus/grafana/provisioning/dashboards/kafka-overview.json


 

Import “kafka-overview.json” file into Grafana dashboard.

 


Click on import button.

 




 

Click on Upload JSON file button and Select “kafka-overview.json” file from local drive.




 


Once selected file, click on import then Dashboard will be imported into Grafana.

 








Very important is Job Name in Prometheus yml (prometheus.yml) file and Grafana Dashboard JSON file (kafka-overview.json) Job name should be same.

 





 

Go to Dashboards in the Grafana home page and select Kafka Overview Dashboard.

 


We Can see Dashboard with many panels and all metrics will be represented as Graphs.

 


Dashboard Screen: 1

 




 

Dashboard Screen: 2

 




 

Dashboard Screen: 3

 




 

Dashboard Screen: 4

 








Author


1 comment :

  1. Both text and pictures are great. I will visit often and subscribe.
    Please come to our amazing website and have fun. 오피사이트

    ReplyDelete

Recent Posts

Recent Posts Widget

Popular Posts