Kafka Cluster Monitoring with Prometheus and Grafana

Grafana is popular open-source solution for monitoring applications. It will provide graphical dashboards to build monitoring visualization.

Any graphical view required data or metrics, so metrics data will be provided by Prometheus.

Prometheus is another monitoring tools to pull data from different application with help of JMX Exporter Agent.

Grafana is ability to connect Prometheus to pull metrics data and it will be represented as graphical view. We can build nice dashboards to show metrics data in different graphical view in the Grafana web UI.

 

 

Prerequisite

Setup a Zookeeper Cluster

http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html

 

Setup a Kafka Cluster

http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html

 

Following are steps to Demonstrate Kafka Cluster Monitoring.

 

Setup Prometheus for Kafka Cluster

Install Grafana

Configure Prometheus Data source in Grafana

Create Kafka Cluster Dashboard

 

Setup Prometheus for Kafka Cluster

Follow the below article to setup Prometheus for Kafka cluster.

http://www.liferaysavvy.com/2021/07/kafka-monitoring-with-prometheus.html

 

Install Grafana

Follow the below article to install Grafana on windows.

 

http://www.liferaysavvy.com/2021/07/grafana-installation-on-windows.html

Configure Prometheus Data source in Grafana

Now its time to configure Prometheus Data source in Grafana. Prometheus already have metrics data which is pulled from Kafka with help of JMX exporter Java agent. That is already covered in the previous step.

 

Access Grafana Web UI with following URL

 

http://localhost:3000/

 

Home page click on Setting and Under Configuration click on “Data sources”

 

 

Click on Add Data source.

 

 

Select Prometheus Data source in the List.

 

 

Provide Prometheus URL where its running. Default port is 9090. Once provided required information click on Test &Save.  Grafana successfully connected to Prometheus data source.

 

http://localhost:9090/

 

 

Create Kafka Cluster Dashboard

Grafana home page click on Dashboard icon and Click on Manage

 

There is free available Grafana dashboards for Kafka. Creating Grafana dashboard is very easy and its just need to import Dashboard JSON file.

 

Go to following URL and get Grafana Kafka overview dashboard file to local.

 

https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka-overview.json

 

OR

 

https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.0.1-post/jmxexporter-prometheus-grafana/assets/prometheus/grafana/provisioning/dashboards/kafka-overview.json

 

Import “kafka-overview.json” file into Grafana dashboard.

 

Click on import button.

 

 

Click on Upload JSON file button and Select “kafka-overview.json” file from local drive.

 

Once selected file, click on import then Dashboard will be imported into Grafana.

 

Very important is Job Name in Prometheus yml (prometheus.yml) file and Grafana Dashboard JSON file (kafka-overview.json) Job name should be same.

 

 

Go to Dashboards in the Grafana home page and select Kafka Overview Dashboard.

 

We Can see Dashboard with many panels and all metrics will be represented as Graphs.

 

Dashboard Screen: 1

 

 

Dashboard Screen: 2

 

 

Dashboard Screen: 3

 

 

Dashboard Screen: 4

 

Author

Meera Prince

Liferay Contributor of the Year 2016
Liferay Top Contributor Award Winner 2015

Liferay Contributor of the Year 2014

Liferay Top Contributor Award Winner 2013

http://www.liferaysavvy.com