Grafana is popular open-source solution for monitoring applications. It will provide graphical dashboards to build monitoring visualization.
Any graphical view required data or metrics, so metrics data will be provided by Prometheus.
Prometheus is another monitoring tools to pull data from different application with help of JMX Exporter Agent.
Grafana is ability to connect Prometheus to pull
metrics data and it will be represented as graphical view. We can build nice
dashboards to show metrics data in different graphical view in the Grafana web
UI.
Prerequisite
Setup a Zookeeper Cluster
http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html
Setup a Kafka Cluster
http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html
Following are steps to Demonstrate Kafka Cluster
Monitoring.
Setup Prometheus for Kafka Cluster
Install Grafana
Configure Prometheus Data source in
Grafana
Create Kafka Cluster Dashboard
Setup Prometheus for Kafka Cluster
Follow the below article to setup Prometheus for Kafka cluster.
http://www.liferaysavvy.com/2021/07/kafka-monitoring-with-prometheus.html
Install Grafana
Follow the below article to install Grafana on
windows.
http://www.liferaysavvy.com/2021/07/grafana-installation-on-windows.html
Configure Prometheus Data source in Grafana
Now its time to configure Prometheus Data source in
Grafana. Prometheus already have metrics data which is pulled from Kafka with
help of JMX exporter Java agent. That is already covered in the previous step.
Access Grafana Web UI with following URL
|
|
Home page click on Setting and Under Configuration
click on “Data sources”
Click on Add Data source.
Select Prometheus Data source in the List.
Provide Prometheus URL where its running.
Default port is 9090. Once provided required information click on Test &Save.
Grafana successfully connected to
Prometheus data source.
|
|
Create Kafka Cluster Dashboard
Grafana home page click on Dashboard icon and
Click on Manage
There is free available Grafana dashboards for Kafka. Creating Grafana dashboard is very easy and its just need to import Dashboard JSON file.
Go to following URL and get Grafana Kafka overview
dashboard file to local.
https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka-overview.json
OR
Import “kafka-overview.json” file into Grafana
dashboard.
Click on import button.
Click on Upload JSON file button and Select “kafka-overview.json”
file from local drive.
Once selected file, click on import then Dashboard
will be imported into Grafana.
Very important is Job Name in Prometheus yml (prometheus.yml)
file and Grafana Dashboard JSON file (kafka-overview.json) Job name
should be same.
Go to Dashboards in the Grafana home page and select Kafka
Overview Dashboard.
We Can see Dashboard with many panels and all metrics
will be represented as Graphs.
Dashboard Screen: 1
Dashboard Screen: 2
Dashboard Screen: 3
Dashboard Screen: 4
Posts
