Prometheus
is monitoring opensource tool. Previous Article we have enabled Zookeeper
metrics and monitor in the Prometheus.
http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html
This article demonstrates the Kafka monitoring using Prometheus.
Prometheus uses the JMX Exporter agent to get all JVM metrics from Kafka.
We need to run JMX Exported Java agent in each server
where Kafka is running.
Software’s and Tools
|
Windows 10
Java 1.8 or
higher
Zookeeper 3.7.0
Kafka 2.8
JMX Exporter Java
Agent 0.15.0
prometheus-2.28.1
|
Prerequisite
Set up Zookeeper Cluster
http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html
Set up Kafka Cluster
http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html
- Download JMX Exporter
- Configure JMX Exporter for Kafka
- Start Kafka with JMX Exporter Agent
- Install Prometheus
- Configure Prometheus scrape for Kafka
- Verify Kafka brokers in Prometheus
Download and Start JMX Exporter
Download JMX Exporter jar file from following
location.
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/
Direct link for latest JMX Exporter JAR
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar
OR
https://github.com/LiferaySavvy/kafka-monitoring/raw/master/jmx_prometheus_javaagent-0.15.0.jar
Download and get the jar to local directory.
Configure JMX Exporter for Kafka
We need to configure the exporter configuration for
Kafka such a way JMX service exports the required metrics from Kafka.
Get the following JMX Exporter configuration file from
below location and place it in local drive.
https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.1.0-post/shared-assets/jmx-exporter/kafka_broker.yml
OR
https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka_broker.yml
We can use below one too.
|
lowercaseOutputName:
true
rules:
# Special cases
and very specific rules
- pattern :
kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+),
partition=(.*)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
topic: "$4"
partition: "$5"
- pattern :
kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+),
brokerPort=(.+)><>Value
name: kafka_server_$1_$2
type: GAUGE
labels:
clientId: "$3"
broker: "$4:$5"
- pattern :
kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate
name:
kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total
type: GAUGE
- pattern :
kafka.server<type=socket-server-metrics, clientSoftwareName=(.+),
clientSoftwareVersion=(.+), listener=(.+),
networkProcessor=(.+)><>connections
name:
kafka_server_socketservermetrics_connections
type: GAUGE
labels:
client_software_name: "$1"
client_software_version: "$2"
listener: "$3"
network_processor: "$4"
- pattern :
'kafka.server<type=socket-server-metrics, listener=(.+),
networkProcessor=(.+)><>(.+):'
name: kafka_server_socketservermetrics_$3
type: GAUGE
labels:
listener: "$1"
network_processor: "$2"
# Count and
Value
- pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.+),
(.+)=(.+)><>(Count|Value)
name: kafka_$1_$2_$3
labels:
"$4": "$5"
"$6": "$7"
- pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.+)><>(Count|Value)
name: kafka_$1_$2_$3
labels:
"$4": "$5"
- pattern:
kafka.(.*)<type=(.+), name=(.+)><>(Count|Value)
name: kafka_$1_$2_$3
# Percentile
- pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.*),
(.+)=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
"$6": "$7"
quantile: "0.$8"
- pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
"$4": "$5"
quantile: "0.$6"
- pattern:
kafka.(.*)<type=(.+), name=(.+)><>(\d+)thPercentile
name: kafka_$1_$2_$3
type: GAUGE
labels:
quantile: "0.$4"
|
Start Kafka with JMX Exporter Agent
It’s required to start JMX Exporter agent with Kafka. Set
JMX Exporter java agent in KAFKA_OPTS.
We can set KAFKA_OPTS different ways and below example
is setting up directly in windows command prompt and then start Kafka.
JAVA agent syntax
|
-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=<port>:<exporter-config-file-path>
|
Set KAFKA_OPTS in windows as follow
|
set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml
|
Kafka Broker1 Startup
Open command prompt and locate kafka-broker1 root
directory and use following commands.
|
cd
C:\kafka-workspace\kafka-broker1
set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml
bin\windows\kafka-server-start.bat
.\config\server.properties
|
Now Kafka broker1 started with JMX Exporter agent.
Repeat the same for other brokers in the cluster
Kafka Broker2 Startup
|
cd
C:\kafka-workspace\kafka-broker2
set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8282:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml
bin\windows\kafka-server-start.bat
.\config\server.properties
|
Kafka Broker3 Startup
|
cd
C:\kafka-workspace\kafka-broker3
set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8383:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml
bin\windows\kafka-server-start.bat
.\config\server.properties
|
Make sure all JMX Exporter are started successfully with
following URL’s. All java agents are running in same machine, its required to change
ports accordingly.
Install Prometheus
Go to Prometheus download page and download latest
version.
https://prometheus.io/download/
Direct Links as follows
https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.windows-amd64.zip
Extract in local Drive
Configure Prometheus scrape for Kafka
Locate Prometheus directory and update “prometheus.yml”
file with following Kafka scrape.
Find file from following location
https://github.com/LiferaySavvy/kafka-monitoring/blob/master/prometheus.yml
|
- job_name:
"kafka"
static_configs:
- targets: ['localhost:8181','localhost:8282','localhost:8383']
labels:
env: "kafka-dev"
|
Targets should be JMX Exporter java agent host:port.
Complete “prometheus.yml” file
|
# my global
config
global:
scrape_interval: 15s # Set the scrape interval to every
15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules
every 15 seconds. The default is every 1 minute.
# scrape_timeout is set to the global
default (10s).
# Alertmanager
configuration
alerting:
alertmanagers:
- static_configs:
- targets:
# - alertmanager:9093
# Load rules
once and periodically evaluate them according to the global
'evaluation_interval'.
rule_files:
# - "first_rules.yml"
# - "second_rules.yml"
# A scrape
configuration containing exactly one endpoint to scrape:
# Here it's
Prometheus itself.
scrape_configs:
# The job name is added as a label
`job=<job_name>` to any timeseries scraped from this config.
- job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
static_configs:
- targets: ['localhost:9090']
- job_name: "kafka"
static_configs:
- targets:
['localhost:8181','localhost:8282','localhost:8383']
labels:
env: "kafka-dev"
|
Start “Prometheus”
Open command prompt and locate to “Prometheus”
root directory. Use the following start command and pass web listen address
and config file as options.
|
prometheus.exe
--config.file prometheus.yml --web.listen-address ":9090"
--storage.tsdb.path "data"
|
Verify Kafka brokers in Prometheus
Access “Prometheus” web interface with
following URL and its running on 9090 port.
Targets Status
Go to Status menu and click on targets so we can see Kafka
cluster health for each Broker.
Kafka brokers health in the cluster.
Reference
https://github.com/confluentinc/jmx-monitoring-stacks
https://github.com/confluentinc/jmx-monitoring-stacks/tree/6.1.0-post/shared-assets/jmx-exporter
http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html
Author
Posts
