Prometheus is monitoring opensource tool. Previous Article we have enabled Zookeeper metrics and monitor in the Prometheus.
http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html
This article demonstrates the Kafka monitoring using Prometheus.
Prometheus uses the JMX Exporter agent to get all JVM metrics from Kafka.
We need to run JMX Exported Java agent in each server where Kafka is running.
Software’s and Tools
|
Windows 10 Java 1.8 or
higher Zookeeper 3.7.0 Kafka 2.8 JMX Exporter Java
Agent 0.15.0 prometheus-2.28.1 |
Prerequisite
Set up Zookeeper Cluster
http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html
Set up Kafka Cluster
http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html
- Download JMX Exporter
- Configure JMX Exporter for Kafka
- Start Kafka with JMX Exporter Agent
- Install Prometheus
- Configure Prometheus scrape for Kafka
- Verify Kafka brokers in Prometheus
Download and Start JMX Exporter
Download JMX Exporter jar file from following
location.
https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/
Direct link for latest JMX Exporter JAR
OR
https://github.com/LiferaySavvy/kafka-monitoring/raw/master/jmx_prometheus_javaagent-0.15.0.jar
Download and get the jar to local directory.
Configure JMX Exporter for Kafka
We need to configure the exporter configuration for Kafka such a way JMX service exports the required metrics from Kafka.
Get the following JMX Exporter configuration file from
below location and place it in local drive.
OR
https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka_broker.yml
We can use below one too.
|
lowercaseOutputName:
true rules: # Special cases
and very specific rules - pattern :
kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+),
partition=(.*)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" topic: "$4" partition: "$5" - pattern :
kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+),
brokerPort=(.+)><>Value name: kafka_server_$1_$2 type: GAUGE labels: clientId: "$3" broker: "$4:$5" - pattern :
kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate name:
kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total type: GAUGE - pattern :
kafka.server<type=socket-server-metrics, clientSoftwareName=(.+),
clientSoftwareVersion=(.+), listener=(.+),
networkProcessor=(.+)><>connections name:
kafka_server_socketservermetrics_connections type: GAUGE labels: client_software_name: "$1" client_software_version: "$2" listener: "$3" network_processor: "$4" - pattern :
'kafka.server<type=socket-server-metrics, listener=(.+),
networkProcessor=(.+)><>(.+):' name: kafka_server_socketservermetrics_$3 type: GAUGE labels: listener: "$1" network_processor: "$2" # Count and
Value - pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.+),
(.+)=(.+)><>(Count|Value) name: kafka_$1_$2_$3 labels: "$4": "$5" "$6": "$7" - pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.+)><>(Count|Value) name: kafka_$1_$2_$3 labels: "$4": "$5" - pattern:
kafka.(.*)<type=(.+), name=(.+)><>(Count|Value) name: kafka_$1_$2_$3 # Percentile - pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.*),
(.+)=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" "$6": "$7" quantile: "0.$8" - pattern:
kafka.(.*)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: "$4": "$5" quantile: "0.$6" - pattern:
kafka.(.*)<type=(.+), name=(.+)><>(\d+)thPercentile name: kafka_$1_$2_$3 type: GAUGE labels: quantile: "0.$4" |
Start Kafka with JMX Exporter Agent
It’s required to start JMX Exporter agent with Kafka. Set
JMX Exporter java agent in KAFKA_OPTS.
We can set KAFKA_OPTS different ways and below example is setting up directly in windows command prompt and then start Kafka.
JAVA agent syntax
|
-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=<port>:<exporter-config-file-path> |
Set KAFKA_OPTS in windows as follow
|
set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml |
Kafka Broker1 Startup
Open command prompt and locate kafka-broker1 root directory and use following commands.
|
cd
C:\kafka-workspace\kafka-broker1 set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml bin\windows\kafka-server-start.bat
.\config\server.properties |
Now Kafka broker1 started with JMX Exporter agent.
Repeat the same for other brokers in the cluster
Kafka Broker2 Startup
|
cd
C:\kafka-workspace\kafka-broker2 set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8282:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml bin\windows\kafka-server-start.bat
.\config\server.properties |
Kafka Broker3 Startup
|
cd
C:\kafka-workspace\kafka-broker3 set
KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8383:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml bin\windows\kafka-server-start.bat
.\config\server.properties |
Make sure all JMX Exporter are started successfully with
following URL’s. All java agents are running in same machine, its required to change
ports accordingly.
|
|
Install Prometheus
Go to Prometheus download page and download latest version.
https://prometheus.io/download/
Direct Links as follows
Extract in local Drive
Configure Prometheus scrape for Kafka
Locate Prometheus directory and update “prometheus.yml” file with following Kafka scrape.
Find file from following location
https://github.com/LiferaySavvy/kafka-monitoring/blob/master/prometheus.yml
|
- job_name:
"kafka" static_configs: - targets: ['localhost:8181','localhost:8282','localhost:8383'] labels: env: "kafka-dev" |
Targets should be JMX Exporter java agent host:port.
Complete “prometheus.yml” file
|
# my global
config global: scrape_interval: 15s # Set the scrape interval to every
15 seconds. Default is every 1 minute. evaluation_interval: 15s # Evaluate rules
every 15 seconds. The default is every 1 minute. # scrape_timeout is set to the global
default (10s). # Alertmanager
configuration alerting: alertmanagers: - static_configs: - targets: # - alertmanager:9093 # Load rules
once and periodically evaluate them according to the global
'evaluation_interval'. rule_files: # - "first_rules.yml" # - "second_rules.yml" # A scrape
configuration containing exactly one endpoint to scrape: # Here it's
Prometheus itself. scrape_configs: # The job name is added as a label
`job=<job_name>` to any timeseries scraped from this config. - job_name: 'prometheus' # metrics_path defaults to '/metrics' # scheme defaults to 'http'. static_configs: - targets: ['localhost:9090'] - job_name: "kafka" static_configs: - targets:
['localhost:8181','localhost:8282','localhost:8383'] labels: env: "kafka-dev" |
Start “Prometheus”
Open command prompt and locate to “Prometheus”
root directory. Use the following start command and pass web listen address
and config file as options.
|
prometheus.exe
--config.file prometheus.yml --web.listen-address ":9090"
--storage.tsdb.path "data" |
Verify Kafka brokers in Prometheus
Access “Prometheus” web interface with following URL and its running on 9090 port.
|
|
Targets Status
Go to Status menu and click on targets so we can see Kafka
cluster health for each Broker.
Kafka brokers health in the cluster.
|
|
Reference
https://github.com/confluentinc/jmx-monitoring-stacks
https://github.com/confluentinc/jmx-monitoring-stacks/tree/6.1.0-post/shared-assets/jmx-exporter
http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html
Posts
Thanks for the insightful guide! Implementing Prometheus for Kafka monitoring seems seamless following your instructions. Excited to optimize performance using JMX Exporter on each Kafka server.
ReplyDelete