Wednesday, July 21, 2021

Kafka Monitoring with Prometheus

Prometheus is monitoring opensource tool. Previous Article we have enabled Zookeeper metrics and monitor in the Prometheus.


http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html

 

This article demonstrates the Kafka monitoring using Prometheus. Prometheus uses the JMX Exporter agent to get all JVM metrics from Kafka.


We need to run JMX Exported Java agent in each server where Kafka is running.


Software’s and Tools


 

Windows 10

Java 1.8 or higher

Zookeeper 3.7.0

Kafka 2.8

JMX Exporter Java Agent 0.15.0

prometheus-2.28.1

 

 







Prerequisite



Set up Zookeeper Cluster


http://www.liferaysavvy.com/2021/07/setup-zookeeper-cluster.html

 

Set up Kafka Cluster


http://www.liferaysavvy.com/2021/07/setup-kafka-cluster.html

 


  • Download JMX Exporter
  • Configure JMX Exporter for Kafka
  • Start Kafka with JMX Exporter Agent
  • Install Prometheus
  • Configure Prometheus scrape for Kafka
  • Verify Kafka brokers in Prometheus

 


Download and Start JMX Exporter



Download JMX Exporter jar file from following location.


https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

 

Direct link for latest JMX Exporter JAR


https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.15.0/jmx_prometheus_javaagent-0.15.0.jar

 

OR


https://github.com/LiferaySavvy/kafka-monitoring/raw/master/jmx_prometheus_javaagent-0.15.0.jar

 

 

Download and get the jar to local directory.




 

Configure JMX Exporter for Kafka



We need to configure the exporter configuration for Kafka such a way JMX service exports the required metrics from Kafka.


Get the following JMX Exporter configuration file from below location and place it in local drive.


https://github.com/confluentinc/jmx-monitoring-stacks/blob/6.1.0-post/shared-assets/jmx-exporter/kafka_broker.yml


OR


https://github.com/LiferaySavvy/kafka-monitoring/blob/master/kafka_broker.yml

 

We can use below one too.


 

lowercaseOutputName: true

rules:

# Special cases and very specific rules

- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), topic=(.+), partition=(.*)><>Value

  name: kafka_server_$1_$2

  type: GAUGE

  labels:

    clientId: "$3"

    topic: "$4"

    partition: "$5"

- pattern : kafka.server<type=(.+), name=(.+), clientId=(.+), brokerHost=(.+), brokerPort=(.+)><>Value

  name: kafka_server_$1_$2

  type: GAUGE

  labels:

    clientId: "$3"

    broker: "$4:$5"

 

- pattern : kafka.server<type=KafkaRequestHandlerPool, name=RequestHandlerAvgIdlePercent><>OneMinuteRate

  name: kafka_server_kafkarequesthandlerpool_requesthandleravgidlepercent_total

  type: GAUGE

 

- pattern : kafka.server<type=socket-server-metrics, clientSoftwareName=(.+), clientSoftwareVersion=(.+), listener=(.+), networkProcessor=(.+)><>connections

  name: kafka_server_socketservermetrics_connections

  type: GAUGE

  labels:

    client_software_name: "$1"

    client_software_version: "$2"

    listener: "$3"

    network_processor: "$4"

 

- pattern : 'kafka.server<type=socket-server-metrics, listener=(.+), networkProcessor=(.+)><>(.+):'

  name: kafka_server_socketservermetrics_$3

  type: GAUGE

  labels:

    listener: "$1"

    network_processor: "$2"

 

# Count and Value

- pattern: kafka.(.*)<type=(.+), name=(.+), (.+)=(.+), (.+)=(.+)><>(Count|Value)

  name: kafka_$1_$2_$3

  labels:

    "$4": "$5"

    "$6": "$7"

- pattern: kafka.(.*)<type=(.+), name=(.+), (.+)=(.+)><>(Count|Value)

  name: kafka_$1_$2_$3

  labels:

    "$4": "$5"

- pattern: kafka.(.*)<type=(.+), name=(.+)><>(Count|Value)

  name: kafka_$1_$2_$3

 

# Percentile

- pattern: kafka.(.*)<type=(.+), name=(.+), (.+)=(.*), (.+)=(.+)><>(\d+)thPercentile

  name: kafka_$1_$2_$3

  type: GAUGE

  labels:

    "$4": "$5"

    "$6": "$7"

    quantile: "0.$8"

- pattern: kafka.(.*)<type=(.+), name=(.+), (.+)=(.*)><>(\d+)thPercentile

  name: kafka_$1_$2_$3

  type: GAUGE

  labels:

    "$4": "$5"

    quantile: "0.$6"

- pattern: kafka.(.*)<type=(.+), name=(.+)><>(\d+)thPercentile

  name: kafka_$1_$2_$3

  type: GAUGE

  labels:

    quantile: "0.$4"

 

 



 

Start Kafka with JMX Exporter Agent



It’s required to start JMX Exporter agent with Kafka. Set JMX Exporter java agent in KAFKA_OPTS.


We can set KAFKA_OPTS different ways and below example is setting up directly in windows command prompt and then start Kafka.


JAVA agent syntax


 

-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=<port>:<exporter-config-file-path>




Set KAFKA_OPTS in windows as follow


 

set KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml

 

 

Kafka Broker1 Startup


Open command prompt and locate kafka-broker1 root directory and use following commands.


 

cd C:\kafka-workspace\kafka-broker1

 

set KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8181:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml

 

bin\windows\kafka-server-start.bat .\config\server.properties

 

 




Now Kafka broker1 started with JMX Exporter agent.

 

Repeat the same for other brokers in the cluster


Kafka Broker2 Startup



 

cd C:\kafka-workspace\kafka-broker2

 

set KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8282:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml

 

bin\windows\kafka-server-start.bat .\config\server.properties

 

 


Kafka Broker3 Startup



 

cd C:\kafka-workspace\kafka-broker3

 

set KAFKA_OPTS=-javaagent:C:/kafka-workspace/kafka-monitoring/jmx_prometheus_javaagent-0.15.0.jar=8383:C:/kafka-workspace/kafka-monitoring/kafka_broker.yml

 

bin\windows\kafka-server-start.bat .\config\server.properties

 

 


Make sure all JMX Exporter are started successfully with following URL’s. All java agents are running in same machine, its required to change ports accordingly.



 

http://localhost:8181/

http://localhost:8282/

http://localhost:8383/

 

 


 



 

Install Prometheus



Go to Prometheus download page and download latest version.


https://prometheus.io/download/



Direct Links as follows


https://github.com/prometheus/prometheus/releases/download/v2.28.1/prometheus-2.28.1.windows-amd64.zip

 

Extract in local Drive





Configure Prometheus scrape for Kafka



Locate Prometheus directory and update “prometheus.yml” file with following Kafka scrape.


Find file from following location


https://github.com/LiferaySavvy/kafka-monitoring/blob/master/prometheus.yml

 



- job_name: "kafka"

    static_configs:

      - targets: ['localhost:8181','localhost:8282','localhost:8383']

        labels:

          env: "kafka-dev"

 


 

Targets should be JMX Exporter java agent host:port.

 



Complete “prometheus.yml” file



# my global config

global:

  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.

  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

  # scrape_timeout is set to the global default (10s).

 

# Alertmanager configuration

alerting:

  alertmanagers:

  - static_configs:

    - targets:

      # - alertmanager:9093

 

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.

rule_files:

  # - "first_rules.yml"

  # - "second_rules.yml"

 

# A scrape configuration containing exactly one endpoint to scrape:

# Here it's Prometheus itself.

scrape_configs:

  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.

  - job_name: 'prometheus'

 

    # metrics_path defaults to '/metrics'

    # scheme defaults to 'http'.

 

    static_configs:

    - targets: ['localhost:9090']

   

  - job_name: "kafka"

    static_configs:

      - targets: ['localhost:8181','localhost:8282','localhost:8383']

        labels:

          env: "kafka-dev"

 

 

Start “Prometheus”



Open command prompt and locate to “Prometheus” root directory. Use the following start command and pass web listen address and config file as options.



 

prometheus.exe --config.file prometheus.yml --web.listen-address ":9090" --storage.tsdb.path "data"

 

 



Verify Kafka brokers in Prometheus



Access “Prometheus” web interface with following URL and its running on 9090 port.


 

http://localhost:9090/

 

 




 

Targets Status



Go to Status menu and click on targets so we can see Kafka cluster health for each Broker.




 

Kafka brokers health in the cluster.

 

 

http://localhost:9090/targets

 

 

 

 



 

Reference


https://github.com/confluentinc/jmx-monitoring-stacks


https://github.com/confluentinc/jmx-monitoring-stacks/tree/6.1.0-post/shared-assets/jmx-exporter


http://www.liferaysavvy.com/2021/07/enable-metrics-provider-in-zookeeper.html

 



Author

 

 

1 comment :

  1. Thanks for the insightful guide! Implementing Prometheus for Kafka monitoring seems seamless following your instructions. Excited to optimize performance using JMX Exporter on each Kafka server.

    ReplyDelete

Recent Posts

Recent Posts Widget

Popular Posts