Sunday, July 18, 2021

Liferay Portal Logs Monitoring with ELKK

This article demonstrates the Liferay Portal Logs monitoring using ELKK stack.


 

Elastic Search

Logstash

Kibana

Kafka

 

 

Previous Article We already implemented Liferay Centralized logging system using Kafka.


Now we will use Logstash to pull the Liferay Logs from Kafka and push to Elastic Search.


Kibana will be used as Visualization and Monitoring tool to analyze the Liferay portal logs.


Following is Architecture diagram

 



Software’s and Tools


 

Windows 10

Java 1.8 or higher

Apache24

Liferay 7.4

Zookeeper-3.7.0

Kafka-2.8.0

Logstash-7.13.3

Kibana-7.13.3

Elasticsearch-7.13.3

 

 

 

Prerequisite


Implement Liferay Centralized logging system from below article which covers the Zookeeper, Kafka and Liferay Installation.

 

http://www.liferaysavvy.com/2021/07/centralized-logging-for-liferay-portal.html

 



It’s time to install and configure ELK stack.

 

  • Install Elastic Search Cluster
  • Install Logstash and configure input/output pipeline and Start
  • Validate Index creation in Elastic Search
  • Install Kibana and Start
  • Create Index Pattern in Kibana and Analyze Liferay portal logs.

 


Install Elastic Search Cluster


Follow the below article to install Elastic Search Cluster


http://www.liferaysavvy.com/2021/07/install-elastic-search-cluster.html



Install Logstash and configure input/output pipeline and Start


Follow the blow article to install log stash


http://www.liferaysavvy.com/2021/07/logstash-installation.html


Above Logstash install with dummy input/output and now it’s time to define actual Logstash pipeline. It’s very important step.


Locate to Logstash config location and create “logstash.conf” file.







 

We have all logs in Kafka now we will define Logstash pipeline input to Kafka and output to elastic search.


Use following configuration in “logstash.conf” file


 

input { 

    kafka {

        bootstrap_servers => "localhost:9092,localhost:9093,localhost:9094"

        topics => ["liferay-kafka-logs"]

    }

}

 

output { 

    elasticsearch {

        hosts => ["localhost:9200","localhost:9201","localhost:9202"]

        index => "liferay-index"

    }

}

 

 



 


Input should have Kafka bootstrap servers and Kafka topics. All logs are sending on “liferay-kafka-logs” topic and same topic was used in Liferay Log4J configuration for Kafka Appender.


Output is Elastic search cluster instances that we already installed. We also need to provide index name so that all logs will be tagged with given index.


Open command prompt and locate to Logstash root directory and use following command to start Logstash.


 

bin\logstash.bat -f config\logstash.conf

 

 






Once Logstash started successfully, all logs are collecting from Kafka and push to Elastic Search.



Install Kibana and Start


Follow the below Article to Install Kibana

 

http://www.liferaysavvy.com/2021/07/kibana-installation.html


 

Validate Index creation in Elastic Search


Make sure Logstash given index in configuration (logstash.conf) should be present in the Elastic search index list.


We can use any one of the Elastic cluster Node to confirm the elastic search health and index details. All should be green in the output.


Use below URL

 

http://localhost:9200/_cat/indices?v

 

 

 



Make sure all Stack Started and following is order. If anything, missed, star/restart in the order.

 

 

Start Zookeeper Cluster

Start Kafka Cluster

Start Liferay Portal Cluster

Start Elastic Cluster

Start Logstash
Start Kibana

 

 

Example screen shows all services started in local machine.






ELKK Important Information

 


 

Zookeeper Cluster

 

localhost:2181

localhost:2182

localhost:2183

 

 

Kafka Cluster

 

localhost:9092

localhost:9093

localhost:9094

 

 

Liferay Portal Cluster

 

http://localhost/

 

 

Elastic Cluster

 

http://localhost:9200/

http://localhost:9201/

http://localhost:9202/

 

 

Logstash

 

 

http://localhost:9600/

 

 

Kibana

 

http://localhost:5601/

 

 

Kafka Topic

 

liferay-kafka-logs

 

 

Elastic Search Index

 

liferay-index

 

 


Define Index Pattern in Kibana


To Monitor logs in Kibana we need to create index pattern in Kibana.


Go to Kibana home page and click on Left side toggle panel and Click on “Stack Management” and add Kibana Index pattern.


 



 

 

Click on Kibana à Index Pattern à Create Index Pattern

 



 

Provide the index name which we were provided in the Logstash file. You can provide exact index name or use wildcard pattern (liferay-*).

 



 

 

Select time field and create index pattern




 

Go to Analytics à Discovery. We can see index in the list.

 



 

 

 


Select newly created index and all the logs’ data visible in the page. Change the time frame to play with logs data.

 





Author

0 comments :

Post a Comment

Recent Posts

Recent Posts Widget

Popular Posts