Liferay Portal Logs Monitoring with PLG

PGL is Grafana Labs stack which is similar to ELK stack. PLG is combination of Promtail, Loki and Grafana.

 

P ---> Promtail

L ---> Loki

G ---> Grafana

Promtail

Promtail is independent agent which runs in every server and send logs to Loki.

 

Loki

Grafana Loki is a set of components that can be composed into a fully featured logging stack. Its log aggregation service which collects the logs from Promtail.

 

Grafana

Grafana is open-source analytics and visualization solution. It provides charts, graphs and alerts.  It connects to multiple data sources and fetch the metrics data and monitor in Grafana visualization dashboards. Its also have ability to monitoring applications logs.

 

 

Following are steps to implements Logs Monitoring Solution for Liferay Portal.

 

• Install Promtail and Configure log scraps
• Install Loki and setup Loki configuration file
• Install Grafana
• Configure Loki Data source in Grafana
• Explore logs in Grafana

 

Prerequisite

Install Liferay Cluster

http://www.liferaysavvy.com/2021/07/liferay-portal-apache-webserver.html

 

Install Promtail and Configure log scraps

Go to Grafana Loki release page and download the latest Promtail zip file.

https://github.com/grafana/loki/releases/

 

Direct Link is below

https://github.com/grafana/loki/releases/download/v2.2.1/promtail-windows-amd64.exe.zip

 

Extract the downloaded zip file in local

Get default Promtail configuration file “promtail-local-config.yaml” from below URL and update the scraps.

 

https://github.com/LiferaySavvy/liferay-logs-monitoring-plg/blob/master/promtail-local-config.yaml

OR

 

https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-local-config.yaml

It’s required to create Jobs and log file location so that Promtail will pull the logs from log files and send to Loki.

 

Following is configuration yaml file.

server:   http_listen_port: 7060   grpc_listen_port: 0   positions:   filename: /tmp/positions.yaml   clients:   - url: http://localhost:3100/loki/api/v1/push   scrape_configs: - job_name: grafana   static_configs:   - targets:       - grafana     labels:       job: grafana       __path__: "C:/kafka-workspace/kafka-monitoring/grafana-8.0.6/data/log/grafana.log"   - job_name: liferay-node1   static_configs:   - targets:       - liferay-node1     labels:       job: liferay-node1       __path__: "C:/Liferay/Liferay74/liferay-ce-portal-7.4.1-ga2-node1/logs/*.log"       - job_name: liferay-node2   static_configs:   - targets:       - liferay-node2     labels:       job: liferay-node2       __path__: "C:/Liferay/Liferay74/liferay-ce-portal-7.4.1-ga2-node2/logs/*.log"

 

 

We are running two nodes of Liferay cluster, configured two jobs. It’s also required to specify Loki end point which is running on 3100 default port.

Promtail should be run on every server and configure the log location and file pattern in configuration yaml file.

 

Start Promtail service

Open command prompt and locate to Promtail root directory and use the below command.

promtail-windows-amd64.exe --config.file=promtail-local-config.yaml

 

Need to provide configuration file as input option. Make sure service started successfully. We can see startup logs on console for more details.

 

Install Loki and setup Loki configuration file

 

Go to Grafana Loki release page and download the latest Promtail zip file.

https://github.com/grafana/loki/releases/

 

Direct Link is below

https://github.com/grafana/loki/releases/download/v2.2.1/loki-windows-amd64.exe.zip

 

Extract the downloaded zip file in local

 

Get default Loki configuration file “loki-local-config.yaml” from below URL.

 

https://github.com/LiferaySavvy/liferay-logs-monitoring-plg/blob/master/loki-local-config.yaml

 

OR

 

https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml

Make sure there is no port conflicts and default port 3100 used by Loki. If you change port for Loki, we must update information in Promtail configuration file.

Loki configuration yaml file.

auth_enabled: false   server:   http_listen_port: 3100   grpc_listen_port: 9096   ingester:   wal:     enabled: true     dir: /tmp/wal   lifecycler:     address: 127.0.0.1     ring:       kvstore:         store: inmemory       replication_factor: 1     final_sleep: 0s   chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed   max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h   chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first   chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)   max_transfer_retries: 0     # Chunk transfers disabled   schema_config:   configs:     - from: 2020-10-24       store: boltdb-shipper       object_store: filesystem       schema: v11       index:         prefix: index_         period: 24h   storage_config:   boltdb_shipper:     active_index_directory: /tmp/loki/boltdb-shipper-active     cache_location: /tmp/loki/boltdb-shipper-cache     cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space     shared_store: filesystem   filesystem:     directory: /tmp/loki/chunks   compactor:   working_directory: /tmp/loki/boltdb-shipper-compactor   shared_store: filesystem   limits_config:   reject_old_samples: true   reject_old_samples_max_age: 168h   chunk_store_config:   max_look_back_period: 0s   table_manager:   retention_deletes_enabled: false   retention_period: 0s   ruler:   storage:     type: local     local:       directory: /tmp/loki/rules   rule_path: /tmp/loki/rules-temp   alertmanager_url: http://localhost:9093   ring:     kvstore:       store: inmemory   enable_api: true

 

 

Start Loki

Open command prompt and locate Loki exe file location in local drive and use following command.

loki-windows-amd64.exe --config.file=loki-local-config.yaml

 

 

Once service is up it can accessible on 3100 port with below URL.

http://localhost:3100/metrics

 

It confirms that Loki is installed successfully.

 

Install Grafana

Now it’s time to install Grafana and configure the Loki data source in Grafana.

Follow the below URL to install Grafana.

http://www.liferaysavvy.com/2021/07/grafana-installation-on-windows.html

 

Configure Loki Data source in Grafana

To explore logs in Grafana, need to configure the Loki Data source.

Access Grafana with following URL

 

http://localhost:3000/

 

Go to setting -->  Configuration --> Data sources

 

Click Add data source

Select Loki Data source

 

 

Provide Loki URL in configuration. Click on Save and Test button then Loki data source configured in Grafana.

http://localhost:3100

 

 

Explore logs in Grafana

Go to Grafana home page and click on Explore and Select Loki in explore list.

 

 

 

Click on Log Browser -->  Select Jobs --> Click on Show logs.

 

 

 

 

Now we can see Liferay logs and we can also stream the logs.

 

Logs Screen: 1

 

Logs Screen: 2

Author

Meera Prince

Liferay Contributor of the Year 2016
Liferay Top Contributor Award Winner 2015

Liferay Contributor of the Year 2014

Liferay Top Contributor Award Winner 2013

http://www.liferaysavvy.com