Saturday, July 24, 2021

Liferay Portal Logs Monitoring with PLG

PGL is Grafana Labs stack which is similar to ELK stack. PLG is combination of Promtail, Loki and Grafana.

 


P ---> Promtail

L ---> Loki

G ---> Grafana



Promtail


Promtail is independent agent which runs in every server and send logs to Loki.

 


Loki


Grafana Loki is a set of components that can be composed into a fully featured logging stack. Its log aggregation service which collects the logs from Promtail.

 


Grafana


Grafana is open-source analytics and visualization solution. It provides charts, graphs and alerts.  It connects to multiple data sources and fetch the metrics data and monitor in Grafana visualization dashboards. Its also have ability to monitoring applications logs.

 




 

Following are steps to implements Logs Monitoring Solution for Liferay Portal.

 


  • Install Promtail and Configure log scraps
  • Install Loki and setup Loki configuration file
  • Install Grafana
  • Configure Loki Data source in Grafana
  • Explore logs in Grafana

 



Prerequisite



Install Liferay Cluster


http://www.liferaysavvy.com/2021/07/liferay-portal-apache-webserver.html

 



Install Promtail and Configure log scraps



Go to Grafana Loki release page and download the latest Promtail zip file.


https://github.com/grafana/loki/releases/

 


Direct Link is below


https://github.com/grafana/loki/releases/download/v2.2.1/promtail-windows-amd64.exe.zip

 


Extract the downloaded zip file in local




Get default Promtail configuration file “promtail-local-config.yaml” from below URL and update the scraps.

 


https://github.com/LiferaySavvy/liferay-logs-monitoring-plg/blob/master/promtail-local-config.yaml


OR

 

https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/promtail/promtail-local-config.yaml


It’s required to create Jobs and log file location so that Promtail will pull the logs from log files and send to Loki.

 


Following is configuration yaml file.


 


server:

  http_listen_port: 7060

  grpc_listen_port: 0

 

positions:

  filename: /tmp/positions.yaml

 

clients:

  - url: http://localhost:3100/loki/api/v1/push

 

scrape_configs:

- job_name: grafana

  static_configs:

  - targets:

      - grafana

    labels:

      job: grafana

      __path__: "C:/kafka-workspace/kafka-monitoring/grafana-8.0.6/data/log/grafana.log"

 

- job_name: liferay-node1

  static_configs:

  - targets:

      - liferay-node1

    labels:

      job: liferay-node1

      __path__: "C:/Liferay/Liferay74/liferay-ce-portal-7.4.1-ga2-node1/logs/*.log"

     

- job_name: liferay-node2

  static_configs:

  - targets:

      - liferay-node2

    labels:

      job: liferay-node2

      __path__: "C:/Liferay/Liferay74/liferay-ce-portal-7.4.1-ga2-node2/logs/*.log"

 


 



 


We are running two nodes of Liferay cluster, configured two jobs. It’s also required to specify Loki end point which is running on 3100 default port.


Promtail should be run on every server and configure the log location and file pattern in configuration yaml file.

 


Start Promtail service


Open command prompt and locate to Promtail root directory and use the below command.


 

promtail-windows-amd64.exe --config.file=promtail-local-config.yaml


 


Need to provide configuration file as input option. Make sure service started successfully. We can see startup logs on console for more details.

 







Install Loki and setup Loki configuration file

 


Go to Grafana Loki release page and download the latest Promtail zip file.


https://github.com/grafana/loki/releases/

 


Direct Link is below


https://github.com/grafana/loki/releases/download/v2.2.1/loki-windows-amd64.exe.zip

 


Extract the downloaded zip file in local




 

Get default Loki configuration file “loki-local-config.yaml” from below URL.

 


https://github.com/LiferaySavvy/liferay-logs-monitoring-plg/blob/master/loki-local-config.yaml

 


OR

 


https://raw.githubusercontent.com/grafana/loki/v2.2.1/cmd/loki/loki-local-config.yaml


Make sure there is no port conflicts and default port 3100 used by Loki. If you change port for Loki, we must update information in Promtail configuration file.


Loki configuration yaml file.




auth_enabled: false

 

server:

  http_listen_port: 3100

  grpc_listen_port: 9096

 

ingester:

  wal:

    enabled: true

    dir: /tmp/wal

  lifecycler:

    address: 127.0.0.1

    ring:

      kvstore:

        store: inmemory

      replication_factor: 1

    final_sleep: 0s

  chunk_idle_period: 1h       # Any chunk not receiving new logs in this time will be flushed

  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h

  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first

  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)

  max_transfer_retries: 0     # Chunk transfers disabled

 

schema_config:

  configs:

    - from: 2020-10-24

      store: boltdb-shipper

      object_store: filesystem

      schema: v11

      index:

        prefix: index_

        period: 24h

 

storage_config:

  boltdb_shipper:

    active_index_directory: /tmp/loki/boltdb-shipper-active

    cache_location: /tmp/loki/boltdb-shipper-cache

    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space

    shared_store: filesystem

  filesystem:

    directory: /tmp/loki/chunks

 

compactor:

  working_directory: /tmp/loki/boltdb-shipper-compactor

  shared_store: filesystem

 

limits_config:

  reject_old_samples: true

  reject_old_samples_max_age: 168h

 

chunk_store_config:

  max_look_back_period: 0s

 

table_manager:

  retention_deletes_enabled: false

  retention_period: 0s

 

ruler:

  storage:

    type: local

    local:

      directory: /tmp/loki/rules

  rule_path: /tmp/loki/rules-temp

  alertmanager_url: http://localhost:9093

  ring:

    kvstore:

      store: inmemory

  enable_api: true

 


 



 

Start Loki


Open command prompt and locate Loki exe file location in local drive and use following command.


 

loki-windows-amd64.exe --config.file=loki-local-config.yaml


 



 


Once service is up it can accessible on 3100 port with below URL.


 

http://localhost:3100/metrics


 





It confirms that Loki is installed successfully.

 



Install Grafana



Now it’s time to install Grafana and configure the Loki data source in Grafana.


Follow the below URL to install Grafana.


http://www.liferaysavvy.com/2021/07/grafana-installation-on-windows.html

 



Configure Loki Data source in Grafana



To explore logs in Grafana, need to configure the Loki Data source.


Access Grafana with following URL

 

 

http://localhost:3000/


 

Go to setting -->  Configuration --> Data sources




 

Click Add data source




Select Loki Data source


 



 

Provide Loki URL in configuration. Click on Save and Test button then Loki data source configured in Grafana.


 

http://localhost:3100


 



 

Explore logs in Grafana



Go to Grafana home page and click on Explore and Select Loki in explore list.

 




 

 

Click on Log Browser -->  Select Jobs --> Click on Show logs.

 

 



 

 

Now we can see Liferay logs and we can also stream the logs.

 

Logs Screen: 1




 

Logs Screen: 2






Author

1 comment :

Recent Posts

Recent Posts Widget

Popular Posts