How to Monitor and Alert Bold BI Using Grafana and Prometheus in Kubernetes

Configure Grafana and Prometheus:

Follow these steps to configure Grafana and Prometheus in the Kubernetes cluster.

1. Connect the Kubernetes cluster which you want to monitor.
2. Install helm using Chocolatey if you do not have it on your machine using the following command.

   Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.ServicePointManager]::SecurityProtocol -bor 3072; iex ((NewObjectSystem.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))
   

3. Add the Prometheus helm repo using the following command.

    helm repo add stable https://charts.helm.sh/stable
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   

4. Update the helm repo.

   helm repo update
   

5. Add the Prometheus community helm chart in Kubernetes.

   helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
   

6. After running the above command Prometheus and Grafana helm chart, move to the kube Prometheus stack.
7. Install stable Prometheus-community/kube-Prometheus-stack.

   helm install stable prometheus-community/kube-prometheus-stack
   

8. Once installed, you will see the following output.
   
   

   kubectl get pods
   

   

Edit Grafana and Prometheus services for external access:

1. By default, Prometheus and Grafana service is available within the cluster using ClusterIP. To access it outside, change it to either NodePort or Loadbalancer.

   kubectl edit svc stable-kube-prometheus-sta-prometheus
   

2. Here, change from ClusterIP to LoadBalancer/NodePort.
   

   

3. Edit the Grafana service as same as you did for Prometheus.

4. Run the below command to get services.

   Kubectl get svc
   

5. Load Balancer assigned an External IP to both Grafana and Prometheus services.
   

   

6. Access Grafana and Prometheus with the mentioned** IP address and port**.
   

   
   

7. Add this Prometheus as a data source in Grafana.
   

   

8. Click Add a data source.
   

   

9. Search Prometheus and give your Prometheus service external Ip with the port number in the URL.
   

   

10. Then click Save and Test.
    

    

11. Now, Data Source is configured in Grafana.

Dashboard Creation and Monitoring:

1. Grafana has a default dashboard for monitoring our cluster performance.

2. Use the following Grafana dashboard ID for overall cluster monitoring.
   

   

3. Use this dashboard ID – 9873 and select Prometheus as a data source for dashboard creation.
   

   

4. Refer to here for more dashboards.

Configure Alerting:

1. Add a notification channel for receiving alerting.
   

   

2. Enter your email and click save contact point.
   

   

3. Refer to here for configuring SMTP for Grafana for receiving mail.

4. Save and test the contact point

5. Refer to here for creating an alert in Grafana.

6. For example, now we are creating pods down the alert rule, go to alert, and click add a new alert.
   

   

7. Search a prom SQL query for finding pods in our namespace.
   

   

8. Set the threshold equal to 1 and add the other details.
   

   

9. Click save.
   

   

10. Find your alert rule under alert rules.
    

    

11. After creating the alert in Grafana, It will send the alert mail after it is triggered to mention Gmail as follows.