This directory contains chaos interleaved grafana dashboards along with the utilities needed to get started with monitoring chaos experiments and workflows.

• Grafana Dashboards

  Contains chaos interleaved grafana dashboards for various native k8s and application metrics.

• Utilities

  Contains utilities required to setup monitoring infrastructure on a kubernetes cluster.

• Install the litmus chaos operator and CRDs

  kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v1.13.6.yaml
  

• Install the litmus-admin serviceaccount for centralized/admin-mode of chaos execution

  kubectl apply -f https://litmuschaos.github.io/litmus/litmus-admin-rbac.yaml
  

• Install the chaos experiments in admin(litmus) namespace

  kubectl apply -f https://hub.litmuschaos.io/api/chaos/1.13.6?file=charts/generic/experiments.yaml -n litmus
  

kubectl create ns monitoring

Deploy prometheus components

kubectl -n monitoring apply -f utils/prometheus/prometheus-scrape-configuration/

Deploy metrics exporters

kubectl -n monitoring apply -f utils/metrics-exporters/node-exporter/
kubectl -n monitoring apply -f utils/metrics-exporters/kube-state-metrics/

Deploy chaos-exporter when the cluster is not connected to litmus 2.0 control plane via litmus agent (exporter is installed as a part of the agent bundle)

kubectl -n litmus apply -f utils/metrics-exporters/litmus-metrics/chaos-exporter/

Create the operator to instantiate all CRDs

kubectl -n monitoring apply -f utils/prometheus/prometheus-operator/

Deploy monitoring components

kubectl -n monitoring apply -f utils/metrics-exporters-with-service-monitors/node-exporter/
kubectl -n monitoring apply -f utils/metrics-exporters-with-service-monitors/kube-state-metrics/

Deploy chaos-exporter when the cluster is not connected to litmus 2.0 control plane via litmus agent (exporter is installed as a part of the agent bundle)

kubectl -n litmus apply -f utils/metrics-exporters-with-service-monitors/litmus-metrics/chaos-exporter/

Deploy prometheus instance and all the service monitors for targets

kubectl -n monitoring apply -f utils/prometheus/prometheus-configuration/

Note: To change the service type to NodePort, perform a kubectl edit svc prometheus-k8s -n monitoring and replace type: LoadBalancer to type: NodePort

optional: Alert manager

kubectl -n monitoring apply -f utils/alert-manager-with-service-monitor/

kubectl -n monitoring apply -f utils/grafana/

• You may access the grafana dashboard via the LoadBalancer (or NodePort) service IP or via a port-forward operation on localhost

  View the services running in the monitoring namespace

  kubectl get svc -n monitoring
  

  Now copy the EXTERNAL-IP of grafana and view it in the browser

  Default username/password credentials: admin/admin

  The dashboard and the datasource are imported automatically by Grafana during deployment, if something goes wrong, you can manually set it up as shown below.

• Add the prometheus datasource from monitoring namespace as DS_PROMETHEUS for Grafana via the Grafana Settings menu

  

• Import the grafana dashboards

  

• Import the grafana dashboard "Node and Pod Chaos Demo" provided here