Monitor Kubernetes applications#

Your KaaS environment comes with two instances of Prometheus. One is preconfigured to monitor the Kubernetes cluster itself and the other to monitor your cloud-native applications. Rackspace does not recommend that you change anything in the Prometheus instance that monitors your Kubernetes cluster. As for the other Prometheus instance, you can use it to watch your applications, as well as set up Grafana dashboards to visualize Prometheus’ data.

If your organization already has a monitoring solution that you want to use to monitor your Kubernetes applications, contact your Rackspace representative to discuss implementation details.

If you decide to use the Prometheus instance deployed by KaaS to monitor your applications, you need to configure all the required configuration files that specify how Prometheus Operator communicates with service monitors and Prometheus exporters. Prometheus provides exporters for the most commonly used applications, and many third-party vendors develop their own Prometheus exporters. The main functionality of an exporter is to provide an endpoint for Prometheus to scrape metrics from application services.

At a high level, Prometheus sends an HTTPS request to the exporter. The exporter interrogates the application and provides metrics in the Prometheus format to its endpoint conforming to the Prometheus HTTP API.

Because each application is different, there is no solution that fits all use cases. Therefore, you need to define unique configuration files and select appropriate exporters for each application.

Typically, deploying an application with Prometheus monitoring involves the following steps:

  1. Create a deployment and service configuration file for your application.
  2. Expose an application-specific Prometheus exporter endpoint.
  3. Create a service monitor in Kubernetes that defines how Prometheus polls the exporter for data.
  4. Configure Grafana notifications and dashboards.

Example: Deploy a MySQL database with Prometheus monitoring#

As an example, deploy a MySQL database with Prometheus monitoring. You can use the MySQL Prometheus exporter called mysqld_exporter to expose MySQL metrics to Prometheus.

For this example, you need to create the following items:

  • A MySQL deployment
  • A MySQL service
  • A PersistentVolumeClaim for the MySQL database.
  • A MySQL exporter deployment
  • A MySQL exporter service
  • A MySQL exporter service monitor

The Prometheus Operator deployed by KaaS searches for the monitor-app label in the service monitor configuration files. If the label is different from that, KaaS can still deploy the application, but Prometheus will not be able to discover it. Therefore, create the service monitor in the monitoring namespace and use the monitor-app label as shown in the .yaml file below.

The following text is the MySQL configuration file (mysql.yaml):

apiVersion: apps/v1 # for versions before 1.9.0 use apps/v1beta2
kind: Deployment
metadata:
  name: mysql
  labels:
    name: mysql
spec:
  replicas: 1
  selector:
    matchLabels:
      name: mysql
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        name: mysql
        app: mysql
    spec:
      containers:
      - image: mysql:5.6
        name: mysql
        env:
        - name: MYSQL_ROOT_PASSWORD
          value: password
        ports:
        - containerPort: 3306
          name: mysql
        volumeMounts:
        - name: mysql-persistent-storage
          mountPath: /var/lib/mysql
      volumes:
      - name: mysql-persistent-storage
        persistentVolumeClaim:
          claimName: mysql-pv-claim
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: mysql-pv-claim
  labels:
    app: mysql
spec:
  accessModes:
   - ReadWriteOnce
  resources:
    requests:
      storage: 200Gi
---
apiVersion: v1
kind: Service
metadata:
  name: mysql
  labels:
    app: mysql
spec:
  type: ClusterIP
  ports:
    - port: 3306
  selector:
    app: mysql

The following text is the MySQL exporter configuration file (mysqld.yaml):

apiVersion: v1
kind: Service
metadata:
  name: mysqld-exporter
  labels:
    app: mysqld-exporter
spec:
  type: ClusterIP
  ports:
    - name: http-metrics
      port: 9104
selector:
  app: mysqld-exporter

---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: mysqld-exporter
  labels:
    app: mysqld-exporter
spec:
  selector:
    matchLabels:
      app: mysqld-exporter
  replicas: 1
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9104"
      labels:
        app: mysqld-exporter
    spec:
      containers:
      - name: mysqld-exporter
        image: prom/mysqld-exporter
        env:
        - name: DATA_SOURCE_NAME
          value: root:password@(mysql:3306)/
        ports:
        - containerPort: 9104
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: mysqld-exporter
  namespace: monitoring
  labels:
    monitor-app: mysqld-exporter
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      app: mysqld-exporter
  namespaceSelector:
    matchNames:
    - default
  endpoints:
  - port: http-metrics
    interval: 30s

To deploy the MySQL database with Prometheus monitoring, complete the following steps:

  1. Deploy the MySQL database by using the mysql.yaml from the example above:

    kubectl apply -f deployments/stable/mysql.yaml
    

    System response:

    deployment "mysql" created
    persistentvolumeclaim "mysql-pv-claim" created
    service "mysql" created
    
  2. Deploy the MySQL exporter by using the mysqld.yaml from the example above:

    kubectl apply -f deployments/stable/mysqld.yaml
    

    System response:

    service "mysqld-exporter" created
    deployment "mysqld-exporter" created
    servicemonitor "mysqld-exporter" created
    
  3. Verify that the deployment was created:

    kubectl get deployment,service,pvc -l app=mysql
    

    Example of system response:

    NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    mysql     1         1         1            1           20s
    
    NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
    svc/mysql   ClusterIP   10.3.33.6    <none>        3306/TCP   20s
    
    NAME                 STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    pvc/mysql-pv-claim   Bound     pvc-4d2fed9f-ce3d-11e8-a58f-fa163ea62d3e   200Gi      RWO            openstack      20s
    
  4. Deploy the MySQL exporter:

    kubectl apply -f mysqld.yaml
    

    Example of system response:

    service "mysqld-exporter" created
    deployment "mysqld-exporter" created
    servicemonitor "mysqld-exporter" created
    
  5. Verify that the deployment and service were created:

    kubectl get deployment,service -l app=mysqld-exporter
    

    Example of system response:

    NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    mysql     1         1         1            1           2m
    
    NAME              TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
    mysqld-exporter   ClusterIP   10.3.163.206   <none>        9104/TCP   2m
    
  6. Verify that the service monitor was created:

    kubectl get servicemonitor mysqld-exporter -n monitoring
    

    Example of system response:

    NAME              AGE
    mysqld-exporter   2m
    
  7. Log in to the KaaS Control Panel and then to the Prometheus UI.

  8. Go to Status > Targets.

    The mysql exporter must be present in the list of endpoints:

    ../../../_images/scr_prometheus_mysql_target.png
  9. Go to the Grafana dashboard.

  10. Import or create a dashboard to monitor your MySQL database.

  11. In the prometheus field, select prometheus-customer as a source. In the screenshot below the MySQL Overview Grafana dashboard is used as an example.

    ../../../_images/scr_prometheus_mysql_grafana.png
  12. Configure Grafana’s notifications for your communication channel as described in the Grafana documentation

Troubleshooting#

If you cannot see the mysql-exporter endpoint in the Prometheus UI, follow these troubleshooting guidelines:

  1. Check that deployments, services, service monitor, and the PVC have been successfully created:

    kubectl get deployment,service,pvc -l app=mysql
    

    Example of system response:

    NAME        TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
    svc/mysql   ClusterIP   10.3.33.6    <none>        3306/TCP   1h
    
    NAME                 STATUS    VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
    pvc/mysql-pv-claim   Bound     pvc-4d2fed9f-ce3d-11e8-a58f-fa163ea62d3e   200Gi      RWO            openstack      1h
    
  2. Verify that the MySQL exporter’s service and deployment were created:

    kubectl get deployment,service -l app=mysqld-exporter
    

    Example of system response:

    NAME                     DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
    deploy/mysqld-exporter   1         1         1            1           1h
    
    NAME                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
    svc/mysqld-exporter   ClusterIP   10.3.228.251   <none>        9104/TCP   1h
    
  3. Verify that the MySQL service monitor was created:

    kubectl get servicemonitor -l monitor-app=mysqld-exporter -n monitoring
    

    Example of system response:

    NAME              AGE
    mysqld-exporter   51m
    
  4. Check your .yaml files:

    • Verify that you use the monitor-app label for the service monitor and that the service monitor is deployed in the monitoring namespace.
  5. Check the mysql-exporter log files for any error:

    kubectl logs deployment/mysqld-exporter
    

    Example of system response:

    time="2018-10-12T16:39:22Z" level=info msg="Starting mysqld_exporter (version=0.11.0, branch=HEAD, revision=5d7179615695a61ecc3b5bf90a2a7c76a9592cdd)" source="mysqld_exporter.go:206"
    time="2018-10-12T16:39:22Z" level=info msg="Build context (go=go1.10.3, user=root@3d3ff666b0e4, date=20180629-15:00:35)" source="mysqld_exporter.go:207"
    time="2018-10-12T16:39:22Z" level=info msg="Enabled scrapers:" source="mysqld_exporter.go:218"
    time="2018-10-12T16:39:22Z" level=info msg=" --collect.global_variables" source="mysqld_exporter.go:222"
    time="2018-10-12T16:39:22Z" level=info msg=" --collect.slave_status" source="mysqld_exporter.go:222"
    time="2018-10-12T16:39:22Z" level=info msg=" --collect.info_schema.tables" source="mysqld_exporter.go:222"
    time="2018-10-12T16:39:22Z" level=info msg=" --collect.global_status" source="mysqld_exporter.go:222"
    time="2018-10-12T16:39:22Z" level=info msg="Listening on :9104" source="mysqld_exporter.go:232"