Kubernetes Monitoring with Prometheus

By | March 11, 2024
Reading Time: 16 minutes

In this article I am going show how to install Prometheus in a Kubernetes cluster using a Kubernetes operator. I will use the Operator Lifecycle Manager to install the operator, all with the ulterior motive to have infrastructure as code and in the extension be able to automate installation and configuration to an as large extent as possible. Finally, I am going to create a Prometheus instance running in the cluster and configure Prometheus to monitor the cluster itself.
The example in this article will use K3D to create a Kubernetes cluster running in Docker containers.
The article assume some familiarity with Kubernetes and using kubectl.

Monitoring Kubernetes
Monitoring Kubernetes.

Prerequisites

In order to follow the example in this article, the following need to be installed:

I assume that K3D can be substituted with, for example, K3S (K3D is actually K3S running in Docker containers), kind or Tanzu Community Edition which are all available for free.

Operators – a brief presentation

If you are already familiar with the concept of Kubernetes operators, feel free to skip this section as it is merely a very brief introduction on the subject.
A Kubernetes operator is an extension of Kubernetes that allows you to use the Kubernetes API to deploy and manage an application that is to run in a Kubernetes cluster. In the case of deployment, an operator can be seen as a package that deploys the different parts needed in a Kubernetes cluster in order for the application to be able to run in the cluster.
One example of an application for which an operator exists is Elasticsearch. The figure below lists the custom resource definitions that the Elasticsearch operator adds.

Custom resource definitions in the Elasticsearch operator.

Custom resource definitions in the Elasticsearch operator.

The Kibana custom resource, for example, makes it possible to deploy a Kibana instance by applying one single resource. For each Kibana instance in a cluster there will be one Kibana resource.

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-sample
spec:
  version: 8.12.0
  count: 1
  elasticsearchRef:
    name: elasticsearch-sample
  podTemplate:
    metadata:
      labels:
        foo: bar
    spec:
      containers:
        - name: kibana
          resources:
            requests:
              memory: 1Gi
              cpu: 0.5
            limits:
              memory: 2Gi
              cpu: 2

Example Kibana definition deploying one instance of Kibana version 8.12.0.

In a similar fashion, the Elasticsearch Cluster custom resource, which in yaml-files has the kind “Elasticsearch”, allows for creation of an entire Elasticsearch cluster in a Kubernetes cluster.

apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-sample
spec:
  version: 8.12.0
  nodeSets:
    - name: default
      config:
        node.roles:
          - master
          - data
        node.attr.attr_name: attr_value
        node.store.allow_mmap: false
      podTemplate:
        metadata:
          labels:
            foo: bar
        spec:
          containers:
            - name: elasticsearch
              resources:
                requests:
                  memory: 4Gi
                  cpu: 1
                limits:
                  memory: 4Gi
                  cpu: 2
      count: 3

Example Elasticsearch cluster definition deploying version 8.12.0 of Elasticsearch in a cluster with three nodes.

Upgrading an Elasticsearch cluster is as simple as changing the desired version number in an Elasticsearch Cluster resource and applying the resource. The usual caveats concerning the current and desired versions still apply. For example going from version 1.2.3 to version 1.2.4 is considered a small step which most likely will pose no problems while going to version 3.2.2 from version 1.2.3 would be considered a large step which may be more problematic regardless of whether using an operator or not.
Much more can be said about Kubernetes operators and the interested reader is encouraged to research the topic further.

Preparations

Before being able to install Prometheus in a Kubernetes cluster, a few things need to be prepared.

A cluster

I have, without any particular reasons, chosen to create a Kubernetes cluster with one master node and three agents. You are free to change this as you wish. The cluster is created using the following command.

k3d cluster create -a 3 -s 1

The output should look like the following, indicating the successful creation of the cluster:

INFO[0000] Prep: Network                                
INFO[0000] Created network 'k3d-k3s-default'            
INFO[0000] Created image volume k3d-k3s-default-images  
INFO[0000] Starting new tools node...                   
INFO[0001] Creating node 'k3d-k3s-default-server-0'     
INFO[0002] Pulling image 'docker.io/rancher/k3s:v1.27.4-k3s1' 
INFO[0004] Pulling image 'ghcr.io/k3d-io/k3d-tools:5.6.0' 
INFO[0007] Starting Node 'k3d-k3s-default-tools'        
INFO[0012] Creating node 'k3d-k3s-default-agent-0'      
INFO[0012] Creating node 'k3d-k3s-default-agent-1'      
INFO[0012] Creating node 'k3d-k3s-default-agent-2'      
INFO[0012] Creating LoadBalancer 'k3d-k3s-default-serverlb' 
INFO[0013] Pulling image 'ghcr.io/k3d-io/k3d-proxy:5.6.0' 
INFO[0019] Using the k3d-tools node to gather environment information 
INFO[0019] Starting new tools node...                   
INFO[0019] Starting Node 'k3d-k3s-default-tools'        
INFO[0021] Starting cluster 'k3s-default'               
INFO[0021] Starting servers...                          
INFO[0021] Starting Node 'k3d-k3s-default-server-0'     
INFO[0025] Starting agents...                           
INFO[0025] Starting Node 'k3d-k3s-default-agent-2'      
INFO[0025] Starting Node 'k3d-k3s-default-agent-1'      
INFO[0025] Starting Node 'k3d-k3s-default-agent-0'      
INFO[0030] Starting helpers...                          
INFO[0030] Starting Node 'k3d-k3s-default-serverlb'     
INFO[0036] Injecting records for hostAliases (incl. host.k3d.internal) and for 6 network members into CoreDNS configmap... 
INFO[0039] Cluster 'k3s-default' created successfully!  
INFO[0039] You can now use it like this:                
kubectl cluster-info

Operator Lifecycle Manager

Operator Lifecycle Manager, abbreviated OLM, is a toolkit for managing Kubernetes operators. In this article OLM will only be used to install operators.

OLM is installed using the following procedure:

Find the latest version of OLM at this URL.
At the time of writing, this is v0.27.0.

Copy the script snippet for the version of OLM that is to be installed from the above web-page.
For version 0.27.0 the installation script looks like this:

curl -L https://github.com/operator-framework/operator-lifecycle-manager/releases/download/v0.27.0/install.sh -o install.sh
chmod +x install.sh
./install.sh v0.27.0

Execute the installation script snippet.
The output should end with the following lines, indicating successful installation of OLM:

…
Waiting for deployment "olm-operator" rollout to finish: 0 of 1 updated replicas are available...
deployment "olm-operator" successfully rolled out
deployment "catalog-operator" successfully rolled out
Package server phase: Installing
Package server phase: Succeeded
deployment "packageserver" successfully rolled out

Verify the OLM deployment using the following command:

kubectl -n olm get deployments

The output should include the olm-operator and indicate that it is ready, up-to-date and available as can be seen on line three in the following output:

NAME               READY   UP-TO-DATE   AVAILABLE   AGE
catalog-operator   1/1     1            1           98s
olm-operator       1/1     1            1           98s

Prometheus

With the Kubernetes cluster created and OLM installed, time has now come to install the Prometheus operator and create a Prometheus instance that can collect metrics in the cluster. However, before getting to the practical details, lets look at how collecting metrics with Prometheus in a Kubernetes cluster is configured.

Prometheus custom resource definitions

The Prometheus operator used in this article contains the following custom resource definitions, CRDs, that are used to specify how to gather metrics from one or more items:

  • PodMonitor
    Gather metrics from one or more pods.
  • Probe
    Gather metrics from ingresses and static targets.
  • ScrapeConfig
    Gather metrics from items outside of the Kubernetes cluster.
  • ServiceMonitor
    Gathers metrics from one or more Kubernetes Services.

This article will only use the ServiceMonitor CRD. In addition when using the PodMonitor, Probe and ServiceMonitor CRDs then access control need to be configured in order to allow Prometheus to collect metrics. In this article RBAC access control will be used – see the next section for further information.

Install the Prometheus operator

There are several different ways that the Prometheus operator can be installed. It may come as no surprise that I have chosen to use the OLM to install the operator. In order to be able to check in the installation YAML-file and, if desired, configure as to receive automatic updates of the operator, the OLM Subscription custom resource is used.
Some additional exercises will be undertaken in order to become acquainted with the OLM.

List all available operators using the following command in a terminal window:

kubectl get packagemanifests

Find the operators with “prometheus” in their names.

kubectl get packagemanifests | grep prometheus

At the time of writing this article, there are three matching operators:

prometheus-exporter-operator               Community Operators   9d
ack-prometheusservice-controller           Community Operators   9d
prometheus                                 Community Operators   9d

The last one with the name “prometheus” is the operator that will be installed.

Obtain information about the Prometheus operator.

kubectl describe packagemanifests prometheus

There will be a lot of information about the operator which will not be reproduced here.

Create a file named “prometheus-operator.yaml” with the following contents:

# Installs the Prometheus operator in the operators namespace using the OLM.
apiVersion: operators.coreos.com/v1alpha1
kind: Subscription
metadata:
  name: prometheus-operator
  namespace: operators
spec:
  channel: beta
  name: prometheus
  source: operatorhubio-catalog
  sourceNamespace: olm

At the time of writing, the Prometheus operator is not available in the “stable” channel and so the “beta” channel has to be used.

Install the Prometheus operator.

kubectl create -f prometheus-operator.yaml

Wait until the Prometheus operator has been successfully installed.

kubectl get deployments -n operators -w

The result should look like this, indicating that the Prometheus operator is ready, up-to-date and available:

NAME                  READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator   1/1     1            1           8s

The Prometheus operator has now been successfully installed.

Create a Prometheus instance

With he Prometheus operator installed, a CRD installed by the operator can now be used to create a Prometheus instance.

Create a file named “prometheus-instance.yaml” with the following contents:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
  namespace: default
spec:
  serviceAccountName: prometheus
  serviceMonitorNamespaceSelector: {}
  serviceMonitorSelector: {}
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: true

The above will create a Prometheus instance that requests 400MiB of memory, has the administration API enabled and use the service account “prometheus”. Documentation of the Prometheus CRD is available here.
Note the “serviceMonitorNamespaceSelector” and the “serviceMonitorSelector” keys. These make it possible to limit discovery of ServiceMonitor, PodMonitor and Probe objects to those found in certain namespace(s), as specified by “serviceMonitorNamespaceSelector”, and meeting certain conditions, specified by “serviceMonitorSelector”. If these keys have the empty dictionary value then no filtering being performed on ServiceMonitor, PodMonitor and Probe objects and all ServiceMonitor, PodMonitor and Probe objects in all namespaces will be discovered.

Create a file named “prometheus-serviceaccount.yaml” with the following contents:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus

This is a ServiceAccount that will be used by the Prometheus instance for example when accessing services in the cluster to collect metrics data.

Create a file named “prometheus-clusterrole.yaml” with the following contents:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]

The ClusterRole gives permission to read miscellaneous sources of metrics as well as ConfigMaps.

Create a file named “prometheus-clusterrolebinding.yaml” containing the following:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default

The ClusterRoleBinding associates the ServiceAccount with the ClusterRole and thus allows

Apply all the resource files.

kubectl create -f prometheus-serviceaccount.yaml
kubectl create -f prometheus-clusterrole.yaml
kubectl create -f prometheus-clusterrolebinding.yaml
kubectl create -f prometheus-instance.yaml

Verify that the Prometheus instance has been successfully created.

kubectl get prometheus -w

The instance should be ready, reconciled and available.

NAME         VERSION   DESIRED   READY   RECONCILED   AVAILABLE   AGE
prometheus                       1       True         True        5m2s

Find the Service exposing the Prometheus web UI.

kubectl get service

In my case, the service is named “prometheus-operated”:

NAME                  TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes            ClusterIP   10.43.0.1    <none>        443/TCP    13d
prometheus-operated   ClusterIP   None         <none>        9090/TCP   14m

Notice the service´s name and it´s port, “prometheus-operated” and 9090.

Make the port available on localhost:

kubectl port-forward service/prometheus-operated 9090:9090

View the Prometheus web UI in a browser using the following URL: http://localhost:9090

Prometheus web UI.
Prometheus web UI.

A Prometheus instance has been successfully created in the default namespace and it´s web UI can be viewed in a local browser.

Sources of metrics

In this article only two readily available sources for metrics will be examined; Metrics Server and Kubelet. Both are sources of cluster metrics albeit for different purposes as we will soon learn.

Metrics Server

Metrics Server, as introduced on its web-page, is a source of container resource metrics for Kubernetes built-in autoscaling pipelines that collects metrics from Kubelets. Metrics Server is not recommended as an accurate source of resource usage metrics. Nevertheless this article will have a brief look at Metrics Server and show how to collect metrics from it using Prometheus.

Getting to know Metrics Server

Before attempting to access the metrics made available by Metrics Server, let’s become a little more acquainted first.

Determine whether Metrics Server is present in the cluster:

kubectl get deployments -n kube-system

In my cluster there are the following deployments in the kube-system namespace:

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
local-path-provisioner   1/1     1            1           8m12s
coredns                  1/1     1            1           8m12s
metrics-server           1/1     1            1           8m11s
traefik                  1/1     1            1           7m21s

Is Metrics Server exposing a service?

kubectl get service -n kube-system

The services in the example-cluster are:

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP                                   PORT(S)                      AGE
kube-dns         ClusterIP      10.43.0.10     <none>                                        53/UDP,53/TCP,9153/TCP       11m
metrics-server   ClusterIP      10.43.72.155   <none>                                        443/TCP                      11m
traefik          LoadBalancer   10.43.82.225   172.18.0.3,172.18.0.4,172.18.0.5,172.18.0.6   80:32710/TCP,443:31047/TCP   10m

Yes, indeed – Metrics Server is exposing a service!


Let’s examine the service exposed by Metrics Server:

kubectl get service metrics-server -n kube-system -o yaml

The metrics-server service resource, with some data having been omitted for brevity, should look something like this:

apiVersion: v1
kind: Service
metadata:
  annotations:
    objectset.rio.cattle.io/owner-gvk: k3s.cattle.io/v1, Kind=Addon
    objectset.rio.cattle.io/owner-name: metrics-server-service
    objectset.rio.cattle.io/owner-namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Metrics-server
  name: metrics-server
  namespace: kube-system
  resourceVersion: "303"
spec:
  clusterIP: 10.43.72.155
  clusterIPs:
  - 10.43.72.155
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: https
  selector:
    k8s-app: metrics-server
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Note that:

  • At spec.ports.name, the name of the port, port 443, which is exposed is “https”.
    This will come in handy later when defining a ServiceMonitor to scrape metrics from the metrics-server.
  • At spec.selector, the key is “k8s-app” and the value is “metrics-server”.
    This is the name and value of the label used to select the pod(s) which the service is to expose.

Manually retrieving metrics from Metrics Server

Before letting Prometheus have an attempt at scraping metrics from Metrics Server, let’s see if we are able to request metrics from Metrics Server manually.

Before being able to send any requests to Metrics Server a service account token to enclose with the requests must be retrieved. The steps below shows how to retrieve such a token from the pod in which the Prometheus instance created in this example is running.

Find the name of the pod in which Prometheus is running:

get pod | grep prometheus

Outputs:

prometheus-prometheus-0   2/2     Running   8 (68m ago)   3d22h

Retrieve the service account token from the pod found in the previous step.
The token will be stored locally in a file named “securitytoken”.

kubectl exec -it prometheus-prometheus-0 -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > securitytoken

Expose the Metrics Server Service on localhost:

kubectl port-forward service/metrics-server -n kube-system 8123:443

Request metrics from Metrics Server.
Note that this will have to be executed in a new terminal window since the port-forward in the previous step needs to be running in order for the request not to fail.

curl -k --header "Authorization: Bearer $(cat securitytoken)" https://localhost:8123/metrics

If the request was successful a lot of data will be listed in the console window, the last of it looking something like this:

# TYPE metrics_server_manager_tick_duration_seconds histogram
metrics_server_manager_tick_duration_seconds_bucket{le="0.005"} 0
metrics_server_manager_tick_duration_seconds_bucket{le="0.01"} 0
metrics_server_manager_tick_duration_seconds_bucket{le="0.025"} 4
metrics_server_manager_tick_duration_seconds_bucket{le="0.05"} 208
metrics_server_manager_tick_duration_seconds_bucket{le="0.1"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="0.25"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="0.5"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="1"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="2.5"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="5"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="10"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="12.5"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="15"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="22.5"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="30"} 223
metrics_server_manager_tick_duration_seconds_bucket{le="+Inf"} 223
metrics_server_manager_tick_duration_seconds_sum 9.149395879999995
metrics_server_manager_tick_duration_seconds_count 223
# HELP metrics_server_storage_points [ALPHA] Number of metrics points stored.
# TYPE metrics_server_storage_points gauge
metrics_server_storage_points{type="container"} 20
metrics_server_storage_points{type="node"} 4

Metrics from Metrics Server has been retrieved successfully. Feel free to stop the port-forward exposing the Metrics Server Service.

Kubelet

Kubelet is the primary node agent that runs on each node in a Kubernetes cluster. Kubelet is also the recommended source for monitoring metrics on the Metrics Server web page.

Getting to know kubelet

For kubelet there is no deployment resource but the Prometheus operator will create a Service resource that makes it possible to scrape metrics from Kubelet using Prometheus.

List all services in all namespaces in the cluster:

kubectl get service -A

Given the cluster created for this example, the output look like this:

NAMESPACE     NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP                                   PORT(S)                        AGE
default       kubernetes              ClusterIP      10.43.0.1       <none>                                        443/TCP                        13h
kube-system   kube-dns                ClusterIP      10.43.0.10      <none>                                        53/UDP,53/TCP,9153/TCP         13h
kube-system   metrics-server          ClusterIP      10.43.72.155    <none>                                        443/TCP                        13h
kube-system   traefik                 LoadBalancer   10.43.82.225    172.18.0.3,172.18.0.4,172.18.0.5,172.18.0.6   80:32710/TCP,443:31047/TCP     13h
olm           packageserver-service   ClusterIP      10.43.112.118   <none>                                        5443/TCP                       13h
olm           operatorhubio-catalog   ClusterIP      10.43.233.197   <none>                                        50051/TCP                      13h
kube-system   kubelet                 ClusterIP      None            <none>                                        10250/TCP,10255/TCP,4194/TCP   12h
operators     prometheus-operator     ClusterIP      None            <none>                                        8080/TCP                       12h

Examine the kubelet service closer:

kubectl get service kubelet -n kube-system -o yaml

The kubelet service resource should look something like in the listing below. Some data has been omitted for brevity.

apiVersion: v1
kind: Service
metadata:
  labels:
    app.kubernetes.io/managed-by: prometheus-operator
    app.kubernetes.io/name: kubelet
    k8s-app: kubelet
  name: kubelet
  namespace: kube-system
  resourceVersion: "4127"
spec:
  clusterIP: None
  clusterIPs:
  - None
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  - IPv6
  ipFamilyPolicy: RequireDualStack
  ports:
  - name: https-metrics
    port: 10250
    protocol: TCP
    targetPort: 10250
  - name: http-metrics
    port: 10255
    protocol: TCP
    targetPort: 10255
  - name: cadvisor
    port: 4194
    protocol: TCP
    targetPort: 4194
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Note that:

  • Under spec.ports there are three ports.
    The names of the ports are “https-metrics”, “http-metrics” and “cAdvisor”.
  • The port “https-metrics” maps to the port 10250.
    This will be of use later when we define a ServiceMonitor to scrape metrics from Kubelet.
  • There is an additional port named “cadvisor” which exposes a port related to cAdvisor but metrics from cAdvisor will be expose on the “https-metrics” port so this port is not relevant for this exercise.
  • There is no spec.selector in the kubelet Service.
    If you should try to perform a port-forward to the kubelet service it will fail with the following error:
error: cannot attach to *v1.Service: invalid service 'kubelet': Service is defined without a selector

The reason for this is that there is no pod backing the kubelet service. Port-forward exposes a port and forwards request on that port to the pod backing a service and this will not be possible if there is no backing pod. Instead there is an EndpointSlice that maps the kubelet service to the addresses of each of the node in the Kubernetes cluster, which can be seen if examining the kubelet EndpointSlice that can be examined using the following command:

kubectl get EndpointSlice -l k8s-app=kubelet -n kube-system -o yaml

Manually retrieving metrics from kubelet

As with Metrics Server earlier, lets attempt to retrieve metrics from kubelet manually. Given that port-forward cannot be used to make the kubelet service accessible a slightly more complex approach will be taken to accomplish the goal. It is assumed that a service account token has already been retrieved as described when manually retrieving metrics from Metrics Server.

List all the names of the nodes in the cluster:

kubectl get nodes -o jsonpath='{range .items[*]}{@.metadata.name}{"\n"}{end}'

In the example cluster there are four nodes:

k3d-k3s-default-server-0
k3d-k3s-default-agent-2
k3d-k3s-default-agent-0
k3d-k3s-default-agent-1

Start a pod with Ubuntu and enter the bash shell in the pod:

kubectl run ubuntu -it --image=ubuntu --restart=Never -- bash

Install curl in the Ubuntu pod:

apt update
apt install curl -y

Manually copy the security token and replace XXX in the following command with the security token. Execute the following commands in the shell in the Ubuntu pod:

echo “XXX” > securitytoken

Select the name of one of the nodes in the cluster.
I’ll select “k3d-k3s-default-server-0”.

Send a request to the kubelet instance on the selected node, replacing k3d-k3s-default-server-0 with the node name of your choice:

curl -k --header "Authorization: Bearer $(cat securitytoken)" https://k3d-k3s-default-server-0:10250/metrics

If the request was successful quite some output will be generated with the last of it looking something like this:

workqueue_work_duration_seconds_bucket{name="volumes",le="1e-07"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="1e-06"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="9.999999999999999e-06"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="9.999999999999999e-05"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="0.001"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="0.01"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="0.1"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="1"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="10"} 0
workqueue_work_duration_seconds_bucket{name="volumes",le="+Inf"} 0
workqueue_work_duration_seconds_sum{name="volumes"} 0
workqueue_work_duration_seconds_count{name="volumes"} 0

Metrics from kubelet running on one node of the cluster has been successfully retrieved. Feel free to repeat the process for the other nodes.

When done requesting kubelet metrics, exit the Ubuntu pod:

exit

Delete the Ubuntu pod:

kubectl delete pod ubuntu

Scraping metrics with Prometheus

The time has now come to scrape metrics of both Metrics Server and kubelet using Prometheus.

Metrics Server ServiceMonitor

In order to scrape metrics from Metrics Server using Prometheus a ServiceMonitor custom resource, as provided by the Prometheus operator, will be used.
Create a file named “metrics-server-servicemonitor.yaml” with the following contents:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: metrics-server-servicemonitor
  namespace: default
spec:
  namespaceSelector:
    matchNames:
      - kube-system
  selector:
    matchLabels:
      kubernetes.io/name: Metrics-server
  endpoints:
  - port: https
    path: /metrics
    interval: 5s
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
    scheme: https
    tlsConfig:
      insecureSkipVerify: true

Note that:

  • The ServiceMonitor will be created in the default namespace as specified by metadata.namespace.
    While it is to a recommended practice, the namespace is specified for the sake of the example as to make it clear that the ServiceMonitor does not have to reside in the same namespace as Prometheus nor in the same namespace as the service exposing the metrics.
  • The spec.namespaceSelector specifies in which namespace(s) services to scrape are to be sought for.
    In this case just the kube-system namespace is to be examined for matching services.
  • The spec.selector further refine the selection of services from which metrics are to be scraped.
    In this ServiceMonitor all services with the label kubernetes.io/name having the value Metrics-server, which probably will be just one single service, will be matched.
  • The port on which to scrape metrics is specified by endpoints.port and is named https.
    Recall that this is the name found when examining the metrics-server resource earlier.
  • The path at which to scrape metrics is specified by endpoints.path.
    Thus the URL at which to scrape metrics will look something like this: https://1.2.3.4/metrics
  • The interval at which to scrape metrics is 5 seconds, as specified by endpoints.interval.
  • The contents of the file specified by endpoints.bearerTokenFile will be enclosed with each request for metrics as bearer token.
    The bearer token is a security token that allows Prometheus in this case to access the metrics endpoint.
  • HTTPS will be used to access the metrics endpoint, as specified by endpoints.scheme.
  • Finally, SSL certificate verification is turned off, as specified by endpoints.tlsConfig.insecureSkipVerify.

Kubelet ServiceMonitor

To scrape metrics from kubelet using Prometheus, another ServiceMonitor will be used.

Create another file named “kubelet-servicemonitor.yaml” with the following contents:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kubelet-servicemonitor
  labels:
    k8s-app: kubelet
spec:
  jobLabel: k8s-app
  selector:
    matchLabels:
      app.kubernetes.io/name: kubelet
  namespaceSelector:
    matchNames:
    - kube-system
  endpoints:
  - port: https-metrics
    scheme: https
    interval: 5s
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
  - port: https-metrics
    scheme: https
    path: /metrics/cadvisor
    interval: 30s
    honorLabels: true
    tlsConfig:
      insecureSkipVerify: true
    bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token

Most parts of the above ServiceMonitor is similar to the Metrics Server ServiceMonitor. In the kubelet ServiceMonitor there are two endpoints specified from which metrics are to be scraped. The second one scrapes metrics from the path /metrics/cadvisor.

Get scrapin’

With the two ServiceMonitor files in place, let’s apply the files in order for Prometheus to start scraping Metrics Server and kubelet.

kubectl create -f metrics-server-servicemonitor.yaml
kubectl create -f kubelet-servicemonitor.yaml

Expose the the Prometheus web UI on localhost.

kubectl port-forward service/prometheus-operated 9090:9090

View the Prometheus web UI in a browser using the following URL: http://localhost:9090

Then go to the Status menu and select the Targets alternative and click all the Show Less buttons.
The UI should look something like this:

Prometheus scraping metrics from kubelet and Metrics Server targets.

Prometheus scraping metrics from kubelet and Metrics Server targets.

Note that:

  • There are two targets for kubelet.
    If the Show More button is clicked on both the kubelet targets the target endpoints are revealed with the target of one path being /metrics and the other being /metrics/cadvisor.
    Recall that the kubelet ServiceMonitor has two endpoints with the paths shown.
  • There are four endpoints, the 4/4 in parentheses after each target, for each of the kubelet targets.
    Recall that one instance of kubelet runs on each of the nodes in the cluster and that there are four nodes in the cluster created for the example in this article.
  • Similarly there is one endpoint, (1/1), for the Metrics Server target.
    This is consistent with what is known about Metrics Server – there is one instance of Metrics Server.

The IP addresses of the nodes in the cluster can be found using the following command:

kubectl get nodes -o jsonpath='{range .items[*]}{range @.status.addresses[*]}{@.address}{"\n"}{end}'

Given the example cluster, my output looks like this:

172.18.0.5
k3d-k3s-default-server-0
172.18.0.3
k3d-k3s-default-agent-2
172.18.0.6
k3d-k3s-default-agent-1
172.18.0.4
k3d-k3s-default-agent-0

Examine the IP addresses displayed in the Prometheus UI for a kubelet target, which on my computer looks like this:

Examine one kubelet target in the Prometheus web UI.

Examine one kubelet target in the Prometheus web UI.

The IP addresses of the nodes in the above output matches the IP addresses of the kubelet target endpoints in the Prometheus UI.

To find the IP address of the Metrics Server pod, use the following command in a terminal window:

kubectl get pod -l k8s-app=metrics-server -n kube-system -o jsonpath='{.items[0].status.podIP}{"\n"}'

The IP address of the Metrics Server pod in my example cluster is:

10.42.2.36

Click the Show More button in the Prometheus UI next to the Metrics Server target. The result should look something like this:

Examine the Metrics Server target in the Prometheus UI.

Examine the Metrics Server target in the Prometheus web UI.

Note that the endpoint IP address match the IP address of the Metrics Server pod obtained earlier.

Tips’n’tricks

This section contains a few pieces of advise for issues that I have come across when using Prometheus with the Prometheus operator to scrape metrics.

If no target appears in Prometheus after having applied for example a ServiceMonitor a common issue is a mistake as far as the namespace and/or the label matching is concerned.

Using the following command, the Prometheus configuration can be retrieved and examined for instance with the purpose to make sure that a scraping-job has been correctly created:

kubectl -n default get secret prometheus-prometheus -ojson | jq -r '.data["prometheus.yaml.gz"]' | base64 -d | gunzip

Note that the above require the following commands in addition to kubectl: jq, base64 and gunzip.

Final words

With cluster-metrics in Prometheus the next step may be to create one or more dashboards in Grafana or perhaps use a ready-made dashboard. I will leave this for a future article or as an exercise for the reader.

Happy coding!

Leave a Reply

Your email address will not be published. Required fields are marked *