Monitor Aerospike Kubernetes Operator

This page describes how to monitor Aerospike Kubernetes Operator.

Aerospike Kubernetes Operator (AKO) exposes metrics on the /metrics endpoint. This endpoint is by default secure (HTTPS) and requires TLS certificates for secure communication. If no TLS certificates are provided, self-signed certificates are generated. Prometheus must have the required permissions to scrape these metrics.

Configure system to scrape AKO metrics

To collect metrics, configure Prometheus to scrape the /metrics endpoint. You can do this manually by installing and configuring the Prometheus Operator from GitHub, or by using the pre-built AKO monitoring stack.

Using Prometheus Operator

See Install Prometheus Operator to install the kube-prometheus-stack, which includes the Prometheus Operator and Grafana.

You can create the ServiceMonitor resource using any one of the following methods based on your AKO installation mode:

OLM
Helm

AKO uses a Kubernetes Service labeled control-plane: controller-manager to make its metrics available for scraping. The Prometheus Operator uses ServiceMonitor resources to learn which pods to scrape for endpoints and how they should be scraped. Create a ServiceMonitor resource to configure Prometheus to find this Kubernetes Service and monitor it at regular intervals to scrape the AKO /metrics endpoint.

Create a file named service-monitor.yaml.

With custom TLS certificates (Recommended for production)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: controller-manager
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s # Scraping interval
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        serverName: aerospike-operator-controller-manager-metrics-service.operators.svc   # Naming Format: <service-name>.<namespace>.svc
        insecureSkipVerify: false
        ca:
          secret:
            name: metrics-server-cert
            key: ca.crt
        cert:
          secret:
            name: metrics-server-cert
            key: tls.crt
        keySecret:
          name: metrics-server-cert
          key: tls.key
  selector:
    matchLabels:
      control-plane: controller-manager    # AKO metric service label

With TLS certificate validation disabled (insecureSkipVerify: true) for testing environment

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: controller-manager
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s # Scraping interval
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true # Disables TLS certificate verification. Not recommended for production.
  selector:
    matchLabels:
      control-plane: controller-manager    # AKO metric service label

AKO uses a Kubernetes Service labeled chart: aerospike-kubernetes-operator to make its metrics available for scraping. The Prometheus Operator uses ServiceMonitor resources to learn which pods to scrape for endpoints and how they should be scraped. Create a ServiceMonitor resource to configure Prometheus to find this Kubernetes Service and monitor it at regular intervals to scrape the AKO /metrics endpoint.

Create a file named service-monitor.yaml.

With custom TLS certificates (Recommended for production)

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    chart: aerospike-kubernetes-operator
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s # Scraping interval
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        serverName: aerospike-operator-controller-manager-metrics-service.aerospike.svc   # Naming Format: <service-name>.<namespace>.svc
        insecureSkipVerify: false
        ca:
          secret:
            name: metrics-server-cert
            key: ca.crt
        cert:
          secret:
            name: metrics-server-cert
            key: tls.crt
        keySecret:
          name: metrics-server-cert
          key: tls.key
  selector:
    matchLabels:
      chart: aerospike-kubernetes-operator    # AKO metric service label

With TLS certificate validation disabled (insecureSkipVerify: true) for testing environment

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    chart: aerospike-kubernetes-operator
  name: aerospike-operator-controller-manager-metrics-monitor
spec:
  endpoints:
    - path: /metrics
      interval: 15s # Scraping interval
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: true # Disables TLS certificate verification. Not recommended for production.
  selector:
    matchLabels:
      chart: aerospike-kubernetes-operator    # AKO metric service label

Verify that the Prometheus deployment is configured to select the AKO ServiceMonitor using serviceMonitorSelector. Ensure that the serviceMonitorSelector includes the labels used in the ServiceMonitor.
Terminal window
```
kubectl -n PROMETHEUS_NAMESPACE get prometheus PROMETHEUS_CR_NAME -o yaml
```

Apply the ServiceMonitor resource:

kubectl apply -f service-monitor.yaml -n AKO_NAMESPACE

AKO exposes two types of Prometheus metrics:

K8s controller runtime metrics – These are the default metrics exposed by the controller-runtime framework. See the full list here: Kubebuilder Metrics.
AKO custom metrics – Additional metrics exposed by AKO to provide more insights.

Metric Name	Description
`aerospike_ako_aerospikecluster_phase`	Phase of AerospikeCluster CRs. (InProgress, Error or Completed)
`aerospike_ako_eviction_webhook_requests_total`	Total number of pod eviction requests processed by the webhook. Labels: `eviction_namespace`, `decision`

Using Aerospike Kubernetes Operator Monitoring Stack

If you have cloned the AKO repository, you can apply the monitoring configurations in one step with kubectl:

kubectl apply -k config/monitoring

Grafana dashboards

To visualize metrics exposed by AKO, import pre-built Grafana dashboards from Grafana Dashboard.