Grafana Prometheus Dashboard Tutorial

Introduction

Prometheus is an open-source event monitoring software for high-volume distributed applications. It features real-time metrics and alerting, flexible queries, an HTTP pull model, and is a good choice for monitoring Kubernetes clusters.

While Prometheus is a stand-alone application, using it with a visualization dashboard helps maintain a better overview of a Kubernetes cluster. A popular choice for this scenario is Grafana, an open-source visualization software, which supports Prometheus as a data source out-of-the-box.

This tutorial will show you how to install Grafana in your Kubernetes cluster, connect it to Prometheus and organize data in dashboards.

Grafana Prometheus Dashboard Tutorial

Prerequisites

Install Grafana

To install Grafana on a Kubernetes cluster, create and apply a series of yaml files.

1. Start by creating a ConfigMap in a file titled grafana-datasource-config.yaml:

nano grafana-datasource-config.yaml

The file consists of the following declarations:

apiVersion: v1 
kind: ConfigMap 
metadata:
 name: grafana-datasources
 namespace: default 
data: 
  prometheus.yaml: |- 
    { 
        "apiVersion": 1, 
        "datasources": [ 
            { 
               "access":"proxy", 
                "editable": true, 
                "name": "prometheus", 
                "orgId": 1, 
                "type": "prometheus", 
                "url": "http://prometheus-service.monitoring.svc:9090", 
                "version": 1 
            } 
        ] 
    }

2. Adjust the file’s contents to fit your needs by specifying the namespace you wish to install Grafana in.

3. Save and exit the file.

4. Apply changes to your cluster using kubectl:

kubectl create -f grafana-datasource-config.yaml

The output confirms the creation of the ConfigMap:

Using kubectl to create grafana-datasources configmap
5. Next, create a yaml file for Grafana deployment:

nano deployment.yaml

The file contains the Grafana deployment configuration:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
  namespace: default
spec:
  replicas: 1
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      name: grafana
      labels:
        app: grafana
    spec:
      containers:
      - name: grafana
        image: grafana/grafana:latest
        ports:
        - name: grafana
          containerPort: 3000
        resources:
          limits:
            memory: "1Gi"
            cpu: "1000m"
          requests: 
            memory: 500M
            cpu: "500m"
        volumeMounts:
          - mountPath: /var/lib/grafana
            name: grafana-storage
          - mountPath: /etc/grafana/provisioning/datasources
            name: grafana-datasources
            readOnly: false
      volumes:
        - name: grafana-storage
          emptyDir: {}
        - name: grafana-datasources
          configMap:
              defaultMode: 420
              name: grafana-datasources

6. Save the file, then apply it to the cluster:

kubectl create -f deployment.yaml

The output confirms the deployment creation:

Using kubectl to create grafana deployment
7. Lastly, create the service.yaml file to configure Grafana service:

nano service.yaml

The file consists of the following declarations:

apiVersion: v1
kind: Service
metadata:
  name: grafana
  namespace: default
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '3000'
spec:
  selector: 
    app: grafana
  type: NodePort  
  ports:
    - port: 3000
      targetPort: 3000
      nodePort: 32000

8. Adjust the file to the needs of your cluster, then save it and apply it:

kubectl create -f service.yaml

The output confirms the service was created successfully:

Using kubectl to create Grafana service
9. Use the kubectl port-forward command to forward Grafana service to port 3000:

kubectl port-forward svc/grafana 3000

Forwarding Grafana to port 3000 using kubectl
10. Now open a browser and navigate to http://localhost:3000/.

11. The Grafana login screen appears, asking for your username and password. Type admin in both fields and click Log in.

Grafana login screen
12. Grafana asks you to set a new password, after which the home page is shown.

Add Prometheus as a Grafana Data Source

To start working with Prometheus in Grafana, add Prometheus as a data source.

1. In the sidebar on the left, hover over the cogwheel icon to reveal the configuration submenu.

2. Click Data Sources.

Accessing Grafana data sources management in the Grafana Configuration menu
2. A configuration page opens. Click the Add data source button.

Adding a data source in Grafana
3. Select Prometheus as the data source type by clicking the Select button on the right.

Selecting Prometheus as a data source in Grafana
4. The configuration page for Prometheus opens. Here you specify the name of the source, the URL and port for the Prometheus service, and the type of access (server access is the default option).

Setting up Prometheus as a data source in Grafana
5. Once you are done, scroll down to the bottom of the page and click the Save & Test button.

Saving and testing Prometheus as a data source in Grafana

Grafana tests the connection with Prometheus and finishes adding the data source.


Note: If you receive the “Error Bad Gateway” error, try changing the HTTP Access option from Server to Browser.


Import Prometheus Stats as a Dashboard into Grafana

Grafana supports Prometheus out-of-the-box and you can easily access the premade Prometheus dashboard.

1. Navigate back to the Data Sources section and select Prometheus.

2. Open the Settings drop-down menu.

3. Then click Dashboards.

Accessing the dashboard section in Grafana data sources
4. A list of available dashboards appears. Import the dashboard you want by clicking the Import button to the right of the dashboard name.

Selecting a dashboard in Prometheus section of data sources in Grafana
5. To access the dashboard, hover over the four-squares icon in the left sidebar to reveal the Dashboards menu.

6. Click Manage.

Accessing Prometheus dashboard in Grafana

7. Select the Prometheus dashboard you want to manage from the list.

Selecting the Prometheus Stats dashboard in Grafana

Statistics from Your Prometheus Server

The Prometheus dashboard consists of several single-stat graphs that are described below.

Single Stat Graphs

After you select the Prometheus Stats dashboard on the Dashboard Management page, the stats appear organized in panels.

The look of Prometheus dashboard in Grafana

The first row consists of the following panels:

      • Uptime: How much time has passed since the Prometheus server was started
      • Local Storage Memory Series: The number of series Prometheus currently holds in memory
      • Internal Storage Queue Length: The lower this number is the better. The ‘Empty’ message means the queue number is zero.

The Samples ingested panel provides information about the number of sample messages Prometheus ingested at the rate of 5 minutes.

Prometheus Scrape Metrics

Scrape metrics panels are just below the Samples ingested panel.

      • The Target Scrapes panel shows the frequency of scraping the target, i.e., Prometheus, measured over the last five minutes, per time series in the range vector.
      • The Scrape Duration panel shows the duration of scrapes, measured over the same five-minute period.

Prometheus scrape metrics section in Grafana Prometheus dashboard

Evaluations Duration

The Rule Evaluation Duration panel found below the Scrape metrics panels provides information on the length of all evaluations to execute.

Create a New Dashboard

To create a new Grafana dashboard:

1. Navigate to the Dashboards page and click the New Dashboard button.

Creating a new dashboard in Grafana
2. A screen appears where you can add new empty panels and rows.

Starting a new dashboard in Grafana
3. A new screen appears with a single dummy panel.

A new Grafana dashboard with an empty panel

Customize the Dashboard Using Prometheus Query Editor

To edit a specific Grafana panel:

1. Click the panel’s title to open a drop-down menu.

2. In the menu, click Edit.

Accessing the edit section to edit a graph in query editor
3. The edit window appears. Customize panels by introducing new queries or modifying the current ones. The querying is performed using Prometheus query language.

Editing a graph in query editor in Grafana
4. The following metrics are not featured in Grafana’s premade Prometheus dashboard, but are useful for monitoring Prometheus:

      • prometheus_local_storage_memory_chunks – Monitors memory chunks Prometheus stores in memory.
      • prometheus_local_storage_memory_series – Monitors memory series Prometheus stores in memory.
      • prometheus_local_storage_ingested_samples_total – Measures the ingestion rate for the samples.
      • prometheus_target_interval_length_seconds – Measures the amount of time between target scrapes.
      • prometheus_local_storage_chunk_ops_total – Monitors the per-second rate of all Prometheus storage chunk operations.

Use Premade Dashboards and Plugins

Grafana offers a large selection of premade dashboards and plugins available for download on its website. To install a premade dashboard:

1. Download the plugin’s json file.

2. Click the Import button and follow the instructions to upload the file to Grafana.

Importing a new dashboard using a JSON file in Grafana
3. To install plugins, log into the Grafana pod in the cluster using kubectl exec.

kubectl exec -it [grafana-pod] -- [shell]

Using kubectl exec to log into Grafana pod
4. Now use the grafana-cli command-line tool to install the plugin:

grafana-cli plugins install grafana-worldmap-panel

Installing a plugin using grafana-cli tool
The plugin installs to the Grafana plugins directory. The default location is /var/lib/grafana/plugins. Once the plugin is successfully installed, it is automatically available in Grafana.

Conclusion

After reading this tutorial, you should be able to install and set up Grafana to use it with Prometheus, an important DevOps tool. If you are interested in Prometheus in terms of cloud monitoring, read the guide dedicated to the best cloud monitoring tools.

Marko Aleksic
Marko Aleksić is a Technical Writer at phoenixNAP. His innate curiosity regarding all things IT, combined with over a decade long background in writing, teaching and working in IT-related fields, led him to technical writing, where he has an opportunity to employ his skills and make technology less daunting to everyone.
Next you should read
Install Elasticsearch On Kubernetes Using Helm Chart
March 18, 2021

There are several methods for deploying Elasticsearch and the rest of the ELK stack on Kubernetes. Using Helm…
Read more
19 Kubernetes Best Practices For Building Efficient Clusters
September 23, 2020

Kubernetes is a feature-rich orchestration tool. The best practices outlined in this article are going to…
Read more
How To Monitor Kubernetes With Prometheus
February 24, 2020

This tutorial shows you how to create a series of .yml files to set up Prometheus Monitoring on your…
Read more
Building Optimized Containers For Kubernetes
December 18, 2019

Container deployment took the software development world by storm. Use the outlined guidelines and learn how…
Read more
  • © 2021 Copyright phoenixNAP | Global IT Services. All Rights Reserved.