In this article, we will show how to install Prometheus and Grafana to collect and display system performance metrics.

Prometheus is an open source monitoring and alerting toolkit for bare metal systems, virtual machines, containers, and microservices. Grafana allows you to query, visualize, and alert on metrics using fully customizable dashboards.

We will install Prometheus and Grafana on a dedicated bare-metal host (10.100.118.43) and the Prometheus Node Exporter on the bare-metal system(s) we want to collect data from (10.100.118.42). We will not cover installing the components in Docker containers. There are instructions in the documentation for this. Additionally, we will not cover how to secure the environment

Install Prometheus

Before we proceed, it’s good to note that the Prometheus server listens on TCP Port 9090. This is the same port that Fedora Cockpit listens on. As a result, if you are not using the Cockpit Web Service Manager, disable and stop both the service and socket to free port 9090.

$ sudo netstat -tulpn | grep -i 9090
tcp6       0      0 :::9090                 :::*                    LISTEN      1/systemd

$ sudo systemctl stop cockpit cockpit.socket
$ sudo systemctl disable --now cockpit cockpit.socket

$ sudo netstat -tulpn | grep -i 9090
$

Create Prometheus System User and Group

To create prometheus system user and group, run:

$ sudo useradd -M -r -s /bin/false prometheus
$ sudo getent passwd prometheus

Download and Configure Prometheus

On the monitoring master node download and install the Linux bundle. Use the latest version available from https://prometheus.io/download/ and install it in /opt.

$ sudo wget -P /opt/ https://github.com/prometheus/prometheus/releases/download/v2.18.1/prometheus-2.18.1.linux-amd64.tar.gz
$ cd /opt/
$ sudo tar xf prometheus-2.18.1.linux-amd64.tar.gz
$ sudo ln -s prometheus-2.18.1.linux-amd64 prometheus
$ cd /opt/prometheus
$ sudo chown -R prometheus:prometheus /opt/prometheus*

Start prometheus

$ sudo /opt/prometheus/prometheus

You can start prometheus listening on a different port using:

$ sudo /opt/prometheus/prometheus --web.listen-address=10.100.118.43:9060

To make permanent changes, copy the prometheus.yaml file to /etc/

$ cp /opt/prometheus/prometheus.yml /etc/prometheus.yml

Edit the /etc/prometheus.yml file and make the change(s). For example, to change the default listening port from 9090 to 9060 change

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9060']

Open the firewall

$ sudo firewall-cmd --permanent --add-port=9090/tcp
$ sudo firewall-cmd --reload

Test the connection by going to http://10.100.118.43:9060 in your browser. Note: Modern browsers such as Chrome, Edge, etc default to forcing HTTPS. If you get ‘Connection Refused’, this is likely the cause since Prometheus does not listen for HTTPS requests by default.

Create a Prometheus Systemd Service

To be able to run Prometheus as a service, you need to create a systemd service file, /etc/systemd/system/prometheus.service, configured as follows. Note that this assumes Prometheus server is listening on default port 9090.

$ sudo vim /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus Time Series Collection and Processing Server
Wants=network-online.target
After=network-online.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

If you configured the Prometheus server to listen on non-default port, edit the ExecStart= section of the service file and add the line, --web.listen-address=0.0.0.0:8080, eg:

...
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries \
    --web.listen-address=0.0.0.0:8080
...

Reload systemd daemon configuration.

$ sudo systemctl daemon-reload

Start and Enable Prometheus service to run at boot time.

systemctl start prometheus
systemctl enable prometheus

Confirm the service is running

# systemctl status prometheus
● prometheus.service - Prometheus Time Series Collection and Processing Server
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-05-13 17:09:23 MDT; 1s ago
 Main PID: 43651 (prometheus)
    Tasks: 50 (limit: 14745)
   Memory: 60.1M
   CGroup: /system.slice/prometheus.service
           └─43651 /opt/prometheus/prometheus --config.file /opt/prometheus/prometheus.yml --storage.tsdb.path /opt/prometheus/data/ --web.console.templates=/opt/prometheus/consoles --web.console.libraries=/opt/prometheus/console_libraries --web.listen-address=0.0.0.0:9090

Prometheus is now running as a service.

Install Node Exporter

Node_Exporter is a Prometheus exporter plugin that collects data from a target system and pushes the information to the Prometheus server. It can be found on the https://prometheus.io/download/ page.

Create a Node Exporter System User and Group

We’ll create a systemd server shortly, but first, we need to create a user and group. The instructions say the systemd service needs a user named node_exporter, whose shell should be /sbin/nologin and should not have any special privileges.

$ sudo useradd -M -r -s /sbin/nologin node_exporter
$ sudo getent passwd node_exporter

Download and Install Node Exporter

Install node_exporter to /opt on the target system.

$ sudo wget -P /opt/ https://github.com/prometheus/node_exporter/releases/download/v1.0.0-rc.0/node_exporter-1.0.0-rc.0.linux-amd64.tar.gz
$ cd /opt/
$ sudo tar xf node_exporter-1.0.0-rc.0.linux-amd64.tar.gz
$ sudo ln -s /opt/node_exporter-1.0.0-rc.0.linux-amd64 /opt/prometheus_node_exporter
$ sudo chown -R node_exporter:node_exporter /opt/prometheus_node_exporter node_exporter*
$ sudo /opt/prometheus_node_exporter/node_exporter --web.listen-address="0.0.0.0:9100"

Open port 9100 in the firewall

$ sudo firewall-cmd --permanent --add-port=9100/tcp
$ sudo firewall-cmd --reload

Use a web browser to verify the connection is working – http://10.100.118.42:9100/metrics. This will return the metrics which looks like this:

# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 0.000108763
go_gc_duration_seconds{quantile="0.25"} 0.000173862
go_gc_duration_seconds{quantile="0.5"} 0.000240759
go_gc_duration_seconds{quantile="0.75"} 0.000272835
go_gc_duration_seconds{quantile="1"} 0.000307421
go_gc_duration_seconds_sum 0.001686148
go_gc_duration_seconds_count 8
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 9
[...snip...]

Note: Modern browsers such as Chrome, Edge, etc default to forcing HTTPS. If you get ‘Connection Refused’, this is likely the cause.

Use Ctrl-C to stop the process and we’ll create a systemd service to manage Node Exporter.

Create a Node Exporter Systemd Service

To be able to run the Node Exporter as a service, you need to create a systemd service file, /etc/systemd/system/node_exporter.service, configured as follows.

$ sudo vi /etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter

[Service]
User=node_exporter
Group=node_exporter
EnvironmentFile=/etc/sysconfig/node_exporter
ExecStart=/opt/prometheus_node_exporter/node_exporter --web.listen-address="0.0.0.0:9100"\
        --collector.textfile.directory /var/lib/node_exporter/textfile_collector

[Install]
WantedBy=multi-user.target

Create the required directories

$ sudo mkdir -p /var/lib/node_exporter/textfile_collector
$ sudo chown node_exporter:node_exporter /var/lib/node_exporter/textfile_collector
$ sudo touch /etc/sysconfig/node_exporter
$ sudo chown node_exporter:node_exporter /etc/sysconfig/node_exporter

Reload systemd daemon configuration.

$ sudo systemctl daemon-reload

Start and Enable Prometheus service to run at boot time.

$ sudo systemctl start node_exporter
$ sudo systemctl enable node_exporter

Confirm the service is running

$sudo systemctl status node_exporter
● node_exporter.service - Node Exporter
   Loaded: loaded (/etc/systemd/system/node_exporter.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-05-13 17:34:14 MDT; 56s ago
 Main PID: 258735 (node_exporter)
    Tasks: 132 (limit: 460908)
   Memory: 31.7M
      CPU: 5.605s
   CGroup: /system.slice/node_exporter.service
           └─258735 /opt/prometheus_node_exporter/node_exporter --web.listen-address=0.0.0.0:9100 --collector.textfile.directory /var/lib/node_exporter/textfile_collector

Install Grafana

See the instructions here – https://grafana.com/docs/grafana/latest/installation/rpm/#install-from-yum-repository.

Add the Grafana package repository:

$ cat <<EOF | sudo tee /etc/yum.repos.d/grafana.repo
[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

Install Grafana:

$ sudo dnf -y install grafana

To start grafana service and enable it to start on boot, run:

$ sudo systemctl start grafana-server
$ sudo systemctl enable grafana-server

Confirm the service state:

$ sudo systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; disabled; vendor preset: disabled)
   Active: active (running) since Wed 2020-05-13 07:43:38 MDT; 2s ago
     Docs: http://docs.grafana.org
 Main PID: 38299 (grafana-server)
    Tasks: 24 (limit: 14745)
   Memory: 67.6M
   CGroup: /system.slice/grafana-server.service
           ├─38299 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --pac>
           └─38317 /var/lib/grafana/plugins/grafana-image-renderer/plugin_start_linux_amd64

The default HTTP port is 3000. You can change this by editing the configuration file in /etc/grafana/grafana.ini. By default Grafana will log to /var/log/grafana.

Open port 3000 in the firewall

$ sudo firewall-cmd --permanent --add-port=3000/tcp
$ sudo firewall-cmd --reload

Verify you can access the Grafana server using your browser to http://%5BServer IP|Hostname]:3000

Figure 1: Grafana Login Screen

The default login credentials is:

username: admin
Password: admin

Don’t forget to change the admin password during the first login. Grafana will prompt you to change the password each time if it detects the username and password are the defaults:

Figure 2: Change Password on initial login

After logging in and change the Admin password, you will be greeted with the home page.

Figure 3: Grafana Home Page

Along the top of the home page is a step-by-step guide to complete the install and setup of Grafana.

Figure 4: Grafana Initial Setup Steps

Click ‘Add data source’ and you’ll be presented with a long list of sources we can configure. At the bottom is a an option to “Find more data source plugins on grafana.com”.

Figure 5: Add Grafana Data Source Plugins

For our purposes, click ‘Prometheous’ and you’ll see the configuration options for this plugin.

Figure 6: Configure the Prometheus Data Source Plugin

For an initial test, we only need to specify the URL of the Prometheus instance and click ‘Save & Test’. For production, we would want to configure security and use HTTPS.

Figure 7: Successfully Configured and Tested the Prometheus Data Source Plugin

Now that we have the data source configured, we need to create a dashboard to visualize the data. Creating a dashboard from scratch is a lot of work, so I recommend using one that has been created by the community to begin with. We can customize it later.

You can find a lot of community created dashboards at https://grafana.com/grafana/dashboards. For this article, we will use ‘Node Exporter Full‘ Dashboard by rfraile.

Figure 8: Node Exporter Full Grafana Dashboard

Make a note of the Dashboard ID (1860) or click ‘Copy ID to Clipboard’.

In Grafana, click ‘New Dashboard’ in the setup steps along the top

Figure 9: Add a New Dashboard

You will be presented with a blank Dashboard with a few options.

Figure 10: Blank Dashboard

We are going to Import the ‘Node Exporter Full’ dashboard. You can either click the ‘New Dashboard’ drop down in the top left, or select the ‘+’ from the options on the left, then select ‘Import’ from the popup menu. Both methods will get you to the Import page where we enter the Dashboard ID (1080) or URL https://grafana.com/grafana/dashboards/1860. Alternatively, if you want to make changes to to the dashboard, you can download the dashboard JSON file, make the change(s), then upload it.

Figure 11: Import the Node Exporter Full Dashboard

On the next screen, we need to complete a couple of drop-down fields and hit ‘Import’ to complete the process.

Figure 12: Configure the Node Exporter Full Dashboard

You will be redirected back to the Dashboard where you’ll see the many stats reported by this dashboard.

Figure 13: Node Exporter Full Dashboard showing statistics from one host

That’s all Folks! Navigate around and enjoy looking at the metrics.