We are starting a serie of articles focusing on monitoring system design for your nodes.

Table of contents

  1. Prometheus
  2. Server side installation
  3. Client side installation
  4. Prometheus config file
  5. Prometheus basic GUI

The topics we will cover along the serie are:

Components
  1 Prometheus
  2 Exporters
    . localhost node_exporter
    . Tendermint daemons metrics
    . Polkadot telemetry_exporter
  3 Grafana
    . Metrics queries
    . Alerts
      . email alerts
      . Telegram alerts    

Prometheus

Prometheus is 100% open source and community-driven. All components are available under the Apache 2 License on GitHub. https://prometheus.io

Prometheus includes a backend and a simple front very usefull to start playing with graphs and metrics queries.

Server side installation

You could compile the source code or you better just download it directly to your monitoring server:

root@monitor:~# cd /usr/local
root@monitor:/usr/local# wget https://github.com/prometheus/prometheus/releases/download/v2.13.0/prometheus-2.13.0.linux-amd64.tar.gz
root@monitor:/usr/local# tar xvzf prometheus-2.13.0.linux-amd64.tar.gz
root@monitor:/usr/local# ln -s prometheus-2.13.0.linux-amd64/ prometheus

What we have done here is to download the tarball to a common linux folder, extract it and finally we have created a simbolic link to shorten the path.

Client side installation

Now we will install the client side in a different machine. If you also want to monitor your monitoring server you can install de client software in the same machine.

root@client:~# cd /usr/local
root@client:/usr/local# wget https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz
root@client:/usr/local# tar xzvf node_exporter-0.18.1.linux-amd64.tar.gz
root@client:/usr/local# ln -s node_exporter-0.18.1.linux-amd64 node_exporter
root@client:/usr/local# cd node_exporter
root@client:/usr/local/node_exporter# ./node_exporter

After last command we have started our metrics exporter daemon and you will see an output like this:

INFO[0000] Starting node_exporter (version=0.18.1, branch=HEAD, revision=3db77732e925c08f675d7404a8c46466b2ece83e)  source="node_exporter.go:156"
INFO[0000] Build context (go=go1.12.5, user=root@b50852a1acba, date=20190604-16:41:18)  source="node_exporter.go:157"
INFO[0000] Enabled collectors:                           source="node_exporter.go:97"
INFO[0000]  - arp                                        source="node_exporter.go:104"
INFO[0000]  - bcache                                     source="node_exporter.go:104"
INFO[0000]  - bonding                                    source="node_exporter.go:104"
INFO[0000]  - conntrack                                  source="node_exporter.go:104"
INFO[0000]  - cpu                                        source="node_exporter.go:104"
INFO[0000]  - cpufreq                                    source="node_exporter.go:104"
INFO[0000]  - diskstats                                  source="node_exporter.go:104"
                                 
[...]

INFO[0000]  - uname                                      source="node_exporter.go:104"
INFO[0000]  - vmstat                                     source="node_exporter.go:104"
INFO[0000]  - xfs                                        source="node_exporter.go:104"
INFO[0000]  - zfs                                        source="node_exporter.go:104"
INFO[0000] Listening on :9100                            source="node_exporter.go:170"

The daemon will be listening on port 9100 by default. We will create a systemd service for this daemon later. By now we are just testing that every piece of the puzzle is working.

Prometheus config file

Next step is telling our Prometheus to talk to our first client machine. This is done in prometheus.yml file of our monitoring server.

root@monitor:~# vim /usr/local/prometheus/prometheus.yml

And paste this. Pay attention to the "YOUR_IP" and change it for your client ip numbers.

global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

  external_labels:
    monitor: 'prometheus-monitor'

scrape_configs:
  - job_name: 'node'
    static_configs:
    - targets: ['YOUR_IP:9100']

Prometheus basic GUI

To finish with this firt article of the serie we are starting our prometheus and see what it shows in the explorer.

root@monitor:~# /usr/local/prometheus/prometheus --config.file=/usr/local/prometheus/prometheus.yml

Browse to: http://YOUR_SERVER_IP:9090

Here is the end of the first article. If you want to play a bit with your new great monitoring system try to write a query or use the "insert metric at cursor" selector.

Query Expression example:   node_load1

You will get Console output value. Have a look to the Graph tab.

That's all folks!!

Stay tuned for the next article.