note:  This article is a litte bit out of date. Kusama is evolving very fast. Please after reading it have a look to this new one for a new and easy kusama data exporter:

https://blog.colmenalabs.org/monitoring-series-5/

In our first article we learn how to setup a Prometheus and Grafana monitoring and alerting system. In our second article we also learn how to monitor Tendermint based blockchains nodes.

Now in this article we'll show you how to monitor Substrate based blockchain nodes (like Polkadot) using our awesome Prometheus/Grafana monitoring stack. To be more specific we'll focus on how to monitor a Kusama network node.

Table of contents

  1. Intro
  2. Install Substrate Telemetry
  3. Install Telemetry Exporter
  4. Configure polkadot node
  5. Configure Prometheus
  6. Get fun!

Intro

In the case of a polkadot daemon, we don't have the possibility of enable a flag in the daemon to directly expose the node metrics to Prometheus. Instead of that, we need to install some aditional software in our polkadot node to be able to do so: Substrate Telemetry (only the backend app) and Telemetry Exporter.

Substrate node monitoring with Prometheus, Substrate Telemetry and Telemetry Exporter

So to get to whole picture, our polkadot node will send their metrics to a local Substrate Telemetry server. Telemetry Exporter (also running locally in the node) then will connect to the Telemetry server and expose the polkadot node metrics to Prometheus. Then we'll able to create graphs, alerts and dashboards based on that metrics in Grafana. Simple, isn't? ;-)

Install Substrate Telemetry

apt update
apt upgrade -y
apt-get install -y build-essential
curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash -
apt-get install -y nodejs
npm install -g yarn
cd /usr/local/
git clone https://github.com/paritytech/substrate-telemetry.git
cd substrate-telemetry
yarn
yarn build:backend
npm install forever --global
forever start -l /var/log/telemetry.backend.log -a /usr/local/substrate-telemetry/packages/backend/build/index.js

You can check the logs like this:

# tail -f /var/log/telemetry.backend.log
[Kusama CC2] New block 170360
[Kusama CC2] PolkaStats imported 170360, block time: 11.428s, average: 10.113100000000001s | latency 2
[System] 1 open telemetry connections; 1 open feed connections
[System] 1 open telemetry connections; 1 open feed connections
[Kusama CC2] New block 170361
[Kusama CC2] PolkaStats imported 170361, block time: 8.457s, average: 9.8841s | latency 1
[System] 1 open telemetry connections; 1 open feed connections
[System] 1 open telemetry connections; 1 open feed connections
[Kusama CC2] New block 170362
[Kusama CC2] PolkaStats imported 170362, block time: 11.063s, average: 10.1852s | latency 0
[System] 1 open telemetry connections; 1 open feed connections
^C

Install Telemetry Exporter

cd /usr/local/
git clone https://github.com/w3f/substrate-telemetry-exporter
cd substrate-telemetry-exporter
yarn
forever start -l /var/log/telemetry.exporter.log -a /usr/local/substrate-telemetry-exporter/src/index.js

You can check the logs like this:

# tail -f /var/log/telemetry.exporter.log
New best block 170347
Block 170347 imported at node 2
data: [10,1570565169413]
data: [1,[170348,1570565171517,9942],6,[2,[170348,"0xaccdf9b375ca420572bb27f21623818b974d1209a1d19f9a0a7ad8bc644f19f4",10743,1570565171517,0]]]
New best block 170348
Block 170348 imported at node 2
data: [10,1570565179416]
data: [1,[170349,1570565180073,9747.8],6,[2,[170349,"0x9dc6e84177d1fdb7b766239f454991945629003bdafb44a89df713d8b55d1be5",8557,1570565180073,0]]]
New best block 170349
Block 170349 imported at node 2
data: [10,1570565189417]
^C

Configure polkadot node

We need polkadot node to send their metrics to local Substrate Telemetry server, so we must start the polkadot daemon with the flag `--telemetry-url` like:

/usr/local/polkadot/target/release/polkadot --telemetry-url ws://127.0.0.1:1024 ... rest of your flags

Configure Prometheus

Now we need to add our new data source at the end of the prometheus.yaml config file (in scrape_configs: section):

 - job_name: 'PolkadotKusamaNode'
    scrape_interval: 5s
    static_configs:
    - targets: ['YOUR_POLKADOT_NODE_IP:3000']
      labels:
        instance: 'your_polkadot_kusama_node_name'

Save and restart your prometheus server.

Get fun!

Let's open the Prometheus GUI. Browse to http://YOUR_SERVER_IP:9090

In the query input enter the metric polkadot_best_finalized and you will get the node's latest finalized block:

To get a graph of the value, you can click on Graph tab:

This is only an example, there is a buch of other interesting node metrics available: