note: This article is a litte bit out of date. Kusama is evolving very fast. Please after reading it have a look to this new one for a new and easy kusama data exporter:
Now in this article we'll show you how to monitor Substrate based blockchain nodes (like Polkadot) using our awesome Prometheus/Grafana monitoring stack. To be more specific we'll focus on how to monitor a Kusama network node.
Table of contents
- Install Substrate Telemetry
- Install Telemetry Exporter
- Configure polkadot node
- Configure Prometheus
- Get fun!
In the case of a polkadot daemon, we don't have the possibility of enable a flag in the daemon to directly expose the node metrics to Prometheus. Instead of that, we need to install some aditional software in our polkadot node to be able to do so: Substrate Telemetry (only the backend app) and Telemetry Exporter.
So to get to whole picture, our polkadot node will send their metrics to a local Substrate Telemetry server. Telemetry Exporter (also running locally in the node) then will connect to the Telemetry server and expose the polkadot node metrics to Prometheus. Then we'll able to create graphs, alerts and dashboards based on that metrics in Grafana. Simple, isn't? ;-)
Install Substrate Telemetry
apt update apt upgrade -y apt-get install -y build-essential curl -sL https://deb.nodesource.com/setup_10.x | sudo -E bash - apt-get install -y nodejs npm install -g yarn cd /usr/local/ git clone https://github.com/paritytech/substrate-telemetry.git cd substrate-telemetry yarn yarn build:backend npm install forever --global forever start -l /var/log/telemetry.backend.log -a /usr/local/substrate-telemetry/packages/backend/build/index.js
You can check the logs like this:
# tail -f /var/log/telemetry.backend.log [Kusama CC2] New block 170360 [Kusama CC2] PolkaStats imported 170360, block time: 11.428s, average: 10.113100000000001s | latency 2 [System] 1 open telemetry connections; 1 open feed connections [System] 1 open telemetry connections; 1 open feed connections [Kusama CC2] New block 170361 [Kusama CC2] PolkaStats imported 170361, block time: 8.457s, average: 9.8841s | latency 1 [System] 1 open telemetry connections; 1 open feed connections [System] 1 open telemetry connections; 1 open feed connections [Kusama CC2] New block 170362 [Kusama CC2] PolkaStats imported 170362, block time: 11.063s, average: 10.1852s | latency 0 [System] 1 open telemetry connections; 1 open feed connections ^C
Install Telemetry Exporter
cd /usr/local/ git clone https://github.com/w3f/substrate-telemetry-exporter cd substrate-telemetry-exporter yarn forever start -l /var/log/telemetry.exporter.log -a /usr/local/substrate-telemetry-exporter/src/index.js
You can check the logs like this:
# tail -f /var/log/telemetry.exporter.log New best block 170347 Block 170347 imported at node 2 data: [10,1570565169413] data: [1,[170348,1570565171517,9942],6,[2,[170348,"0xaccdf9b375ca420572bb27f21623818b974d1209a1d19f9a0a7ad8bc644f19f4",10743,1570565171517,0]]] New best block 170348 Block 170348 imported at node 2 data: [10,1570565179416] data: [1,[170349,1570565180073,9747.8],6,[2,[170349,"0x9dc6e84177d1fdb7b766239f454991945629003bdafb44a89df713d8b55d1be5",8557,1570565180073,0]]] New best block 170349 Block 170349 imported at node 2 data: [10,1570565189417] ^C
Configure polkadot node
We need polkadot node to send their metrics to local Substrate Telemetry server, so we must start the polkadot daemon with the flag `--telemetry-url` like:
/usr/local/polkadot/target/release/polkadot --telemetry-url ws://127.0.0.1:1024 ... rest of your flags
Now we need to add our new data source at the end of the prometheus.yaml config file (in
- job_name: 'PolkadotKusamaNode' scrape_interval: 5s static_configs: - targets: ['YOUR_POLKADOT_NODE_IP:3000'] labels: instance: 'your_polkadot_kusama_node_name'
Save and restart your prometheus server.
Let's open the Prometheus GUI. Browse to http://YOUR_SERVER_IP:9090
In the query input enter the metric
polkadot_best_finalized and you will get the node's latest finalized block:
To get a graph of the value, you can click on Graph tab:
This is only an example, there is a buch of other interesting node metrics available: