Table of contents
In the first article we deployed the backend and the first client node to be monitored. Let's continue adding new nodes to our system. This time we will take the advantage of the metric exporter inside Tendermint daemons.
Tendermint daemons are used in many modern Proof of Stake networks. Cosmos, Irisnet, Sentinel, Kava, Kira, Foam ... The list is incredible, more than 100, and growing. You can find an almost complete list HERE.
Tendermint config file
Tendermind daemon config file includes a commented line for prometheus. Edit your config.toml. By default you will find it in a path similar to this one:
Edit that file:
At the bottom you will find the interesting part. Uncomment lines to leave it like this:
##### instrumentation configuration options ##### [instrumentation] # When true, Prometheus metrics are served under /metrics on # PrometheusListenAddr. # Check out the documentation for the list of available metrics. prometheus = true # Address to listen for Prometheus collector(s) connections prometheus_listen_addr = ":26660" # Maximum number of simultaneous connections. # If you want to accept a larger number than the default, make sure # you increase your OS limits. # 0 - unlimited. max_open_connections = 3 # Instrumentation namespace namespace = "tendermint"
This way we enable prometheus exporter, we give it a port to listen on and we instruct it to start each metric with the prefix tendermint.
Once we save the file we have to restart our daemon. To check if we are ready to receive connections we could show actual listening ports:
:~$ netstat -tulpn (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.) Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name tcp 0 0 127.0.0.1:26657 0.0.0.0:* LISTEN 1986/gaiad tcp 0 0 127.0.0.1:6060 0.0.0.0:* LISTEN 1986/gaiad tcp6 0 0 :::26656 :::* LISTEN 1986/gaiad tcp6 0 0 :::26660 :::* LISTEN 1986/gaiad
Last line shows we are litening on por 26660
Now we have to tell our monitoring server that we have a new node to query for metrics. That is done in prometheus.yaml config file. We have to add the following lines below the firs we included in the first article:
- job_name: 'MySentryNode' scrape_interval: 5s static_configs: - targets: ['YOUR_IP:26660'] labels: instance: 'mysentrynodename'
Save and restart your prometheus. Now you will have a bunch of metrics of your new added node.
Let's have a look in Prometheus GUI. Browse to http://YOUR_SERVER_IP:9090
Input and execute this metric: "tendermint_consensus_height"
Wualah!! Here you have the last block seen by your node.
If you would like you could also install the general node metrics exporter as we already saw in our previous article.
That's all folks!!