Table of contents

  1. Tendermint config file
  2. Scraping Tendermint
  3. Query Tendermint

In the first article we deployed the backend and the first client node to be monitored. Let's continue adding new nodes to our system. This time we will take the advantage of the metric exporter inside Tendermint daemons.

Tendermint daemons are used in many modern Proof of Stake networks. Cosmos, Irisnet, Sentinel, Kava, Kira, Foam ...  The list is incredible, more than 100, and growing. You can find an almost complete list HERE.

Tendermint config file

Tendermind daemon config file includes a commented line for prometheus. Edit your config.toml. By default you will find it in a path similar to this one:

/home/ubuntu/.gaiad/config/config.toml

Edit that file:

vim config.toml	

At the bottom you will find the interesting part. Uncomment lines to leave it like this:

##### instrumentation configuration options #####
[instrumentation]

# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = true

# Address to listen for Prometheus collector(s) connections
prometheus_listen_addr = ":26660"

# Maximum number of simultaneous connections.
# If you want to accept a larger number than the default, make sure
# you increase your OS limits.
# 0 - unlimited.
max_open_connections = 3

# Instrumentation namespace
namespace = "tendermint"

This way we enable prometheus exporter, we give it a port to listen on and we instruct it to start each metric with the prefix tendermint.

Once we save the file we have to restart our daemon. To check if we are ready to receive connections we could show actual listening ports:

:~$ netstat -tulpn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 127.0.0.1:26657         0.0.0.0:*               LISTEN      1986/gaiad              
tcp        0      0 127.0.0.1:6060          0.0.0.0:*               LISTEN      1986/gaiad          
tcp6       0      0 :::26656                :::*                    LISTEN      1986/gaiad          
tcp6       0      0 :::26660                :::*                    LISTEN      1986/gaiad

Last line shows we are litening on por 26660

Scraping Tendermint

Now we have to tell our monitoring server that we have a new node to query for metrics. That is done in prometheus.yaml config file. We have to add the following lines below the firs we included in the first article:

  - job_name: 'MySentryNode'
    scrape_interval: 5s
    static_configs:
    - targets: ['YOUR_IP:26660']
      labels:
        instance: 'mysentrynodename'

Save and restart your prometheus. Now you will have a bunch of metrics of your new added node.

Query Tendermint

Let's have a look in Prometheus GUI. Browse to http://YOUR_SERVER_IP:9090

Input and execute this metric: "tendermint_consensus_height"

Wualah!! Here you have the last block seen by your node.
If you would like you could also install the general node metrics exporter as we already saw in our previous article.

That's all folks!!

Stay tuned